Pluggable Connector Framework accelerates development and release of data connectors
The fast-paced and data-intensive field of biopharmaceutical research and development requires a future-facing scientific data management solution. Storing, securing, and archiving raw data are table stakes. While superior to paper log books, the traditional SDMS model is becoming obsolete. Many SDMSs have few external integrations and lack interoperability across diverse data sources and applications. These features are essential for advanced functions including data engineering and AI use cases. The traditional SDMS functions like a walled garden. It serves its basic role of protecting your scientific data but fails to promote connections that allow data to be used by outside applications to unlock its true value.
A next-generation solution must provide seamless connectivity with every component of a modern lab data ecosystem, from data generators like instruments to data consumers like analytics and AI. The Tetra Scientific Data and AI Cloud™ has the tools to do just that, including the Pluggable Connector Framework. Released in Tetra Data Platform v3.6, the Pluggable Connector Framework enhances the speed and efficiency of updating connectors, bringing agility to lab connectivity and data management. Let’s explore how it works and the benefits it provides.
See the Pluggable Connector Framework in action for real-time connector updates. In this example, an Amazon Web Services (AWS) S3 bucket Tetra Connector is updated with just a few clicks to a newer version that is capable of file type filtering using glob patterns.
Dynamic integration capabilities with Tetra Connectors
The new Pluggable Connector Framework introduced with Tetra Data Platform v3.6 provides functionality that accelerates the connector release cycle, offers rapid iteration, and scales to the growing needs of your organization.
The Tetra Data Platform v3.6 improves lab connectivity in 3 ways:
- Independent connector release cycles
The new Pluggable Connector Framework allows scientific data to be rapidly integrated by allowing new and updated connectors to be deployed independent of Tetra Data Platform releases. Pluggable Connectors can be upgraded and reconfigured using a user-friendly interface in the Tetra Data Platform. Moreover, this interface improves connector lifecycle efficiency to accelerate data assembly, enabling much faster integration with new data sources.
- Designed for rapid development
For technical solutions, the freedom to create and iterate is a key to success. The Tetra Scientific Data and AI Cloud is designed with iteration in mind, unlike monolithic traditional SDMSs. The Pluggable Connector Framework unlocks the potential to update and deploy new connectors and versions as needed. Rapid iteration in connector development can accelerate data assembly and enable faster deployment of new features, bug fixes, and connectors for new applications and data sources. The framework comes with many built-in features, such as configuration and lifecycle management, and standardized functions that Tetra’s developers can leverage to build connectors faster. The framework responds to shifting priorities by enabling the delivery of beta connector versions to customers that provide users with early access to solutions.
- Scaling at your pace
Business cases with expanding needs require scalable solutions. The Pluggable Connector Framework incorporates design enhancements that unlock more granular scaling for connector deployment. The optimized framework now provisions computing resources for each connector, offering outstanding performance for customers with large numbers of connectors. The result is a platform that can scale data inputs to your growing needs.
A closer look at Tetra Connectors and the Pluggable Connector Framework
Tetra Connectors are integral onramps for data from instruments, software, and file-hosting sources capable of exporting or publishing data. When a connected data source performs an activity (e.g., a Benchling assay run is created), an event record becomes available and actionable. The data source may publish to a networked event broker (such as AWS EventBridge), messaging queue (AWS Simple Queue Service [SQS] or Message Queuing Telemetry Transport [MQTT]), or it may have direct communication with the Tetra Connector through a REST Application Programming Interface (API).
For Tetra Connectors using our subscription model, the broker or queue holds a list of events to be processed by the connector, while the connector is subscribed to the queue and monitors for new events. For those using our API model, the connector queries the data source’s inbound API at a specified polling interval (e.g., every 10 minutes). For both models, the connector processes the event information and queries additional resources if applicable. The relevant scientific data is then replatformed into the Tetra Scientific Data and AI Cloud.
The figure below shows an example of a Tetra Connector workflow. In this workflow, the Benchling notebook will publish an event to EventBridge, which is then transferred to the SQS message queue. The Tetra Benchling Connector monitors SQS for new events and removes messages after an event is processed. The Tetra Connector will then use a URL contained in the event message to contact the Benchling API for event details, and subsequently trigger a data pipeline to process the event.
With Tetra Data Platform v3.6, the Pluggable Connector Framework surfaces important elements of this sophisticated design and provides an intuitive user interface (UI) to deploy or reconfigure connectors. Within the Tetra Data Platform, users can create or update an existing connector with a few mouse clicks. They can customize connector information, including connector type, version, subscription addresses, client credentials, and other parameters. The connector configuration is updated immediately, and has status checks at a configured interval. Release notes for each version can be found in the Connector READ ME in the Details tab on the Tetra Data Platform.
Summary
The Tetra Scientific Data and AI Cloud represents a significant departure from the traditional SDMS model. It has been purpose built to integrate all data producers and consumers throughout the modern laboratory environment, from instruments and software to analytics and AI, independently of the vendors. This functionality exemplifies what a next-generation solution should do. It is achieved, in part, through a flexible and dynamic integration strategy that provides data accessibility and interoperability with speed. The Pluggable Connector Framework helps meet these requirements with faster release cycles, rapid development, and scalable solutions. This framework increases the cadence of innovation, establishing the necessary interoperability for analytics and AI use cases.
To learn more about Pluggable Connectors and other exciting features of Tetra Data Platform v3.6, read the release notes. To learn how to create, configure, update, and monitor your own Pluggable Connector, read the documentation.
If you’d like to discuss how our Pluggable Connector Framework can help your team, contact one of our experts today.
If you’re interested in other new features released with Tetra Data Platform v3.6, you can read about our self-service pipelines, file journey, and multi-byte character support.
Want to try out these new features?
TetraU provides in-depth training for the Tetra Scientific Data and AI Cloud, including our new TDP v3.6 features. Check out TetraU for workshops and training content developed by the data and life science experts at TetraScience.