Evolving the Scientific Data Engineering Experience in TDP

For pharmaceutical data teams to get the most out of their scientific data, they need the ability to build, test, deploy, and maintain reliable and performant data pipelines—with speed and confidence. Such an experience is more than a convenience; it’s a strategic enabler for organizations aiming to leverage their data effectively. By focusing on usability, automation, and scalability, TetraScience empowers scientific data engineers (SDEs) to deliver high-quality pipelines more quickly and reliably. 

This blog post will explore some of the current challenges data teams face in the pipeline development space, and introduce several key features we’ve built to address those challenges. 

Challenges

1. Slow iteration cycles
   • Configuration-heavy, low-code protocols like python-exec often require many on-platform iterations as developers refine their code. Rapid prototyping and deployment are crucial to minimizing lengthy iteration cycles and boosting productivity.
2. Limited pipeline tracking
   • Troubleshooting: When pipelines break or produce out-of-spec results downstream, users need an easy way to revert to the last working version while troubleshooting the issue. 
   • Auditing and compliance: Tracking pipeline configurations and identifying who made changes can be difficult. Auditors and compliance officers often rely on painstaking, ad hoc approaches to piece together how critical data was produced.

Solution Overview

The latest version of the Tetra Data Platform (TDP) introduces two significant updates to tackle these challenges head-on.

1. Improved pipeline user experience (UX): A completely revamped experience makes it easier to quickly configure and update pipelines in TDP. 
2. Pipeline versioning: Users can now easily view the versioned history of their pipelines and roll back to previous versions with a click.

Improved Pipeline UX

We’ve completely redesigned the pipeline creation and editing experience in TDP, focusing on two key principles for developers:

• Accessibility: Ensuring all relevant information and actions are at their fingertips
• Optimized layout: Maximizing screen real estate when configuring more information-dense low-code pipelines

To access the new pipeline UX, navigate to Pipeline Manager > New Pipeline.

The new creation screen places all pipeline metadata anchored at the top, with an easy-to-navigate ribbon helping to quickly access the parts of your pipeline that matter most. Each tab in the ribbon dynamically updates its status, helping keep track of the configuration process. This redesigned configuration flow will make it much easier for developers to quickly configure, edit, and maintain their pipeline. 

Pipeline Versioning

Pipeline versioning provides a detailed history of changes to your pipeline’s configuration. You can see what modifications were made, who made them, and when. This feature captures the evolution of a pipeline and facilitates debugging by helping to identify when, where, and by whom issues arose.

To get started with pipeline versioning, users can access it anywhere on the platform where pipeline information is available. For example, they can navigate to the Pipeline Manager page, select a pipeline, and click View Version History.

This will bring them to the Version History page, where they can see the full version history of the pipeline and compare the current version against any of its historical counterparts. The current configuration will be anchored on the left as users explore previous versions of the pipeline in the right pane, with changes highlighted.

It's also possible to roll back to any previous version of the pipeline with a single click using the Restore Version button. 

Looking Ahead

TetraScience remains committed to enabling scientific data engineers to produce stellar scientific data products for their organizations. Future improvements will focus on:

• Accelerating speed-to-iteration for developers
• Boosting confidence in their deployed artifacts
• Granting more flexibility in how users can orchestrate their pipelines

Conclusion

The future of scientific data engineering hinges on tools and processes that are dynamic and intelligent. Investing in a better development experience is not just an incremental improvement—it’s an impactful competitive advantage. The improvements offered by the new pipeline configuration experience and pipeline versioning capabilities provide SDEs with much-needed tools to deliver high-value, trusted datasets as efficiently as possible.

For a closer look at these new features, check out the demo video.