Last week we announce Orchestra Runtime: The Control Plane for AI Agents.
With Orchestra Runtime, anyone can easily build, run and monitor AI Agents. Orchestra Runtime combines the world-class usability of Orchestra with a completely new, secure, serverless infrastructure for running agents.
The Orchestra Runtime comprises four elements.
Orchestra runtime: scalable, sandboxed, on-demand infrastructure for running AI Agents. The Runtime has built-in integrations, git-access, AI Skills, metadata, alerting, and monitoring out-of-the-box.
Agent Builder: a user interface for easily defining, building and conversing with AI Agents.
Agent Orchestrator: leveraging the Orchestra Platform, the ability to trigger and run Agents as part of existing or new flows — sandboxed, version-controlled and backed by code.
Agent Context: Orchestra automatically creates a context graph of clean, reliable data that acts as an “engineering brain” — a repository of information required to reliably build, test and fix workflows.
What people achieve with Orchestra Runtime
Our customers are already using the Orchestra Runtime to run migration projects, create self-healing systems requiring less maintenance, and do more with less.
Octopus Energy created a self-healing architecture where agents automatically detect and fix issues with dbt pipelines. It’s expected to save hundreds of data engineering hours while increasing trust in the data.
We are super excited for this one and have been continually amazed by the feedback. This is truly a great time to be using AI to get your time back and spend less time on busywork and more time on what matters.
The best thing? A lot of Runtime is included in all existing (And new) Orchestra Accounts - start here today.
Orchestra Product Features
New Readonly API Tokens in Orchestra
NEW Assets and Lineage - you can update Assets via API to create a banging context layer
NEW Check-out our sick Cloud HOsted MCP (link)
NEW Check-out our new case study with FEELD - how the dating app powers ML and AI workflows with Orchestra
We run
dbt debugwhen validating an orchestra dbt connection now which is pretty great!!Custom Webhook Alerts (send custom webhook payloads to any monitoring tool e.g. Datadog)
Postgres Sensor enabled (trigger a pipeline whenever something happens in Postgres)
trigger on any condition (link)
Trigger an Orchestra Pipeline as a task (link)
Scaling sequential Tasks in the MetaEngine - one config, infinite possibilities. Metadata-driven frameworks in ADF are a thing of the past!
Large Task Outputs - you can now pass data between tasks up to many Megabytes
Force Cancel - you can force cancel any task, pushing it into a cancelled state
Omni Integration
Pipeline run concurrency is live! Read more here
Monitor your Estuary Syncs from Orchestra (read more here)
MCP Video is live. This is the only MCP you need or API you need to reliably fix your data pipelines automatically with AI.
You can now run claude agents in Orchestra. These can automate anything, especially things you can design as code submission tasks.
AI Assistant is in Orchestra too - ask our assistant questions without having to leave to the docs!
Released our EKS operator - you can now trigger and monitor flows on Kubernetes in AWS
UI RESKIN! DARK MODE AND LIGHT MODE! TELL YOUR FRIENDS!
Improved Docs Site — CHECK OUT OUR DOCS AND ASK IT QUESTIONS
Workspaces - say goodbye to multiple airflow instances
Improved Lineage Filtering
Medium 🧠
🧠 Knowing When You’re Done: Coordinating Async Flows with Wait & Notify (link)
🧠 We built all our pipelines with Claude, AI and the Orchestra Runtime (link)
🧠 I’m quitting my role as a Data Engineer: AI is too good (link)
🧠 Loop Engineering in Snowflake CoCo (link)
🧠 Nobody Told Me Data Engineering Was Just One Giant System (link)
🧠 AI Agents Explained: What Is a ReAct Loop and How Does It Work? (link)
🧠 GenPage: Towards End-to-End Generative Homepage Construction at Netflix (link)
🧠 What Can We Do When Memory Becomes the New Bottleneck in Data Engineering? (link)
LinkedIn 🕴
🕴 What If AI Built Your Entire Data Pipeline? (link)
🕴 The Future of Data Engineering with AI (link)
🕴 Join the Team That’s Moving Fast (link)
🕴 What If Two-Day Sprints Became the New Normal? (link)
🕴 How to Optimize Large-Scale ML Pipelines (link)
🕴 Why GitHub Actions Is Failing Data Engineering (link)
News 📰
📰 Ticketmaster Hit With Discovery Sanction in Snowflake Hack Suit (link)
📰 Global venture funding hits record $510B in first half as AI boom accelerates (link)
📰 Crunchbase Data: Global Startup Investment Hit Record $510B In H1 2026 As AI Boom Accelerates Funding And Exits (link)
YouTube and Podcast 🎥
Editor’s Pick
🎥 Join THE CUBE: The Community Data Professionals Have Been ASKING For (link)
🎥 Power BI Tutorial for Beginners (Step-by-Step in 30 Minutes) (link)
🎥 ORCHESTRA RUNTIME | Closing the Agentic Control Loop with Orchestra Runtime to AI-fix Data Pipelines (link)
🎥 ORCHESTRA RUNTIME | Build Data and AI Pipelines with an AI Agent Runtime (link)
🎥 Creating Skills and Agents in Orchestra (link)
🎥 Using Claude Code to Build an Iceberg Lakehouse (link)
🎥 AI is relearning everything databases already knew ft. Stephanie Wang (link)
Special 💫
💫 Data Engineering Weekly #276 (link)
💫 Announcing leetdata.ai — A Practice Ground for Data Engineers (link)
💫 Benchmarking Hardwood 1.0 on a Threadripper 9980X (link)
💫 1BRC on a Threadripper 9980X (link)
Jobs 💼
💼 Senior Data Engineer at Farm to People (link)
💼 Analytics Engineer - no visa sponsorship at Home Chef (link)
💼 Senior AI/ML Engineer at BrooklynData (link)
💼 Contract: Senior Data Analytics Engineer at Feeld (link)
Run dbt models cheaply and easily with state?
If you’re looking for an easy way to run your dbt core models, look no further than Orchestra. Orchestra gives you state-aware orchestration out the box, which can reduce your dbt core costs, and make scheduling much easier!
dbt, dbt core and dbt labs are all trademarks of dbt labs inc