OpenAI’s models breached Hugging Face during testing, Airflow Projects migrated in minutes, we forgot about Parabellum acquiring Crux #145 w/e 24 July 2026
Join the 8,500-strong data herd getting all you need to know about Data for your Friday roundup
Pretty insane week with Open AI’s models deciding to breach out of Hugging Face. Does this look like greater AI capabilities or is Open AI simply jealous of Anthropic’s mythos scare-marketing? We will need to find out.
Separately, we’ve recently built an AI that is capable of migrating Airflow projects to Orchestra in minutes. It just goes to show that the language of orchestration is becoming a massive commodity. How it interfaces with AI is not.
We also forgot to report that Crux Data, the platform for Financial Data Data Pipelines, was acquired by Private Equity. An interesting foray into verticalised data solutions is also showing signs of consolidation.
And of course there is still a lot of hubbub about databricks raising yet more cash.
What people achieve with Orchestra Runtime
We’ve recently launched the ability to build, run and monitor any AI agents with Orchestra context. Check-out what people are doing with it!
Our customers are already using the Orchestra Runtime to run migration projects, create self-healing systems requiring less maintenance, and do more with less.
Octopus Energy created a self-healing architecture where agents automatically detect and fix issues with dbt pipelines. It’s expected to save hundreds of data engineering hours while increasing trust in the data.
We are super excited for this one and have been continually amazed by the feedback. This is truly a great time to be using AI to get your time back and spend less time on busywork and more time on what matters.
The best thing? A lot of Runtime is included in all existing (And new) Orchestra Accounts - start here today.
Orchestra Product Features
New: Anomaly Detection native in Orchestra (link)
The Orchestra Runtime: The Control Plane for AI Agents is absolutely flying
Readonly API Tokens in Orchestra
Assets and Lineage - you can update Assets via API to create a banging context layer
NEW Check-out our sick Cloud Hosted MCP (link)
NEW Check-out our new case study with FEELD - how the dating app powers ML and AI workflows with Orchestra
We run
dbt debugwhen validating an orchestra dbt connection now which is pretty great!!Custom Webhook Alerts (send custom webhook payloads to any monitoring tool e.g. Datadog)
Postgres Sensor enabled (trigger a pipeline whenever something happens in Postgres)
trigger on any condition (link)
Trigger an Orchestra Pipeline as a task (link)
Scaling sequential Tasks in the MetaEngine - one config, infinite possibilities. Metadata-driven frameworks in ADF are a thing of the past!
Large Task Outputs - you can now pass data between tasks up to many Megabytes
Force Cancel - you can force cancel any task, pushing it into a cancelled state
Omni Integration
Pipeline run concurrency is live! Read more here
Monitor your Estuary Syncs from Orchestra (read more here)
MCP Video is live. This is the only MCP you need or API you need to reliably fix your data pipelines automatically with AI.
You can now run claude agents in Orchestra. These can automate anything, especially things you can design as code submission tasks.
AI Assistant is in Orchestra too - ask our assistant questions without having to leave to the docs!
Released our EKS operator - you can now trigger and monitor flows on Kubernetes in AWS
UI RESKIN! DARK MODE AND LIGHT MODE! TELL YOUR FRIENDS!
Improved Docs Site — CHECK OUT OUR DOCS AND ASK IT QUESTIONS
Workspaces - say goodbye to multiple airflow instances
Improved Lineage Filtering
Medium 🧠
🧠 Why Senior Data Engineers Write SQL Differently (link)
🧠 Loop Engineering for RAG Generation: Iterate top-k One at a Time (link)
🧠 CRM Fields to Deal Wins: How Cortex AI Changed Our Sales Forecast (link)
🧠 When Data Science Makes Us Sad: The Story of an Overbooked Flight (link)
🧠 Your Best Data Engineer Is About to Become Your Worst Manager (link)
🧠 Data Modeling for AI Applications: Designing Databases Beyond Users and Orders (link)
🧠 How Much of a Data Science Workflow Can Run on a GPU Today? Part 1: Accelerating Data Preparation (link)
🧠 How to Migrate from Microsoft SQL Server to Snowflake: A Technical Playbook for Enterprise Data Teams (link)
LinkedIn 🕴
🕴 Will AI Agents Replace Traditional Data Engineering Tasks? (link)
🕴 Building the Graph Instead of Documenting Five Viewpoints (link)
🕴 DuckDB Per Agent: A New Approach to Agent Architectures (link)
🕴 Snowflake Cortex vs. Databricks Genie: Choosing the Right AI Model (link)
🕴 Why Skill Architecture Matters More Than More Skills (link)
🕴 Snowflake CoWork Is Now Generally Available (link)
🕴 Enable Snowflake CoCo and Unlock Enterprise AI Productivity (link)
🕴 8 Lessons from Using dbt Since 2019 (link)
News 📰
📰 Enterprise Data Platforms in 2026: Databricks, Snowflake, Microsoft Fabric & AI Analytics (link)
📰 Open AI breaches Hugging Face (link)
📰 Databricks to raise strategic funding at $188bn valuation (link)
📰 Utah-based AI-FinTech, Databento, has Landed a $97 Million Series B Funding to Build-out its Global Market-data Infrastructure (link)
📰 Some old but important news - Crux acquired by PE (Link)
YouTube and Podcast 🎥
Editor’s Pick
🎥 Migrate Airflow to Orchestra - The Agentic Way (link)
🎥 How to Change a Measure in Power BI SAFELY (link)
🎥 Is Your Orchestra Account Following Best Practices? AI Agents Can Tell You (link)
🎥 Run your JIra ticket automations seamlessly (link)
Special 💫
💫 Data Engineering Weekly #279 (link)
💫 Our podcast with the man Daniel Beach! (Link)
https://substack.com/@dataengineeringcentral/p-204554354
Jobs 💼
💼 Analytics Engineer at Extend (link)
💼 Senior Analytic Engineer at Ericsson (link)
💼 Senior Data Scientist at Juul (link)
💼 Senior Data Engineer at WalletConnect (link)
💼 Global Lead, CRRD (Crisis Response, Recovery, and Development Department) Data & Analytics at International Rescue Committee (link)
Run dbt models cheaply and easily with state?
If you’re looking for an easy way to run your dbt core models, look no further than Orchestra. Orchestra gives you state-aware orchestration out the box, which can reduce your dbt core costs, and make scheduling much easier!
dbt, dbt core and dbt labs are all trademarks of dbt labs inc