More data news is incoming following a quiet start to August, with Databricks raising more money (presumably to provide employee liquidity and maintain funding momentum) with Snowflake committing to $6bn in AWS Compute spend.
Meanwhile, we’re continuing to roll out integrations like SFTP, Step Functions, AWS Glue advancements (logs) and a couple of really big features in the pipeline.
You can also
Install orchestra-cli via brew: `brew install orchestra-hq/tap/orchestra-cli`
which is pretty cool.
Ton of awesome jobs on the market too!
Orchestra Product Features
New: Databricks Cost Agents - an automated AI Agent in Orchestra for reducing and mitigating Databricks costs.
New: SFTP Loaders (docs)
New: AWS Step Functions
New; sources in dbt lineage
New: n8n integration
New: dbt live alerting logs.
New: Anomaly Detection native in Orchestra (link)
The Orchestra Runtime: The Control Plane for AI Agents is absolutely flying
Readonly API Tokens in Orchestra
Assets and Lineage - you can update Assets via API to create a banging context layer
AWS Step Functions are Released (link)
NEW Check-out our sick Cloud Hosted MCP (link)
NEW Check-out our new case study with FEELD - how the dating app powers ML and AI workflows with Orchestra
We run
dbt debugwhen validating an orchestra dbt connection now which is pretty great!!Custom Webhook Alerts (send custom webhook payloads to any monitoring tool e.g. Datadog)
Postgres Sensor enabled (trigger a pipeline whenever something happens in Postgres)
trigger on any condition (link)
Trigger an Orchestra Pipeline as a task (link)
Scaling sequential Tasks in the MetaEngine - one config, infinite possibilities. Metadata-driven frameworks in ADF are a thing of the past!
Large Task Outputs - you can now pass data between tasks up to many Megabytes
Force Cancel - you can force cancel any task, pushing it into a cancelled state
Omni Integration
Pipeline run concurrency is live! Read more here
Monitor your Estuary Syncs from Orchestra (read more here)
MCP Video is live. This is the only MCP you need or API you need to reliably fix your data pipelines automatically with AI.
You can now run claude agents in Orchestra. These can automate anything, especially things you can design as code submission tasks.
AI Assistant is in Orchestra too - ask our assistant questions without having to leave to the docs!
Released our EKS operator - you can now trigger and monitor flows on Kubernetes in AWS
UI RESKIN! DARK MODE AND LIGHT MODE! TELL YOUR FRIENDS!
Improved Docs Site — CHECK OUT OUR DOCS AND ASK IT QUESTIONS
Workspaces - say goodbye to multiple airflow instances
Improved Lineage Filtering
Medium 🧠
🧠 Airflow 3 vs Airflow 2 | Should you upgrade Airflow? (link)
🧠 The Future of Data Engineering Is Autonomous — Meet Snowflake AIM (link)
🧠 A Day in the Life of a Data Scientist in 2026 (link)
🧠 Announcing Orchestra’s Partnership with Databricks (link)
🧠 How to Set Up Cross-Region Inference for CoCo in APJ (link)
🧠 I Made an LLM Lay Siege to My Minecraft House (link)
🧠 Top-N Per Group: The SQL Pattern Behind Hundreds of Interview Questions (link)
🧠 SDA 06: Detecting Primary Keys, Foreign Keys, and Join Relationships (link)
LinkedIn 🕴
🕴 Paying Too Much for Your AI? (link)
🕴 The “X Is Dead” Cycle in Data and AI (link)
🕴 Same Models, Better Results: The CoCo AI Benchmark (link)
🕴 Why Databricks Needs an Agentic Control Plane (link)
🕴 Building a Data Platform Is About More Than Tools (link)
🕴 Dynatrace + Arize: Is AI Observability Becoming One Platform? (link)
News 📰
📰 Databricks Grows >80% YoY, Surpasses $7B Revenue Run-Rate, Scales Lakebase, Genie, and Unity AI Gateway (link)
📰 Why Snowflake’s $6B AWS Commitment and Natoma Acquisition Matters for Streaming (link)
📰 Databricks wanted to raise $1B, investors wanted $15B. It settled on $5B at a $190B valuation. (link)
📰 AI firm Databricks valued at $190 billion as it bags $5 billion in funding (link)
📰 Databricks Agrees to Acquire Panther, Further Establishing the Security Lakehouse Category (link)
📰 Databricks acquires Electric to give every AI agent its own Postgres database (link)
YouTube and Podcast 🎥
Editor’s Pick
🎥 When Should You START Using Microsoft Fabric? (link)
🎥 CLAUDE FABLE IS INSANE | Using Fable to migrate Dagster to Orchestra in under 10 MINUTES (link)
🎥 Top 5 dbt Alternatives in 2026 (link)
🎥 AI codes with outdated libraries. Why is this unsolved? (link)
Special 💫
💫 Data Engineering Weekly #282 (Link)
💫 What an Ontology for AI Agents Actually Needs (Link)
Jobs 💼
💼 AWESOME ROLE for my Utah network (Link)
💼 Clickup are HIRING in the awesome Michael Revelo’s team (Link)
💼 Great opportunity to join a growing data team in Leeds UK at Red Badger (Link)
💼 Senior Analytics Engineer at Confido (Link)
💼 Product Analytics Engineer at Gratia Health (Link)
Run dbt models cheaply and easily with state?
If you’re looking for an easy way to run your dbt core models, look no further than Orchestra. Orchestra gives you state-aware orchestration out the box, which can reduce your dbt core costs, and make scheduling much easier!
dbt, dbt core and dbt labs are all trademarks of dbt labs inc