No-one is using MCP, the ultimate Databricks Orchestrator, $100k H1B Visas #102 w/e 19 Sep 2025
Join the 6,200-strong data herd getting all you need to know about Data for your Friday roundup
This week the news dominating the headlines is the $100k addition to the price of a H1B Visa, with many VCs and people in tech up in arms about increased labour costs for silicon valley tech start-ups.
Touted as a huge once-in-a-lifetime opportunity for European Tech Scale-ups….what will become of this Trumpian policy?
Orchestra Product Updates
Want to see our latest? sign-up to our webinar in October ——> https://luma.com/90wwlwjk
The ultimate Orchestrator for Databricks
Back in April we wrote about what the ultimate architecture would look like for Databricks.
The ultimate Metadata-driven architecture for Databricks
Creating the most powerful and scalable data architecture on Databricks
I’m please to announce that with the launch of the Orchestra MetaEngine this idea is now officially a reality.
With the Orchestra MetaEngine pretty much everything can now be handled declaratively.
Ingestion parameters can be stored as config in Orchestra
Lakeflow pipelines can be handled declaratively, so we would just parameterise these with the metaengine and handle scheduling from there
We have not tested this, but we believe asset bundles allow users to define dashboards declaratively too. This is already possible with tools such as lightdash
Provisioning resources and IAM roles is possible in Databricks using their SCIM integration and terraform
Previously, users would need to spin-up Airflow or similar, and without a metadata framework scheduling, dependency, and monitoring logic inevitably ends up in the enemy (frenemy?) of many data practitioners: notebooks.
The Databricks Notebook, leveraged properly, can become a modular, reusable component capable of ingesting or transforming any data.
Now, with Neon, Metadata Configurations can also be stored and accessed securely at scale within Databricks itself.
You can see a live demo below:
We are super excited about this one and giving a LIVE WEBINAR on October 1st. Sign-up now for exclusive access. SIGNUP LINK
Medium 🧠
🧠 Snowflake Data Clean Rooms: Free Form SQL Queries (link)
🧠 Rearchitecting Data Pipelines: A Real-World Guide (link)
🧠 TDS Newsletter: How to Make Smarter Business Decisions with AI (link)
🧠 Azure Container Applications Orchestration (link)
🧠 8 Things AWS Does Better Than Azure (Or Vice Versa) (link)
🧠 Seamless Data Sharing with Client-Specific Column Names (link)
🧠 Setting Up and Loading Delta Tables in Databricks (link)
🧠 Unstructured Data is Eating the World: Unlocking Data Engineering’s Next Frontier (link)
🧠 From Python to JavaScript: A Playbook for Data Analytics in n8n with Code Node Examples (link)
LinkedIn🕴
🕴 Why Data Projects Fail (and Succeed): A European Perspective (link)
🕴 From Data Infra to Data Apps: Introducing MooseStack (link)
🕴 The High Performance Data & AI Debate | Big Data LDN Special (link)
🕴 Workday Data Meets Snowflake: Real-Time Access Coming Soon (link)
🕴 Big Data LDN Afterparty with Orchestra & Friends (link)
News 📰
📰 Snowflake tops Fortune Future 50, new CFO highlights AI leadership (link)
📰 Databricks’ $1bn raise headlines red-letter week for FinTech deals (link)
📰 Aleph Raises $29 Million in Series B (link)
YouTube and Podcast 🎥
Editor’s Pick
🎥 AZURE CONTAINER APPS! Run Azure Container Apps in Data Pipelines using Orchestra (link)
🎥 Agentic AI Workshop: Connect Claude to Enterprise Data with Dremio's MCP Server (link)
Special 💫
💫 Data Engineering Weekly #237 (link)
💫 When Dimensions Change Too Fast for Iceberg (link)
Jobs 💼
Don’t forget CURRY’s in the UK are building out an entirely new data platform team. It’s going to be lit. please message Hugo if you want an intro to the hiring manager
💼 Senior Analytics Engineer at GetGround (link)
💼 AI Platform Engineer at Merantix Momentum (link)
💼 Sr. Analytics Engineer at Salve (link)
💼 Senior Analytics Engineer at EvolutionIQ (link)
💼 Senior Analyst Engineer at Data-Sleek (link)
Run 4500 dbt models?
We’re completely rewriting our dbt engine and interested in hearing from YOU if you run more than 4,500 models. If so, please let us know as it would be great to chat!
dbt, dbt core and dbt labs are all trademarks of dbt labs inc