Post-conference quiet. Databricks move to tackle Azure #89 w/e 20 June 2025
Join the 5,400-strong data herd getting all you need to know about Data for your Friday roundup
After the Snowflake and Databricks summits things have been pleasantly quiet this week.
It is now clear that Snowflake and Databricks are finally begining to diverge. Snowflake, remaining as a warehouse and appealing to folks that are more “upmarket” (business users) while Databricks aim to directly compete with Microsoft (ambitious).
People are still talking a lot about ducklake
Other than that check-out the awesome content and product releases from Orchestra
Managed CI/CD for dbt in Orchestra
Has anyone else struggled to set-up CI/CD for dbt?
I LOVE dbt. I remember the first dbt run I did and it really did feel like black magic.
But builing a production workflow requires CI/CD, setting up different environments. This normally involves different tables/schema/databases, multiple instances of Airflow, and setting up git runners in finicky ways.
This is quite time-consuming to do, hard to debug, and ends up being a real blocker for analytics folks trying to scale dbt beyond a small time.
So I'm really excited to share that Orchestra now supports dbt-core in your CI/CD flows 🥳
With Orchestra you can
☑️ Use a pre-made git-action to run dbt-core in Orchestra
☑️ No need to set-up multiple Airflow instances. Just use serverless environments
☑️ No need to worry about artifact management for only running changed nodes; Orchestra automatically stores the manifest.json from the latest production (successful) run you can easily do dbt run state:modified
☑️ No need to set-up alerting (handled via the Orchestra UI)
☑️ No need to worry about aggregating metadata from your CI/CD flow (all collated by Orchestra and pushed automatically to your warehoues)
✅ Complete lineage and logs rendered in Orchestra to monitor CI/CD runs alongside production runs
✅ Orchestrate actions beyond dbt. Want to ensure your the dashboards or downstream dbt processes aren't affected by your change? This can be handled in Orchestra too (as it's a full orchestrator); just set run_downstream_tasks to true
Medium 🧠
🧠 How to implement CI/CD using dbt-core and Orchestra (link)
🧠 Everything from the Databricks 2025 summit (link)
🧠 Dev Day @Snowflake Summit 2025 (link)
Editor’s Pick
🧠 The Snowflake Databricks Chess war is over (link)
🧠Introducing Serverless Compute for Workflows: Simpler, Smarter Job Execution (link)
🧠 From Configuration to Orchestration: Building an ETL Workflow with AWS Is No Longer a Struggle (link)
🧠 From Data Assumptions to Data Quality Assurance: Why we need Testing, Cleaning, and Monitoring (link)
LinkedIn🕴
🕴 From Data Warehousing to AI: Are We Repeating the Same Mistakes? (link)
🕴 Empowering the Next Gen of Data Leaders with Snowflake & AI (link)
🕴 Snowflake vs. Databricks: The Real Difference in RBAC Security (link)
🕴 Struggling with CI/CD for dbt? Here’s a Better Way (link)
🕴 From Legacy to AI: Don’t Miss the Great Data Migration 🚀 June 23 (link)
🕴 AI and the Platform Wars: Are We Solving the Wrong Problem? (link)
Editor’s Pick
🕴 Furries LOVE data architects (link)
News 📰
Editor’s Pick
📰 Intapp Partners With Snowflake to Add Analytics to DealCloud (link)
📰 SBJ Power Up: Teamworks boosting AI, data science with latest funding (link)
📰 Sports Data Firm Teamworks Hits $1 Billion Valuation With Funding Round (link)
📰 In case anyone missed it — Twirl Orchestration joins Modal (link)
YouTube and Podcast 🎥
Editor’s Pick
🎥 CI/CD in Orchestra for dbt-core (VERY COOL)
🎥 How to use Dataform for Data Transformation in an Orchestra Pipeline (link)
🎥 Financial Analytics on SAP Data with Databricks SQL (link)
Special 💫
💫 Data Engineering Weekly #224 (link)
Jobs 💼
💼 Analytics Engineer at Covariance.ai (link)
💼 Developer, Business Intelligence at Uncommon Schools (link)
💼 Data Engineer / Sr. Data Engineer at FinQore (link)
💼 Senior Software Engineer, Data Infrastructure at Ro (link)
💼 Senior Developer, BI at Uncommon Schools (link)
Want to save on your ingestion bills? You’ll love this
You can leverage Python for lightweight ELT integrations. Here you’re only paying for compute and not being penalised by row-based pricing models. Pretty neat right? Check it out below / head to Orchestra and start today.
The best place to run dbt?
Don’t believe us? Watch the video below.
dbt, dbt core and dbt labs are all trademarks of dbt labs inc