Snowflake/dbx intensifies, Orchestra announces dbt Core support ⚡#56
Join the 4,200-strong data herd getting all you need to know about Data for your Friday roundup
The latest and greatest for w/e 27 October 2024. As always if this was helpful, please do subscribe. If not, please let us know why not. We hate generic-ness.
💡 Read more about the Orchestra Lightdash integration here.
Want to keep an eye on other news? Hit subscribe below.
Or read on Medium https://medium.com/@hugolu87
A deployment environment for dbt Core
We’re super excited to announce that Orchestra now supports dbt Core. This means you can connect your git to Orchestra and Orchestra will run your dbt models for you, in the context of an e2e DAG.
There are a lot of folks that don’t want the pain of maintaining infrastructure and monitoring for dbt Core, but also are happy developing locally and don’t need to pay for an IDE. Orchestra charges dbt Core minutes at $0.033 which scales down, so if you’re running dbt core jobs for 2 hours a day thats $4 a day. Pretty good.
The free plan also comes with 30 free minutes, and Scale-up includes 400 minutes.
Are your current providers’ reps warning you about rate hikes? Why not see if Orchestra can help you on your bill, give you a platform that’s incredible for orchestration, and give them the ol’ Switcheroo?? Chat here.
Read the full Orchestra dbt Core announcement here.
Drama
Snowflake came out with a graph of TCO going up with Databricks and brushed it off as “Spark-based SAAS”.
Generally, people actually quite like Databricks’ managed spark. After all, they’ve been doing it for 12 years.
Widely being looked at as a big marketing blunder and I think I agree..Snowflake is such a good product. It needs to focus on what it does best and stop bashing others. Other bashing is not good! We are data engineers! We’re not idiots!
Meme Drop
Medium 🧠
**Editors Pick**
🧠 Introducing Orchestra dbt Core; Lightweight dbt Core transformations (link)
🧠 dbt™ latest features and takeaways from Coalesce™ 2024: One dbt™ (link)
🧠 Idbt™ latest features and takeaways from Coalesce 2024: iceberg (link)
🧠 You dont’ have a data problem (link)
🧠 A fantastic comparison of Delta Live Tables in Databricks to Dynamic Tables in Snowflake (link)
🧠 Big Data Vendors embrace Apache Iceberg (link)
🧠 Incremental but Incomplete (link)
🧠 IThe problem with Snowberg (link)
🧠 IHow to empower citizen analytics engineers with Coalesce and Orchestra (link)
LinkedIn🕴
🕴dbt Snapshots (link)
🕴Probably the funniest thing I saw on Linkedin all week -FML level cringe (link)
🕴A great take on the next steps in Iceberg Catalogs (link)
🕴Is Snowflake’s Databricks positioning a huge marketing blunder? This insightful chap thinks so (link)
🕴Recruiters learned about Chat GPT and everyone started spearheading things (link)
News 📰
**Editors Pick**
📰 AWS and Databricks strike 5 year deal on AI Chips (link)
Scale AI raise $1BN AND MOVE TO $13.8bn valuation (link)
YouTube and Podcast 🎥
**Editors Pick**
🎥 What’s New in Orchestra??
🎥 A brief dip into Google Cloud Vertex AI (link)
🎥 The only constant is Data Change (link)
Jobs 💼
💼 Analytics Engineer at Loop returns (link)
💼 Senior Product Analyst at ProductBoard
💼 Very interesting data engineering role at Landmarc (recommended (link))
💼 Interested in Data Engineering for one of the best charitable orgs in the UK? Follow Enthuse or get in touch (link)
Other 💫
💫 What type of engineer are you? : https://www.confessionsofadataguy.com/the-3-types-of-data-engineers-which-one-are-you/