Happy New Year! Here's what we offer in 2025 (Spoiler; the latest data news)
Join the 4,800-strong data herd getting all you need to know about Data for your Friday roundup
Happy New Year and welcome to 2025.
There is a hell of a lot of data content out there. We all live and breathe data, spending far too many hours scouring forums, discourse channels, and god forbid even Linkedin in our daily lives.
We take a note of everything we learn, see and here and put it in this newsletter. We include articles, videos, podcasts, news, thought pieces, and even jobs.
That’s why you should stay subscribed in 2025!
Microsoft Power Automate support in Orchestra and improved dbt core
We’re excited to announce support for Power Automate in Orchestra.
Users can now incorporate Power Automate Flows into their end-to-end pipelines.
This ensures that if Power Automate Flows fail, which in turn are powering important business processes, that the data team or analytics team are the first to know about failures.
Furthermore, where Power Automate is used to power data use-cases like “EL-ing” data back from a data warehouse like Snowflake or Databricks to Azure (think data backt o dynamics, Sharepoint, blob storage, cosmos, etc.), Power Automate Jobs must only run if the data quality is met.
By running Power Automate Flows in Orchestra, Data Teams can ensure that only high quality data gets put back into source and business sytems.
To find out more, please check out the instructional videos below produced by one of our software engineering leads, Tommy!
If you’re ready to get started, check out the Orchestra portal here ⚡
Improved dbt core
Now, whenever you run a dbt core job on Snowflake, BigQuery, Databricks, Fabric etc. the data assets (and quality state etc.) will show up in the Orchestra Data Catalog:
Winter Data Conference
Excited to share that anyone using our special code HUGO50 can get a 50% discount to the Winter Data Conference in Zell Am See - check it out here.
Meme Drop
Data Engineers who try to build a data platform with airflow
Medium 🧠
🧠 Part 2: A Survey of Analytics Engineering Work at Netflix (link)
🧠 Deep Learning for Outlier Detection on Tabular and Image Data (link)
🧠 Learning ML or Learning About Learning ML? (link)
🧠 Top Considerations when considering a migration from SQL Server to Cloud (link)
🧠Unleashing Data’s Full Potential: The Synergy Between Data Lake and Data Vault (link)
🧠 Iceberg insights for S3 Managed Tables (link)
🧠 How to download Spark Locally on Windows (link)
LinkedIn🕴
🕴 How to successfully run internal data projects (link)
🕴 An Engaging Conversation with Winfried Adalbert Etzel (link)
🕴 2024: Cutting Through the Noise in the Data Ecosystem (link)
🕴 Top 10 Most-Read Data Engineering Articles of 2024 (link)
Editor’s Pick
🕴 Where does Fabric actually belong??? (link)
🕴 The Azure/Databricks ML/Data stack for analysts (link)
News 📰
📰 Apheris raises $20.8M to power world’s leading life sciences data networks (link)
YouTube and Podcast 🎥
🎥 Data + AI World Tour Paris 2024 (link)
🎥 Met Office Streamlines Weather Data Delivery With Snowflake Marketplace (link)
Special 💫
💫 Running Duck DB in the browser with web assembly - WhatTheDuck (link)
Jobs 💼
💼 Analytics Engineer at Branch App
💼 Analytics Engineer (Intermediate) at Hiive
💼 Senior Analytics Engineer at Justworks
💼 Business Intelligence Developer at American Health Marketplace
💼 Senior Analytics Engineer (if desired also: Engagement Manager) at Flatiron Data
Want to save on your ingestion bills? You’ll love this
You can leverage Python for lightweight ELT integrations. Here you’re only paying for compute and not being penalised by row-based pricing models. Pretty neat right? Check it out below / head to Orchestra and start today.
The best place to run dbt?
Don’t believe us? Watch the video below.