Roundup #29 w/e 15 April 2024
We missed you last week. But we're back. Pipelines failing as the clocks change. Blaring omissions in the MAD Landscape. Data Engineering becoming fun again and dbt alternatives raising $$. Read on.
Latest and greatest for w/e 15 April 2024. As always if this was helpful, please do subscribe. And if not, please let us know why not. We hate generic-ness.
Also I am very sorry but I completely mis-scheduled last week’s roundup. Apologies.
Orchestration, Alerting and Observability for free
We launched our ***Free Tier*** a few weeks back and the uptick has been immense.
We’re seeing folks get started with Orchestration who were going to spend $100k on a platform engineer. We’re seeing people build Data Quality Monitoring pipelines using the latest in Anomaly Detection on Snowflake Cortex.
New Integrations
Check out Portable - they have over 1000 connectors for long-tail and hard-to-reach APIs. They’re a must to any stack. Read more here, or watch here.
Google Big Query
We are very excited to announce that we have partnered with the Google Cloud Platform and released our BigQuery data warehouse, to transform how BigQuery users approach orchestration and observability.
Users will be able to:
Trigger and monitor BigQuery tasks context of an entire DAG.
Leverage BigQuery and dbt to track bytes scanned by query for an approximation of cost.
Similar to Snowflake, users of BigQuery can leverage Orchestra to monitor Data Quality. By leveraging Vertex AI under the hood, users can conduct data quality tests in BigQuery to monitor and prevent data quality issues or data incidents.
Users can add BigQuery tasks that check for staging tables and drop them if a run fails - saving data engineers and analytics engineers alike from having to go into the BigQuery UI.
Try Orchestra below
Medium / Blog 🧠
👉 Interesting piece on whether LLMs can be used to label data (link)
👉 How to avoid Data Pipelines failing on these 10 days - a bit late! (link)
👉 Who is Sridhar Ramaswamy and what does this mean for Snowflake? (link)
👉 How Data Engineering became fun again (not just Orchestra being founded!) (link)
👉 A comprehensive guide to running dbt core on GitHub actions (link)
Linkedin🕴
🕴The MAD 2024 Landscape is out with a blaring key omission… (link)
🕴Snowflake Cortex LLM costs (link)
News 📰
**Editor’s Pick**
📰 Congrats to our friends at Coalesce who have raised $50m Series B to transform data for Snowflake users (link) Orchestra is the only tool in the landscape that helps you connect Coalesce with your entire stack, for orchestration, metadata, cost, and lineage. If you’re a Coalesce user get in touch!
📰 Data labelling company, Sapien, raises a $5m seed round (link)
📰 Oden, who specialise in Data & AI solutions for the manufacturing industry, raised a $28.5m Series B (link)
Youtube and Podcast 🎥
🎥 What tools Data Engineers should know about in 2024 with Seattle Data Guy! (link)
Jobs 💼
**Editor’s Pick**
Incredible Roles for Anyone in Data Engineering and Analytics Engineering:
💼 The Big 🐟 - Head of Data’s @ Inigo and Rightmove 🇬🇧
💼 Data Engineers calling @ Rev, Panda, Unmind and Winton 🇬🇧
💼 Great ops for Analytics Engineers @ Branch App, Deel, ResMed & Peloton 🇺🇸
Other 💫
**Editor’s Pick**
What did we miss?
<3
Thanks for including me!