Snowflake and Databricks Team up, dbt microbatch, Orchestra Quality Catalog #54
Join the 4,000-strong data herd getting all you need to know about Data for your Friday roundup
The latest and greatest for w/e 6 October 2024. As always if this was helpful, please do subscribe. If not, please let us know why not. We hate generic-ness. This week its conditional writes - if you don’t know why this is a big deal and you work in data, you probably should.
Looking for opinion articles and cool stuff on trends in data and AI? Check out the medium: https://medium.com/@hugolu87
Orchestra news
New integrations:
Literally any Python running anywhere in the cloud. See: self-hosted Tasks and the Python SDK
Looker (Run Scheduled Plan)
Keboola (run flow)
dbt-core (contact us)
BigQuery Quality Catalog
We’re excited to share the Quality Catalog for BigQuery!
What is it: the BigQuery Quality Catalog is a directory of all your data assets in BigQuery. Orchestra automatically generates a quality map of each asset based on the operations you run in Orchestra.
You can also monitor things like the number of rows manipulated and the number of bytes scanned by query over time, which is useful for identifying long-running / sub-optimal queries
Who is it for: this feature is for data teams to monitor data quality for different assets over time. They can use this to communicate the value of their products to stakeholders
Or head to our news Docs site!
We now have new docs. https://docs.getorchestra.io ←- Check it out!
Meme Dump
Medium 🧠
**Editors Pick**
🧠 Why you should be leveraging a Unified Control Plan for Data Incident Management (link)
🧠 Managing Snowflake with terraform and dbt-core (link)
🧠 Go from Analytics Engineer to Platform engineer in 2 weeks (link)
🧠 Data Architecture (Lessons learned) (link)
🧠 The Slow Painful Death of Agile and Jira (link)
**Editors Pick**
🧠 Hidden Pains of micro batching in dbt (link)
🧠 Your pipeline will fail - here is how data engineers handle it (link)
LinkedIn🕴
🕴Add Snowflake Quality Checks to Pipelines using Snowflake DMFs (link)
News 📰
📰 Altman demands investors shun Perplexity and others (link)
📰 Trigger and Monitor Keboola Flows with Orchestra (link)
📰 Welcome to London, Databricks (link)
**Editors Pick**
📰 Snowflake AND Databricks back Voyage AI in $20m round (link)
YouTube and Podcast 🎥
**Editors Pick**
🎥 Extracting Data from APIs (link)
🎥 Whoop's Carlos Peralta on Building a Data-Driven Culture at Whoop and Moderna (link)
Jobs 💼
Oh dear! We have no new jobs :(
Other 💫
💫 Coalesce Vegas Calendar (link)