BigQuery Streaming, but otherwise a #quiet August 2024 #45
Join the 3,500-strong data herd getting all you need to know about Data for your Friday roundup
The latest and greatest for w/e 9 August 2024. As always if this was helpful, please do subscribe. If not, please let us know why not. We hate generic-ness.
Orchestra adds Okta SSO and Github Actions
As part of our commitment to security Orchestra now supports SSO via Okta.
The team have also created this helpful guide on how to get set-up.
Did you know: Orchestra also supports connecting to Snowflake via Key/Pair Auth
Branching with AWS Sagemaker
Did you know you can leverage Orchestra to run custom dbt-style data quality tests? It’s a powerful feature, and with our branching feature, you can leverage it to do cool things like automate the training and deployment of ML/AI models using Sagemaker.
Fire memes
Lakeflow looks suspiciously like ADF, don’t you think?
Medium 🧠
🧠 Snowflake vs Databricks; a useful comparison for 2024 (link)
🧠 The BigQuery Qualify clause (link)
🧠 How Notion handles 200bn data entities (link)
🧠 A scalable ingestion framework in MS Fabric (link)
🧠 Migrating and testing out pyairbyte (link)
🧠 People do not like this latest article “The rise of the notebook engineer” (link)
**Editors Pick**
🧠 How Pinterest do Analytics (link)
LinkedIn🕴
🕴 BigQuery release continuous Queries and real-time rELT (link)
🕴 Difference between Junior and Senior engineers (link)
🕴 Trio of Leaders leave Open AI (link)
News 📰
📰 Databricks release Lakeflow Oracle CDC (link)
YouTube and Podcast 🎥
🎥 Big Data vs. Thicc data by the Seattle Data Guy
🎥 Check out this awesome chat on how to arrange a data engineering team org chart-wise from Linear (link)
Jobs 💼
Data Engineers @ Meta, Rightmove and Sainsbury’s
Head of Data @ L’Oreal
Head of Data Delivery @ Chubb
Analytics Engineering @ Carta
Data Analysts @ TikTok, Deliveroo and Bumble
Other 💫
💫 Microsoft progresses planned $3.3 billion AI data centre (link)
💫 Check-out Joe’s recent course with Deeplearning AI and AWS on Coursera (link)
💫 The latest data news (link)