Klarna Files for IPO in the US, AWS Lean heavily into S3, Orchestra supports dbt-fabric⚡#58
Join the 4,600-strong data herd getting all you need to know about Data for your Friday roundup
The latest and greatest for w/e 17 Nov 2024. As always if this was helpful, please do subscribe. If not, please let us know why not. We hate generic-ness.
Want to keep an eye on other news? Hit subscribe below.
Or read on Medium https://medium.com/@hugolu87
Orchestra Product Updates - More Azure More Microsoft
Azure Fabric Data Quality Testing → execute T-SQL or stored procs on Fabric and run data quality tests
dbt-fabric improvements (now your data catalog for fabric will be auto populated by dbt-fabric)
Reconciling data across bigquery, databricks, SQL Server, Snowflake or Fabric? These are all now supported (try it here).
Parameters - if you work with a METADATA FRAMEWORK you can set global parameters that are retained for task retrying - check out the docs.
Still use Shipyard? You won’t for much longer
Shipyard has now been turned off. Check out our migration guide here.
Winter Data Conference
Excited to share that anyone using our special code HUGO50 can get a 50% discount to the Winter Data Conference in Zell Am See - check it out here.
Meme Drop
Read more here (link)
Medium 🧠
AWS are leaning heavily into S3. ! million buckets per account. Automatic compaction for iceberg and Express One Zone High Performance Storage class. Are Data Warehouses going to struggle?
🧠 Autopilot Your Data Science: Boost Productivity with LLMs & AI Automation (link)
🧠 Orchestration with Data Quality: Announcing Data Reconciliation (link)
🧠 My Path Towards Data @ Netflix (link)
🧠 Techniques for Exploratory Data Analysis and Interpretation of Statistical Graphs (link)
🧠 Unlocking New Capabilities in Cortex Analyst: Multi-turn Conversations, JOINs, and More! (link)
🧠 Medallion Architecture on Bigquery (link)
🧠 Orchestrating with Data Quality: announcing data reconciliation (link)
🧠 How Delhivery migrated 500 TB of data across AWS Regions using Amazon S3 Replication (link)
LinkedIn🕴
🕴 Check out Snowflake BUILD if you missed it! (link)
🕴 How AI is Revolutionizing Data Workflows – Special Episode with Rivery CEO Itamar Ben Hemo (link)
🕴 Building a Portable Analytics Suite with Docker, DuckDB, dbt, and SQLMesh Integration (link)
News 📰
**Editors Pick**
📰 Trump victory likely to boost deal-making in 2025, likely changes at FTC will benefit deals already in the works (link)
📰 Klarna files for IPO in the US (link)
YouTube and Podcast 🎥
**Editors Pick**
🎥 Dremio & Snowflake: Better Together (link)
🎥 Going deeper into incremental dbt models in bigquery (link)
🎥 Reconcile data between SQL Server and Snowflake (link)
Special 💫
**Editors Pick**
💫 Airflow vs. Step? Or you could just use Orchestra (link)
Jobs 💼
💼 Analytics Engineer at Loop returns (link)
💼 Senior Product Analyst at ProductBoard
💼 Very exciting data engineering role at Landmarc (recommended, (link))
💼 Interested in Data Engineering for one of the best charitable orgs in the UK? Follow Enthuse or get in touch (link)
💼 Interested in building the future of Data in VC at Dawn? Get in touch to learn more about this one.
Some great data roles around platform and architecture at Lundbeck Pharma (link)
💼 Senior BI Developer at SERB Pharma (link)
Running dbt Core™️ ? You’ll love this
💡 Read more about the Orchestra dbt Core™️ integration here
dbt Core obviously needs to run in an orchestrator - if you’re not doing this already, what are you doing? Many Data Teams are realising that their 99% uptime isn’t actually enough to get stakeholders to trust the Data in BI Use-cases; uptime needs to be much higher, so that’s why you need orchestration and visibility of pipelines.
Orchestra supports running dbt, with some great features:
Enhanced debugging! Identify dbt model/test cost bottlenecks easily
Simplification! One less platform to manage; let Orchestra be your dbt™️ HQ
Price! Simple and lightweight usage-based pricing where the unit costs decrease as your models increase
Worth talking? Chat here.