Coalesce Acquires Castor, Orchestra Metadata Framework #76 w/e 21 Mar 2025
Join the 5,200-strong data herd getting all you need to know about Data for your Friday roundup
Orchestra Metadata Framework- announcing sensors, variables and metadata
Today we’re incredibly excited to announce the Orchestra Metadata Framework.
Many architects on Azure have been building metadata frameworks on ADF for years. But the complexity of ADF and highly technical nature of the much-used low-code tool means it is not a scalable way to democratise access to AI and Data workflows.
Orchestra now supports full variable setting and passing between tasks. Combined with sensors for Snowflake, SQL Server, Orchestra Pipelines, ADLS-Gen 2 and AWS S3, it now means you can leverage metadata to trigger pipelines based on configuration.
This means you can build reusable components as databricks jobs, ADF Copy activities, Orchestra Python processes etc. and monitor granular outputs using data that you can send to Orchestra.
Sensors also allow for the processing of data faster, more incrementally, and more efficiently.
We’re going to go into this in more depth next Wednesday 26th March 4pm GMT: sign up here: https://lu.ma/ktissb2h.
Mass consolidation
There is a huge amount of consolidation happening in the data market now. It’s clear people want AI and Data workflow platforms like Orchestra rather than having to stitch together messy point solutions instead of driving value.
Congratulations to our partners Coalesce on their acquisition of Castordoc!
Check out this article we wrote with CastorDoc on the importance of catalogs+orchestration:
https://www.castordoc.com/blog/data-catalogs-and-orchestration
Meme Drop
Medium ðŸ§
🧠Secure Multiple Snowflake Accounts via Private Listing (link)
🧠Six Organizational Models for Data Science (link)
🧠How to drive revenue with streaming data, Snowflake and Embedded reports (link)
🧠Trinity Life Sciences & Snowflake: Advancing Pharmaceutical Insights (link)
🧠Getting Started as a Data Engineer with a Blockchain Scenario — Part VIII (link)
🧠Unlocking the Power of Data: How Data Warehouses Drive Modern Business Success (link)
🧠5 Considerations before you build an Orchestration and Monitoring framework (link)
LinkedIn🕴
🕴 I Am Datapedia 2025: The Ultimate Guide to 330 Data & AI Topics (link)
🕴 Join Snowflake: Work With the Best Minds in AI & Data (link)
🕴 Webinar: What’s New in Orchestra for Effortless Data Pipeline Management (link)
🕴 Real Talk with Nicholas Freund: Startup Truths, No Fluff (link)
News 📰
Editor’s Pick
📰 We want to have a third of our global business in EMEA’: Snowflake’s GVP for EMEA partners - Following Snowflake’s positive full-year results announced at the beginning of March, CRN caught up with Dan Waters, GVP EMEA partners, to find out how the vendor is addressing its partner organisation and where it’s headed in the new financial year… Read More
📰 Silicon Data Secures $4.7M in Seed Funding and Expands Compute Industry Data Product Suite - Strategic Investors include DRW and Jump Trading Group to Support Ongoing Expansion of Cutting-Edge GPU Market Intelligence and Benchmarking Tools… Read More
📰 Advance Auto Parts Probes Potential Cyber Incident Linked To Snowflake - Hackers stole data from some 358,000 employees. Some 44 million loyalty and gas card numbers leaked… Read More
YouTube and Podcast 🎥
Editor’s Pick
🎥 Watch Me Build Microsoft Purview From SCRATCH (link)
🎥 Can DuckDB scale ? ft. Jake Thomas, Jordan Tigani, Michael Ritchie (link)
🎥 Harvesting Insights, Block by Block (Minecraft + Microsoft Fabric) (link)
Special 💫
💫 Cutter Associate’s 2025 trends for Data Platforms (link)
💫 The Cloud Judgement newsletter (link)
Jobs 💼
💼 Senior Product Analytics Specialist at Octus (link)
💼 Senior/Founding Data Engineer at Gratia Health (link)
💼 Data Engineer at Plato (link)
💼 Analytics Engineer for handling "meta" dbt projects at DinMo (link)
💼 Data Platform at HomeToGo (link)
Want to save on your ingestion bills? You’ll love this
You can leverage Python for lightweight ELT integrations. Here you’re only paying for compute and not being penalised by row-based pricing models. Pretty neat right? Check it out below / head to Orchestra and start today.
The best place to run dbt?
Don’t believe us? Watch the video below.