DataGalaxy acquires YOOI, 🔥 Orchestration CI/CD #73 w/e 28 Feb 2025
Join the 5,200-strong data herd getting all you need to know about Data for your Friday roundup
There are going to be some big big things over the coming weeks. Please share and subscribe if the below are of interest
Streaming Wars V2 - the streaming wars are heating up. is this 2015 all over again? We’ll be bringing you an exclusive write-up with inside info from across the data space
PRICING INCREASES - we have uncovered that there is a concerted effort across multiple vendors to increase costs, and we think we know who is next. That’s why we’ll be putting out a GUIDE To saving csots and protecting your P&L
Orchestra deployment environments- if you’ve ever had to manage multiple airflow instances or kubernetes clusters you know how painful it is. Fortunately, Orchestra now offers a way to get around this issue; powerful, elegant.
Want to get more news like this delivered into your inbox? Subscribe now
Continuous end-to-end Integration in Orchestra
Early this week we made two big announcements
We’ve launched our git-action which enables end-to-end continuous integration and deployment on Orchestra
We announced availability of C-Data in the Portal
The Benefits to using Orchestra CI/CD vs. a legacy, monolithic framework like Airflow or Prefect is immanse:
There is only one set of infrastructure to manage, which removes the need for teams to handle dependencies and requirements in different environments. This is especially true with managed Airflow services such as MWAA, where upgrading versions can be difficult and requires careful planning.
CI/CD becomes simple. By separating Orchestration logic, when there are changes to orchestration logic CI/CD ensures only that is tested. When there are changes to dbt, you can just test the dbt aspect. When there are changes to orchestration logic, you can just test that etc.
CI/CD becomes fast. Instead of relying on slow, monolithic Airflow clusters to test rudimentary things, Data Teams get faster feedback loops and can test more quickly / not get blocked by PRs.
Cost is reduced. Engineers do not need to maintain multiple instances of the same thing, which reduces cost. The current approach in a federated “hub and spoke” model (where different teams get access to the same infrastructure to self-serve) is particularly costly here.
Risk is reduced in skillset. Configuration files are straightforward to manage, so the requirement for specialist, niche devops skills is reduced.
Bottlenecks are eliminated: somewhat unique to Orchestra, but by having a Control Plane instead of just an orchestrator, anyone can view the results of CI/CD runs in a governed manner (you don’t want to give the finance team access to an Airflow UI, but you can give them read-only access to Orchestra) which means anyone can debug and maintain. You won’t need a central platform
Winter Data Conference
Excited to share that anyone using our special code HUGO50 can get a 50% discount to the Winter Data Conference in Zell Am See - check it out here.
Meme Drop
I would actually argue that sometimes it’s the other way around. Either way, it’s still legacy so.
Medium 🧠
🧠 Tackling Out Of Memory (OOM) Errors In PySpark (link)
🧠 Write for Towards Data Science (link)
🧠 Announcing support for Azure DevOps Services in Orchestra (link)
🧠 Unlocking Real-Time Analytics: Connecting SingleStore to Snowflake with Snowpark Container Services (link)
🧠 C-Data availability in Orchestra (link)
LinkedIn🕴
🕴 Why Data Lakes Aren’t Always the Answer (link)
🕴 WDC 2025: Final Tickets Available! (link)
🕴 John Thompson on Humanity’s Role in the AI Era (link)
🕴 Optimizing SQL Joins: When to Use Correlated Subqueries (link)
News 📰
Editor’s Pick
📰 Data Galaxy acquire YOOI (link)
📰 Databricks receive FedRAMP High Authorisation on AWS (link)
📰 Quantum Machines raises $170m in oversubscribed funding round — Processor-based quantum controller startup Quantum Machines has raised $170 million in an oversubscribed Series C funding round… Read More
📰 Ligero Raises $4M Seed Round Led by Galaxy Ventures and 1kx to Introduce The New Chapter In Scalable Data Security and Privacy — Galaxy Ventures and 1kx co-led the round, with participation from Franklin Templeton, Nascent, Anagram, Robot Ventures, Digital Currency Group and ZKV, Ligero said Wednesday. The firm began raising for the round in March 2024 and closed in July 2024, but timing the announcement with the launch of its core product, Ligetron, co-founder Matt DiBiase told The Block… Read More
YouTube and Podcast 🎥
Editor’s Pick
🎥 Conquer Circular Dependencies: Sort Months Effortlessly in Power BI (link)
🎥 How to set-up CI/CD Actions for Complex Data Orchestration Flows (link)
🎥 Connect Orchestra to Azure DevOps for End-to-End Pipelines with Coalesce (link)
Special 💫
💫 Cutter Associate’s 2025 trends for Data Platforms (link)
💫 The Cloud Judgement newsletter (link)
Jobs 💼
💼 Analytics Manager at Paytient (link)
💼 Senior DBT Engineer – Healthcare (100% Remote) at Datasignify (link)
💼 Senior Analytics Engineer (Full-Time) at GlossGenius (link)
💼 Analytics Engineer at Honeylove (link)
💼 Sr. Analytics Lead at Air (link)
Want to save on your ingestion bills? You’ll love this
You can leverage Python for lightweight ELT integrations. Here you’re only paying for compute and not being penalised by row-based pricing models. Pretty neat right? Check it out below / head to Orchestra and start today.
The best place to run dbt?
Don’t believe us? Watch the video below.