Databricks open-sources declarative ETL framework powering 90% faster pipeline builds

At this time, at its annual Data + AI Summit, Databricks introduced that it’s open-sourcing its core declarative ETL framework as Apache Spark Declarative Pipelines, making it obtainable to the complete Apache Spark group in an upcoming launch. 

Databricks launched the framework as Delta Stay Tables (DLT) in 2022 and has since expanded it to assist groups construct and function dependable, scalable knowledge pipelines end-to-end. The transfer to open-source it reinforces the corporate’s dedication to open ecosystems whereas marking an effort to one-up rival Snowflake, which not too long ago launched its personal Openflow service for knowledge integration—an important element of knowledge engineering. 

Snowflake’s providing faucets Apache NiFi to centralize any knowledge from any supply into its platform, whereas Databricks is making its in-house pipeline engineering know-how open, permitting customers to run it anyplace Apache Spark is supported — and never simply by itself platform.

Declare pipelines, let Spark deal with the remaining

Historically, knowledge engineering has been related to three important ache factors: complicated pipeline authoring, handbook operations overhead and the necessity to preserve separate programs for batch and streaming workloads. 

With Spark Declarative Pipelines, engineers describe what their pipeline ought to do utilizing SQL or Python, and Apache Spark handles the execution. The framework mechanically tracks dependencies between tables, manages desk creation and evolution and handles operational duties like parallel execution, checkpoints, and retries in manufacturing.

“You declare a sequence of datasets and knowledge flows, and Apache Spark figures out the precise execution plan,” Michael Armbrust, distinguished software program engineer at Databricks, stated in an interview with VentureBeat. 

The framework helps batch, streaming and semi-structured knowledge, together with recordsdata from object storage programs like Amazon S3, ADLS, or GCS, out of the field. Engineers merely must outline each real-time and periodic processing by means of a single API, with pipeline definitions validated earlier than execution to catch points early — no want to keep up separate programs.

“It’s designed for the realities of contemporary knowledge like change knowledge feeds, message buses, and real-time analytics that energy AI programs. If Apache Spark can course of it (the information), these pipelines can deal with it,” Armbrust defined. He added that the declarative method marks the newest effort from Databricks to simplify Apache Spark.

“First, we made distributed computing useful with RDDs (Resilient Distributed Datasets). Then we made question execution declarative with Spark SQL. We introduced that very same mannequin to streaming with Structured Streaming and made cloud storage transactional with Delta Lake. Now, we’re taking the subsequent leap of constructing end-to-end pipelines declarative,” he stated.

Confirmed at scale 

Whereas the declarative pipeline framework is about to be dedicated to the Spark codebase, its prowess is already recognized to 1000’s of enterprises which have used it as a part of Databricks’ Lakeflow resolution to deal with workloads starting from day by day batch reporting to sub-second streaming purposes.

The advantages are fairly comparable throughout the board: you waste approach much less time growing pipelines or on upkeep duties and obtain significantly better efficiency, latency, or price, relying on what you need to optimize for.

Monetary providers firm Block used the framework to chop growth time by over 90%, whereas Navy Federal Credit score Union diminished pipeline upkeep time by 99%. The Spark Structured Streaming engine, on which declarative pipelines are constructed, permits groups to tailor the pipelines for his or her particular latencies, all the way down to real-time streaming.

“As an engineering supervisor, I really like the truth that my engineers can concentrate on what issues most to the enterprise,” stated Jian Zhou, senior engineering supervisor at Navy Federal Credit score Union. “It’s thrilling to see this stage of innovation now being open-sourced, making it accessible to much more groups.”

Brad Turnbaugh, senior knowledge engineer at 84.51°, famous the framework has “made it simpler to help each batch and streaming with out stitching collectively separate programs” whereas decreasing the quantity of code his workforce must handle.

Completely different method from Snowflake

Snowflake, one among Databricks’ largest rivals, has additionally taken steps at its latest convention to deal with knowledge challenges, debuting an ingestion service referred to as Openflow. Nevertheless, their method is a tad completely different from that of Databricks when it comes to scope.

Openflow, constructed on Apache NiFi, focuses totally on knowledge integration and motion into Snowflake’s platform. Customers nonetheless want to wash, remodel and combination knowledge as soon as it arrives in Snowflake. Spark Declarative Pipelines, then again, goes past by going from supply to usable knowledge. 

“Spark Declarative Pipelines is constructed to empower customers to spin up end-to-end knowledge pipelines — specializing in the simplification of knowledge transformation and the complicated pipeline operations that underpin these transformations,” Armbrust stated.

The open-source nature of Spark Declarative Pipelines additionally differentiates it from proprietary options. Customers don’t must be Databricks prospects to leverage the know-how, aligning with the corporate’s historical past of contributing main initiatives like Delta Lake, MLflow and Unity Catalog to the open-source group.

Availability timeline

Apache Spark Declarative Pipelines will likely be dedicated to the Apache Spark codebase in an upcoming launch. The precise timeline, nevertheless, stays unclear.

“We’ve been excited in regards to the prospect of open-sourcing our declarative pipeline framework since we launched it,” Armbrust stated. “Over the past 3+ years, we’ve realized lots in regards to the patterns that work greatest and stuck those that wanted some fine-tuning. Now it’s confirmed and able to thrive within the open.”

The open supply rollout additionally coincides with the final availability of Databricks Lakeflow Declarative Pipelines, the business model of the know-how that features extra enterprise options and help.

Databricks Data + AI Summit runs from June 9 to 12, 2025