What if a conventional SQL database could handle file tracking for a data lakehouse? That concept underpins DuckLake, an open-source table lakehouse format developed by Amsterdam-based DuckLabs. Rather than distributing manifest files across object storage systems, DuckLake stores table metadata within a standard SQL database, functioning as a central coordinator for data management. By treating lakehouse metadata as a relational database problem instead of a distributed systems challenge, the format delivers advanced data lake capabilities through Parquet files and SQL databases while reducing traditional complexity.
DuckLake 1.1 arrived in October alongside DuckDB 2.0.0 LTS. Pedro Holanda, principal engineer at DuckLabs and lead developer of DuckLake, explained the reasoning behind the format's design in an interview at the company's Amsterdam office. Holanda has been involved with the DuckDB open-source initiative since its early research phase and provided insight into DuckLake's development trajectory.
According to Holanda, "These lakehouse formats exist so people can exchange their database systems without having to do migrations." The complexity of managing multiple systems necessitates a database layer to track table versions. DuckLake draws on decades of research to minimize file proliferation and operational overhead during periods of accelerating data growth, leveraging the widespread familiarity developers have with SQL and schema management.
"The simplicity is the way you fetch your data," Holanda noted. "You're going to issue queries somewhere. The server is going to answer these questions back, and you know exactly which Parquet files you have to read. So the Parquet files don't change. Just the catalog."
Data lake administrators face numerous maintenance responsibilities, particularly when file counts grow. DuckLake enables consolidation of multiple files into single objects to reduce query overhead. This consolidation can span multiple snapshots, further decreasing file counts. The format also addresses the "small files problem" common in streaming data lakes through inlining and flushing mechanisms.
Traditional data lakes ingesting from multiple sources simultaneously accumulate substantial file counts. This proliferation forces queries to traverse numerous metadata files to identify relevant Parquet files, then read those files themselves—a process that degrades performance and complicates development workflows. Holanda described the problem: "you end up with so many files, so when you have to query it, you're going to have to traverse a bunch of metadata files just to figure out which Parquet files you need to read, and then you're going to have a bunch of Parquet files to read as well."
DuckLake has earned recognition for speed, ease of use, and minimal infrastructure requirements. Part of this efficiency stems from Parquet's structural similarity to DuckDB storage—both employ columnar organization with row groups—reducing data transformation overhead.
Holanda described a key innovation: "One thing that we realized when we were building DuckLake is that, 'Hey, we have a database system now, we have a catalog; what if we do this radical idea of storing data on tables?' With DuckLake, you get your actual data representation as a table in a catalog, and we call this inlining."
During Amazon's Q3 earnings call in late October, CEO Andy Jassy highlighted the operational burden of transferring data between storage and analytics systems as customer data volumes accelerate. Amazon's acquisition of DuckLabs reflected recognition of DuckDB's approach to this challenge. Jassy stated: "With our acquisition of DuckLabs, DuckDB will become the universal SQL dialect of AWS storage and analytics, simplifying data processing with a modern analytics engine that is fast, simple to use and ubiquitous, and where customers will be able to write SQL queries once and then execute using any AWS analytics service. For the first time in analytics, customers will have zero migration or portability costs between query engines. This is a big deal, and you can imagine how it will also help agents do efficient querying as well."
DuckDB and DuckLake remain available under the MIT open-source license, governed by the independent DuckDB Foundation. The DuckLake 1.1 release includes over 200 bug fixes alongside new functionality, with native integration into DuckDB.