More on the topic…
DuckDB 2.0 ships with async I/O that cuts S3 query times by 2-3x without touching your code. The trick is simple: one thread pool downloads row groups while another decodes them, keeping both the network and CPU busy instead of idle-waiting. On a 2.2 GB Parquet file, queries dropped from 18.8 seconds to 7.7 seconds. The feature runs by default through `read_ahead_depth = -1`, though it doesn't help with tiny files since the overhead is per-file round trips, not data transfer. This matters if you query data lakes on S3 regularly—it's a genuine speedup for the common case.
Recursive CTEs got a complete rewrite that matters most for deep hierarchies like git history or data lineage. The old version re-read the entire table for every level of recursion, so a 20,000-commit ancestry walk took 1.8 to 16 seconds. Version 2.0 reads the table once, builds a single lookup index, and touches only the rows you actually need each round—the same query now runs in 0.10 seconds consistently. Shallow hierarchies like org charts won't see much gain, but anything with thousands of levels moves from "push this to a graph database" to "just write a normal query."
VARIANT is a new data type that compresses JSON by shredding—extracting fields that appear consistently across rows into real columns, leaving irregular fields in a binary remainder. Five million structured log events stored as JSON strings took up significantly more space and ran slower than the same data as VARIANT or as a typed table. The catch is consistency: if `latency_ms` is a number in one row and a string in another, both fall into the remainder and lose the speed benefit. It's worth using for logs and events where most fields have stable types.
Questions about this article
No questions yet.