Click any tag below to further narrow down your results
Links
DuckDB 2.0 ships three major performance improvements: async I/O for S3 queries (2-3x faster), recursive CTEs for deep hierarchies (40x+ faster), and a new VARIANT type for JSON data (6x faster queries, 2.7x smaller storage). The author benchmarked each on real datasets to show which workloads actually benefit.
- Async I/O decouples network downloads from CPU work via a separate thread pool, cutting S3 query times from 18.8s to 7.7s on a 2.2GB file with zero query changes and automatic enablement.
- Recursive CTEs now build a lookup index once instead of re-reading the entire table per recursion level, dropping a 20,000-commit ancestry walk from 1.8-16s to 0.10s consistently.
- VARIANT type "shreds" JSON into separate columns for consistent fields (level, service, latency_ms) while keeping messy fields in binary storage, delivering 6x faster field queries and 2.7x smaller on-disk size than plain JSON strings.
DuckDB has proven to be superior to Polars when handling large datasets, particularly 1TB of data. While DuckDB effectively manages memory and execution with a robust design, Polars struggles with large data processing, leading to out-of-memory errors.
- DuckDB handled the 1TB benchmark successfully while Polars crashed with out-of-memory errors
- DuckDB's memory management and execution engine scale more effectively to large datasets than Polars'
- Polars is optimized more for in-memory, smaller-scale workloads and breaks down at larger scales like 1TB