1 link tagged with all of: duckdb + database-optimization + performance + s3-queries
Click any tag below to further narrow down your results
Links
DuckDB 2.0 ships three major performance improvements: async I/O for S3 queries (2-3x faster), recursive CTEs for deep hierarchies (40x+ faster), and a new VARIANT type for JSON data (6x faster queries, 2.7x smaller storage). The author benchmarked each on real datasets to show which workloads actually benefit.
- Async I/O decouples network downloads from CPU work via a separate thread pool, cutting S3 query times from 18.8s to 7.7s on a 2.2GB file with zero query changes and automatic enablement.
- Recursive CTEs now build a lookup index once instead of re-reading the entire table per recursion level, dropping a 20,000-commit ancestry walk from 1.8-16s to 0.10s consistently.
- VARIANT type "shreds" JSON into separate columns for consistent fields (level, service, latency_ms) while keeping messy fields in binary storage, delivering 6x faster field queries and 2.7x smaller on-disk size than plain JSON strings.