Click any tag below to further narrow down your results
Links
A developer tested whether DuckDB could compete with Spark and Trino at large scales and found that a single powerful machine running DuckDB outperforms distributed clusters on 10TB queries—faster than Trino and half the cost of multi-node setups. The key insight: most time is spent waiting on S3 reads, not computation, so distribution overhead doesn't help.
- At 10TB scale, single-node DuckDB on an r7gd.4xlarge beats Trino on identical queries by 1.7-2.1x, and costs half as much as 5-13 node clusters despite being 2-3x slower than them
- S3 read rate, not compute speed, determines performance at scale—DuckDB's default single-threaded S3 reads leave CPUs 90% idle, but running 4x more threads than cores achieves 3x faster read rates
- DuckDB v2.0 (alpha) expects 2-3x faster S3 reads, potentially making single-node DuckDB the cheapest option for datasets up to 10TB
DuckDB has proven to be superior to Polars when handling large datasets, particularly 1TB of data. While DuckDB effectively manages memory and execution with a robust design, Polars struggles with large data processing, leading to out-of-memory errors.
- DuckDB handled the 1TB benchmark successfully while Polars crashed with out-of-memory errors
- DuckDB's memory management and execution engine scale more effectively to large datasets than Polars'
- Polars is optimized more for in-memory, smaller-scale workloads and breaks down at larger scales like 1TB