Reddit - The heart of the internet

Open original

A developer tested whether DuckDB could compete with Spark and Trino at large scales and found that a single powerful machine running DuckDB outperforms distributed clusters on 10TB queries—faster than Trino and half the cost of multi-node setups. The key insight: most time is spent waiting on S3 reads, not computation, so distribution overhead doesn't help.

  • At 10TB scale, single-node DuckDB on an r7gd.4xlarge beats Trino on identical queries by 1.7-2.1x, and costs half as much as 5-13 node clusters despite being 2-3x slower than them
  • S3 read rate, not compute speed, determines performance at scale—DuckDB's default single-threaded S3 reads leave CPUs 90% idle, but running 4x more threads than cores achieves 3x faster read rates
  • DuckDB v2.0 (alpha) expects 2-3x faster S3 reads, potentially making single-node DuckDB the cheapest option for datasets up to 10TB

The author tested whether DuckDB could handle large-scale queries by building distributed systems on top of it, then discovered the entire premise was wrong. They initially assumed Spark and Trino would dominate at 1TB+ scales, and Trino did win early benchmarks by 4-5x at 100GB across 16 nodes. But digging into the bottleneck revealed CPUs sitting idle at 90%—the real constraint was S3 read throughput, not compute speed. Trino won because it kept more bytes in flight per node. DuckDB's httpfs implementation uses the same threads for both Parquet decoding and S3 requests (one per core), which meant only a few MB in flight at any time given ~30ms latency per read. The author fixed DuckDB's read problem with a custom reader that spawned 4x as many threads as CPU cores, tripling throughput. This leveled the playing field: distributed DuckDB matched Trino on 10TB TPC-DS queries, though distribution overhead erased DuckDB's natural speed advantage. At that point, the author realized distribution was unnecessary. A single r7gd.4xlarge machine running plain single-node DuckDB (r7gd.8xlarge for one query) beat Trino on all three 10TB benchmarks by 1.7-2.1x, and cost about half as much compared to Trino's 5-13 node clusters. The cost math gets clearer at smaller scales: one DuckDB machine beat 17-node Spark or Trino clusters at 100GB while costing roughly 2x less. With v2.0 expected to deliver 2-3x faster S3 reads, the author believes single-node DuckDB on a large machine will become the cheapest option for datasets up to around 10TB. Distribution might eventually push that ceiling higher, but right now the sweet spot is one powerful box.

Ask about this article