Click any tag below to further narrow down your results
Links
PostHog built its warehouse on DuckDB, then hit limits at scale and is now rebuilding it on a custom Trino engine with its own table format, HogLake. The post explains why DuckDB and DuckLake fell short for customers with very large, complex workloads.
- Single-node DuckDB was hard to run safely in the cloud because PostHog had to size each worker before queries arrived, and predicting CPU and memory per query was difficult.
- DuckLake's portable catalog schema limited optimizations: at PostHog's scale a single-table write could take over three minutes, and over 99% of catalog statistics rows belonged to deleted tables.
- At about 2 PB, roughly 1 trillion rows and 15.3 million snapshots, DuckLake broke in many ways, and PostHog ended up carrying patches in a fork instead of contributing upstream.
A developer tested whether DuckDB could compete with Spark and Trino at large scales and found that a single powerful machine running DuckDB outperforms distributed clusters on 10TB queries—faster than Trino and half the cost of multi-node setups. The key insight: most time is spent waiting on S3 reads, not computation, so distribution overhead doesn't help.
- At 10TB scale, single-node DuckDB on an r7gd.4xlarge beats Trino on identical queries by 1.7-2.1x, and costs half as much as 5-13 node clusters despite being 2-3x slower than them
- S3 read rate, not compute speed, determines performance at scale—DuckDB's default single-threaded S3 reads leave CPUs 90% idle, but running 4x more threads than cores achieves 3x faster read rates
- DuckDB v2.0 (alpha) expects 2-3x faster S3 reads, potentially making single-node DuckDB the cheapest option for datasets up to 10TB