Why we're moving our warehouse product off DuckDB

Open original

PostHog built its warehouse on DuckDB, then hit limits at scale and is now rebuilding it on a custom Trino engine with its own table format, HogLake. The post explains why DuckDB and DuckLake fell short for customers with very large, complex workloads.

  • Single-node DuckDB was hard to run safely in the cloud because PostHog had to size each worker before queries arrived, and predicting CPU and memory per query was difficult.
  • DuckLake's portable catalog schema limited optimizations: at PostHog's scale a single-table write could take over three minutes, and over 99% of catalog statistics rows belonged to deleted tables.
  • At about 2 PB, roughly 1 trillion rows and 15.3 million snapshots, DuckLake broke in many ways, and PostHog ended up carrying patches in a fork instead of contributing upstream.

PostHog built its warehouse product on DuckDB about five months before writing this, giving each customer their own single-node instance so that tools like dbt could connect over Postgres. The trouble was scale. Customers ran more complex queries than expected, which forced them into very large instances. Because a single-node engine has to be sized before a query arrives, PostHog couldn't give customers a good way to predict the CPU and memory a given query would need. Adding disks was too slow and expensive to be a real fix, and fetching data from S3 without async support was another bottleneck that DuckDB only recently addressed. The DuckLake lakehouse format, which PostHog installed by default, ran into its own limits. Because its catalog schema has to stay portable across many SQL databases, engine-specific optimizations were off the table. At PostHog's size, a single-table write could take over three minutes, since every commit re-read statistics for the whole catalog. Over 99% of the catalog's statistics rows belonged to tables that no longer existed, and cleaning them up made writes 30 to 50 times faster, but the rows kept coming back. PostHog also put all of its own data into one DuckLake, reaching about 2 PB, roughly 1 trillion rows, and 15.3 million snapshots. Since upstream changes weren't landing fast enough, the team ended up carrying patches in a fork, which undercut the community benefit that was a main reason to use it. Direct Postgres connections on port 5432 proved hard to scale too, with reconnect handling, connection TTLs, and load balancing all harder than with HTTP, plus an endless list of client quirks that kept generating support requests. PostHog is now rebuilding on its own flavor of Trino with a new open table format it calls HogLake, and it is testing that at petabyte scale internally before offering it to design partners. The company says it has stopped hyping the underlying technology and cares more about the experience customers get. The post is also an open invitation to teams with slow or costly warehouses and large query workloads to get in touch.

Ask about this article