CDC at WHOOP: Self-Service Replication for Hundreds of Postgres Tables | WHOOP Engineering
Short summary
WHOOP built a CDC pipeline that automatically syncs Postgres schema changes to Snowflake without manual intervention, by splitting the problem into two layers—a cheap append-only Bronze layer for raw events and a periodic Silver layer for materialized replicas—rather than doing continuous upserts like most systems.
- The old system ran hundreds of expensive Spark streaming jobs doing continuous upserts every 30 minutes; the new design cuts this to one Flink job per database using dynamic routing, eliminating the constant reconciliation overhead.
- Schema changes now propagate automatically: adding a column to Postgres flows through Debezium to Avro to Iceberg without tickets or manual backfills, because the publication definition lives in the service team's own repo alongside their Liquibase migrations.
- Splitting streaming (Bronze, append-only, immediate) from upserting (Silver, every 4 hours, materialized) lets consumers choose their tradeoff—most don't need real-time state and can query the cheaper, simpler Bronze layer.
Full summary
WHOOP replaced a CDC pipeline that was burning money on continuous upserts. Their old system ran one Spark Structured Streaming job per table, hundreds of them constantly reconciling data every thirty minutes just to keep tables fresh—except most consumers didn't actually need that freshness. Schema changes were manual coordination nightmares: adding a Postgres column meant filing tickets and backfilling entire tables. Observability was thin. Configuration lived in the wrong place, split between Kafka settings and Postgres migrations, so new tables would silently fail to replicate for months. The real problem wasn't the technology—Debezium and Iceberg worked fine. It was the operating model: every change made Data Platform the bottleneck instead of letting service teams own their own data.
They fixed it by separating concerns. Bronze is append-only, capturing every CDC event exactly as Debezium emits it—small files, cheap writes, no reconciliation. Silver materializes the actual Postgres replica by upserting from Bronze every four hours instead of continuously. Consumers pick what they need: query Silver if you can tolerate a few hours of staleness and want the table to look like Postgres, or query Bronze if you need change semantics with lower latency. Almost nobody needs the latter, so continuous upserts were the wrong default. One Debezium connector reads from a Postgres publication using logical replication, serializes to Avro against AWS Glue Schema Registry, and publishes one Kafka topic per table. The publication itself is managed with Liquibase migrations inside the service team's repo—adding a table is just an ALTER PUBLICATION statement that gets code-reviewed alongside the schema change. New tables flow through the entire stack automatically without any tickets or PRs.
Flink writes Bronze using one job per database with a fan-in/fan-out pattern, not one job per table. This collapses fifty deployments, checkpoints, and connectors down to one, using Iceberg 1.10's Dynamic Sink to route records based on topic name and create tables on the fly. Schema changes propagate automatically: when Debezium sees a new column, it registers a new Avro schema, Flink writes it to Bronze, and Silver picks it up on the next run. Type widening works the same way through Avro and Iceberg's promotion rules. Connectors deploy through a Terraform module that handles IAM, secrets, Datadog monitors, and JMX metrics—replication lag, event counts, errors all route to the owning team's oncall, not Data Platform's. Bronze records carry five CDC metadata columns (operation type, timestamp, LSN, transaction ID) and partition by day, with nightly compaction on the last two days optimized for the common query pattern: recent changes.