Data + AI 2026 Review 🧱, Lyft’s Golden Metrics 🏅, DuckDB 1.5.4 🦆
This newsletter covers major developments across data and AI, including DoorDash’s new Write-Ahead Intent Log for scalable CDC, OpenAI’s Kepler AI analyst for massive data insights, and Lyft’s Metric Semantic Layer for unified business metrics. It also reviews ClickHouse ingestion optimizations, DuckDB’s latest release and agent use cases, Databricks’ Lakehouse//RT and LTAP pushes, plus tools for self-healing pipelines and data quality monitoring.
- DoorDash replaced Debezium with a Write-Ahead Intent Log (WAIL) that writes mutation intents to Kafka and the DB, enabling faster recovery and much higher CDC throughput at Cassandra scale
- MotherDuck claims a $2.40/hour DuckDB instance beats a $64/hour Snowflake 2XL cluster by ~5x on ClickBench, with median query latencies of 3ms, positioning DuckDB for fleets of lightweight agent workloads
- Lyft's Metric Semantic Layer assigns dual business/operational ownership to "Golden Metrics" defined in versioned JSON, auto-propagating changes across dashboards, APIs, and AI agents to prevent divergent calculations
- OpenAI's Kepler analyst now operates over 600 PB of data using Codex-based crawlers and AST-normalized LLM grading to infer lineage, freshness, and hidden semantics automatically
DoorDash rebuilt its change-data-capture pipelines around a Write-Ahead Intent Log after Debezium faltered at Cassandra scale. WAIL writes mutation intents into Kafka and the database, then a consumer layer checks state, enforces schema rules, and emits events. That extra verification speeds recovery and lets them handle far higher throughput without fragile point-to-point connectors.
At Lyft, a Metric Semantic Layer enforces “Golden Metrics” across engineering and analytics teams. Every key metric has both a business owner and an operational owner. Definitions live in versioned JSON, accessible via Python APIs, a self-service UI, the Amundsen catalog, or AI agents. Any update propagates automatically through dashboards and reports, preventing divergent calculations.
DuckDB’s local-first engine is getting an “agent moment.” MotherDuck claims median query latencies of 3 ms and says a $2.40/hour DuckDB instance outperforms a $64/hour Snowflake 2XL cluster by about 5× in ClickBench. Their vision: fleets of lightweight agents spinning up isolated DuckDB environments for profiling, anomaly detection, lineage tracking and more. Meanwhile DuckDB 1.5.4 fixes bugs around VARIANT types, MERGE INTO, JSON, Parquet, Arrow, gzip and CLI behavior.
On the broader Data + AI front, OpenAI’s Kepler system now works on over 600 PB of data, using Codex-based crawlers to infer grain, lineage, freshness and hidden semantics. They layer user-, team- and global-scoped memories for context and use AST-normalized LLM grading to verify SQL results. AWS jumped into the context layer with a knowledge graph built from enterprise data, rules and domain knowledge, surfaced in Iceberg tables on S3 and queryable via Athena, Redshift or Spark. And at Zepto, optimizing the Kafka Connect–ClickHouse path lifted throughput by 45%, eliminated GC pauses and added smarter batching, with two upstream fixes contributed back.