Click any tag below to further narrow down your results
Links
WHOOP built a CDC pipeline that automatically syncs Postgres schema changes to Snowflake without manual intervention, by splitting the problem into two layers—a cheap append-only Bronze layer for raw events and a periodic Silver layer for materialized replicas—rather than doing continuous upserts like most systems.
- The old system ran hundreds of expensive Spark streaming jobs doing continuous upserts every 30 minutes; the new design cuts this to one Flink job per database using dynamic routing, eliminating the constant reconciliation overhead.
- Schema changes now propagate automatically: adding a column to Postgres flows through Debezium to Avro to Iceberg without tickets or manual backfills, because the publication definition lives in the service team's own repo alongside their Liquibase migrations.
- Splitting streaming (Bronze, append-only, immediate) from upserting (Silver, every 4 hours, materialized) lets consumers choose their tradeoff—most don't need real-time state and can query the cheaper, simpler Bronze layer.
DuckDB 2.0 Alpha adds dynamic attach statements that let you pull credentials from environment variables instead of hardcoding them, solving a major friction point when working across multiple data lakes. Early production testing shows less than 1% failure rates, and the release bundles this with Quack and official DuckLake support to make local-first lakehouse workflows actually practical.
- Dynamic attach statements now accept environment variables for credentials (usernames, passwords, account numbers), eliminating the need to hardcode sensitive data or switch to Python for multi-catalog federation
- Production pilot running hourly jobs showed <1% failure rate over 10 days, reading Parquet from S3 and writing to Iceberg tables via RDS-DMS
- Release combines dynamic attach with Quack and official DuckLake support, completing the local-first lakehouse story but raising open questions about concurrent writer behavior on shared Iceberg tables
Andy Warfield recounts how genomic researchers struggled with moving large datasets between local filesystems and S3, spurring the creation of S3 Files to present S3 buckets as POSIX-compatible file stores. The post then covers how Amazon added S3 Tables for managed Iceberg tables and built-in vector indexing to cut storage friction and speed up analytics and search.
- S3 Files was inspired by genomics researchers at UBC who wasted hours copying sequencing data between NFS and S3 because their GATK4/Spark pipeline still required POSIX filesystem semantics despite massive serverless parallelism.
- Agent-driven development is lowering the barrier for domain experts to build apps, making it more critical to decouple persistent data from any single application or workflow.
- S3 handles over 25 million Parquet requests daily, often via Hive tables, but open formats like Iceberg still force customers to manage object-level APIs, compaction jobs, and metadata services themselves.
- S3 Files aims to provide a filesystem-style interface on top of S3's scale and durability, eliminating manual copy scripts and unifying table-format intelligence under one system.