Click any tag below to further narrow down your results
Links
PostHog built its warehouse on DuckDB, then hit limits at scale and is now rebuilding it on a custom Trino engine with its own table format, HogLake. The post explains why DuckDB and DuckLake fell short for customers with very large, complex workloads.
- Single-node DuckDB was hard to run safely in the cloud because PostHog had to size each worker before queries arrived, and predicting CPU and memory per query was difficult.
- DuckLake's portable catalog schema limited optimizations: at PostHog's scale a single-table write could take over three minutes, and over 99% of catalog statistics rows belonged to deleted tables.
- At about 2 PB, roughly 1 trillion rows and 15.3 million snapshots, DuckLake broke in many ways, and PostHog ended up carrying patches in a fork instead of contributing upstream.
A data engineer runs a side-by-side test of Polars 2.1 and DuckDB 2.0 to see which handles typical analytical workloads faster. The video walks through the setup and the timing results, which is the part worth watching if you're choosing between the two for a pipeline.
- The comparison pits Polars 2.1 directly against DuckDB 2.0 on the same workloads, so the results are a head-to-head rather than vendor numbers.
- The headline claim is in the title, which promises surprising results; check the video for the actual timings and dataset sizes before drawing conclusions.
- The test setup determines how much the results transfer to your own data, so the dataset and hardware details matter more than the final ranking.
This is a free, searchable directory of 575 open-source projects and companies built on DuckDB, the embedded analytics database. It's rebuilt from the awesome-duckdb GitHub list and updated as of October 2026.
- The map covers 516 open-source projects and 59 companies that publicly say they build on DuckDB, each linked to its source.
- The most-starred projects include PyGWalker (16K stars), Kepler.gl (12.1K), and Lance (7.2K), and the top UIs are Preswald and Online DuckDB Shell.
- Snowflake's own Snowset data shows 96.2% of queries scanned under 10 GB, a size one DuckDB instance can handle, so many workloads could move off a warehouse.
turbopuffer is rebuilding its query engine so every query runs through a batch-oriented vectorized pipeline, borrowing an approach usually found in analytical databases like DuckDB. The team argues search engines have skipped this because they ran narrow ranking loops, but customer queries now combine filters, boosts, and aggregations that make operator dispatch a real cost.
- Volcano-style row-at-a-time execution wastes CPU on per-row function calls and dispatch; vectorized engines process cache-friendly batches to cut that overhead.
- Traditional search algorithms like block-max WAND and HNSW resist vectorization because each similarity result decides what to examine next, creating data dependencies the CPU can't predict.
- turbopuffer v1 had one similarity operator, but customers now stack filters, recency decay, aggregations, and custom scoring, so the query plans are heterogeneous enough to justify the switch.
A developer tested whether DuckDB could compete with Spark and Trino at large scales and found that a single powerful machine running DuckDB outperforms distributed clusters on 10TB queries—faster than Trino and half the cost of multi-node setups. The key insight: most time is spent waiting on S3 reads, not computation, so distribution overhead doesn't help.
- At 10TB scale, single-node DuckDB on an r7gd.4xlarge beats Trino on identical queries by 1.7-2.1x, and costs half as much as 5-13 node clusters despite being 2-3x slower than them
- S3 read rate, not compute speed, determines performance at scale—DuckDB's default single-threaded S3 reads leave CPUs 90% idle, but running 4x more threads than cores achieves 3x faster read rates
- DuckDB v2.0 (alpha) expects 2-3x faster S3 reads, potentially making single-node DuckDB the cheapest option for datasets up to 10TB
DuckDB 2.0 Alpha adds dynamic attach statements that let you pull credentials from environment variables instead of hardcoding them, solving a major friction point when working across multiple data lakes. Early production testing shows less than 1% failure rates, and the release bundles this with Quack and official DuckLake support to make local-first lakehouse workflows actually practical.
- Dynamic attach statements now accept environment variables for credentials (usernames, passwords, account numbers), eliminating the need to hardcode sensitive data or switch to Python for multi-catalog federation
- Production pilot running hourly jobs showed <1% failure rate over 10 days, reading Parquet from S3 and writing to Iceberg tables via RDS-DMS
- Release combines dynamic attach with Quack and official DuckLake support, completing the local-first lakehouse story but raising open questions about concurrent writer behavior on shared Iceberg tables
dbt v2, powered by a new Rust engine, ships with DuckDB support built-in instead of requiring a separate Python package. This lets data teams run dbt locally without external databases or warehouses.
- DuckDB adapter is now bundled with dbt v2 and automatically downloads its driver on first run, eliminating the need to install dbt-duckdb separately
- v2 adds catalog support for DuckLake and Iceberg, native SQL comprehension with column-level lineage, and stores metadata as queryable Parquet files instead of massive JSON manifests
- The Rust-based engine compiles to a single binary, making local development faster and enabling the VS Code extension to catch errors in the editor before queries hit the warehouse
The author shares a year of experience using AI for data work, arguing that copying someone else's workflow is pointless — what matters is learning specific techniques. He's settled on DuckDB CLI + Claude as his stack because it reduces hallucinations and keeps agents focused on actual tools instead of generating buggy code.
- AI workflows are creative, not formulaic — tips and tricks transfer better than full process replication, similar to how watching a music producer's exact steps won't let you recreate their song
- DuckDB CLI commands paired with AI agents dramatically cut hallucinations because agents understand the tool's actual capabilities rather than inventing Python code that doesn't work
- Working in a modern terminal (Ghostty) with multiple windows beats IDEs for analytics work — it's faster, gives you exactly the tools you need, and agents are already built to work with CLI tools
DuckDB released a plugin that lets Claude Code query data files directly using SQL instead of writing Python scripts, making it faster and more accurate. The plugin handles everything from local CSVs to remote cloud storage and spatial data, with Claude automatically picking the right tool based on what you ask.
- Claude can now run SQL queries against any file format (CSV, Parquet, JSON, Excel, etc.) on disk or in cloud storage, getting exact answers instead of guessing column names
- The plugin includes specialized skills for S3/GCS exploration, spatial queries (distances, nearest neighbors), searching DuckDB docs, and recalling decisions from past sessions
- When a query fails, Claude reads the error message and retries with corrected SQL automatically, creating a two-layer conversation (natural language with you, SQL with DuckDB)
xlDuckDb is an Excel add-in that lets you write DuckDB SQL queries directly in spreadsheet cells and get results back as normal Excel data. You can query Excel ranges, JSON files, CSV files, Parquet files, and even remote data on HTTPS or AWS S3.
- The add-in works with Excel ranges, named ranges, and tables using simple syntax like =DuckDbQuery(A27,,A1:E25), plus you can query multiple ranges in a single query by passing them as separate arguments.
- DuckDB handles messy data formats automatically—it auto-detects CSV delimiters and data types, extracts nested JSON with JSONPath syntax, and efficiently queries large Parquet files that don't fit in memory.
- You can join data across multiple file types in one SQL query (e.g., combining JSON and CSV data with a CTE and INNER JOIN), making it possible to correlate disparate datasets without leaving Excel.
AWS acquired DuckLabs—the team behind the fast-growing DuckDB analytics database—to control the roadmap of a key data infrastructure technology and build an S3-powered alternative to Databricks and Snowflake. The deal doesn't include the open-source DuckDB foundation or MotherDuck, just the core engineering team and their vision.
- DuckDB, DuckLake, and Quack form a free, composable data stack that runs on AWS's S3 storage; AWS profits from increased compute and storage usage even when customers skip the vendor tax of proprietary platforms.
- AI agents need lightweight, embeddable databases for managing state and telemetry at scale, and DuckDB's architecture—small, fast-starting, process-embedded—makes it ideal for this emerging workload.
- AWS likely paid hundreds of millions or close to $1 billion, similar to Databricks's $1B+ acquisition of Tabular; the value lies in Hannes Mühleisen and Mark Raasveldt's vision and ability to shape DuckDB's evolution toward becoming a general-purpose distributed database.
AWS bought DuckLabs (the team behind DuckDB) to control the roadmap of a fast-growing analytics database that's becoming foundational to modern data infrastructure. The deal is really about acquiring talented engineers and influence over DuckDB's evolution toward becoming a distributed, server-based system that competes with Databricks and Snowflake.
- DuckDB + DuckLake + Quack form an open, S3-powered data stack that lets AWS compete with Databricks and Snowflake while making money on cloud primitives (compute, storage, networking) rather than vendor markup.
- DuckDB is well-suited for AI agents because it's lightweight, embeddable, starts instantly, and handles analytics on telemetry streams—exactly what agents need for self-optimization and parallel task exploration.
- AWS likely paid hundreds of millions to over $1 billion (comparable to Databricks' $1B+ Tabular deal) because even modest improvements to AWS's competitive position in data infrastructure justify massive payouts.
DuckLabs, the company behind the popular DuckDB analytics database, is being acquired by AWS in early September. The core open-source projects will remain free under the MIT license and governed by the nonprofit DuckDB Foundation, but the team gains AWS's infrastructure and reach to scale the technology further.
- DuckDB hits over 1 million downloads daily and the founders realized their bootstrapped model couldn't support the project's growth trajectory without becoming a bottleneck
- All core Duck Stack components (DuckDB, DuckLake, Quack) stay open-source under MIT license with the DuckDB Foundation maintaining stewardship
- AWS has committed long-term support and the DuckLabs team of 30+ people will remain together in Amsterdam, focusing on technical work rather than sales operations
This newsletter covers major developments across data and AI, including DoorDash’s new Write-Ahead Intent Log for scalable CDC, OpenAI’s Kepler AI analyst for massive data insights, and Lyft’s Metric Semantic Layer for unified business metrics. It also reviews ClickHouse ingestion optimizations, DuckDB’s latest release and agent use cases, Databricks’ Lakehouse//RT and LTAP pushes, plus tools for self-healing pipelines and data quality monitoring.
- DoorDash replaced Debezium with a Write-Ahead Intent Log (WAIL) that writes mutation intents to Kafka and the DB, enabling faster recovery and much higher CDC throughput at Cassandra scale
- MotherDuck claims a $2.40/hour DuckDB instance beats a $64/hour Snowflake 2XL cluster by ~5x on ClickBench, with median query latencies of 3ms, positioning DuckDB for fleets of lightweight agent workloads
- Lyft's Metric Semantic Layer assigns dual business/operational ownership to "Golden Metrics" defined in versioned JSON, auto-propagating changes across dashboards, APIs, and AI agents to prevent divergent calculations
- OpenAI's Kepler analyst now operates over 600 PB of data using Codex-based crawlers and AST-normalized LLM grading to infer lineage, freshness, and hidden semantics automatically
The author shares their shift from using Excel and Google Sheets to DuckDB and SQL for handling CSV files, highlighting the efficiency of querying data directly. They discuss the benefits of using SQL for data manipulation and invite readers to share their own CSV handling tips.
- DuckDB lets you run SQL directly on CSV files (joins, window functions, dedup queries) without importing them into Excel/Sheets first
- You can query CSVs from URLs or cloud storage like S3 directly, and export results straight back to CSV
- Several users find SQL queries more intuitive than replicating the same logic with spreadsheet formulas or pivot tables
- Some commenters mix in Parquet files and other extensions alongside DuckDB to further speed up ad-hoc analysis
DuckDB has proven to be superior to Polars when handling large datasets, particularly 1TB of data. While DuckDB effectively manages memory and execution with a robust design, Polars struggles with large data processing, leading to out-of-memory errors.
- DuckDB handled the 1TB benchmark successfully while Polars crashed with out-of-memory errors
- DuckDB's memory management and execution engine scale more effectively to large datasets than Polars'
- Polars is optimized more for in-memory, smaller-scale workloads and breaks down at larger scales like 1TB