The author shares a year of experience using AI for data work, arguing that copying someone else's workflow is pointless — what matters is learning specific techniques. He's settled on DuckDB CLI + Claude as his stack because it reduces hallucinations and keeps agents focused on actual tools instead of generating buggy code.

• AI workflows are creative, not formulaic — tips and tricks transfer better than full process replication, similar to how watching a music producer's exact steps won't let you recreate their song
• DuckDB CLI commands paired with AI agents dramatically cut hallucinations because agents understand the tool's actual capabilities rather than inventing Python code that doesn't work
• Working in a modern terminal (Ghostty) with multiple windows beats IDEs for analytics work — it's faster, gives you exactly the tools you need, and agents are already built to work with CLI tools

DuckDB released a plugin that lets Claude Code query data files directly using SQL instead of writing Python scripts, making it faster and more accurate. The plugin handles everything from local CSVs to remote cloud storage and spatial data, with Claude automatically picking the right tool based on what you ask.

• Claude can now run SQL queries against any file format (CSV, Parquet, JSON, Excel, etc.) on disk or in cloud storage, getting exact answers instead of guessing column names
• The plugin includes specialized skills for S3/GCS exploration, spatial queries (distances, nearest neighbors), searching DuckDB docs, and recalling decisions from past sessions
• When a query fails, Claude reads the error message and retries with corrected SQL automatically, creating a two-layer conversation (natural language with you, SQL with DuckDB)

DuckDB 2.0 ships three major performance improvements: async I/O for S3 queries (2-3x faster), recursive CTEs for deep hierarchies (40x+ faster), and a new VARIANT type for JSON data (6x faster queries, 2.7x smaller storage). The author benchmarked each on real datasets to show which workloads actually benefit.

• Async I/O decouples network downloads from CPU work via a separate thread pool, cutting S3 query times from 18.8s to 7.7s on a 2.2GB file with zero query changes and automatic enablement.
• Recursive CTEs now build a lookup index once instead of re-reading the entire table per recursion level, dropping a 20,000-commit ancestry walk from 1.8-16s to 0.10s consistently.
• VARIANT type "shreds" JSON into separate columns for consistent fields (level, service, latency_ms) while keeping messy fields in binary storage, delivering 6x faster field queries and 2.7x smaller on-disk size than plain JSON strings.

xlDuckDb is an Excel add-in that lets you write DuckDB SQL queries directly in spreadsheet cells and get results back as normal Excel data. You can query Excel ranges, JSON files, CSV files, Parquet files, and even remote data on HTTPS or AWS S3.

• The add-in works with Excel ranges, named ranges, and tables using simple syntax like =DuckDbQuery(A27,,A1:E25), plus you can query multiple ranges in a single query by passing them as separate arguments.
• DuckDB handles messy data formats automatically—it auto-detects CSV delimiters and data types, extracts nested JSON with JSONPath syntax, and efficiently queries large Parquet files that don't fit in memory.
• You can join data across multiple file types in one SQL query (e.g., combining JSON and CSV data with a CTE and INNER JOIN), making it possible to correlate disparate datasets without leaving Excel.

AWS acquired DuckLabs—the team behind the fast-growing DuckDB analytics database—to control the roadmap of a key data infrastructure technology and build an S3-powered alternative to Databricks and Snowflake. The deal doesn't include the open-source DuckDB foundation or MotherDuck, just the core engineering team and their vision.

• DuckDB, DuckLake, and Quack form a free, composable data stack that runs on AWS's S3 storage; AWS profits from increased compute and storage usage even when customers skip the vendor tax of proprietary platforms.
• AI agents need lightweight, embeddable databases for managing state and telemetry at scale, and DuckDB's architecture—small, fast-starting, process-embedded—makes it ideal for this emerging workload.
• AWS likely paid hundreds of millions or close to $1 billion, similar to Databricks's $1B+ acquisition of Tabular; the value lies in Hannes Mühleisen and Mark Raasveldt's vision and ability to shape DuckDB's evolution toward becoming a general-purpose distributed database.

AWS bought DuckLabs (the team behind DuckDB) to control the roadmap of a fast-growing analytics database that's becoming foundational to modern data infrastructure. The deal is really about acquiring talented engineers and influence over DuckDB's evolution toward becoming a distributed, server-based system that competes with Databricks and Snowflake.

• DuckDB + DuckLake + Quack form an open, S3-powered data stack that lets AWS compete with Databricks and Snowflake while making money on cloud primitives (compute, storage, networking) rather than vendor markup.
• DuckDB is well-suited for AI agents because it's lightweight, embeddable, starts instantly, and handles analytics on telemetry streams—exactly what agents need for self-optimization and parallel task exploration.
• AWS likely paid hundreds of millions to over $1 billion (comparable to Databricks' $1B+ Tabular deal) because even modest improvements to AWS's competitive position in data infrastructure justify massive payouts.

DuckLabs, the company behind the popular DuckDB analytics database, is being acquired by AWS in early September. The core open-source projects will remain free under the MIT license and governed by the nonprofit DuckDB Foundation, but the team gains AWS's infrastructure and reach to scale the technology further.

• DuckDB hits over 1 million downloads daily and the founders realized their bootstrapped model couldn't support the project's growth trajectory without becoming a bottleneck
• All core Duck Stack components (DuckDB, DuckLake, Quack) stay open-source under MIT license with the DuckDB Foundation maintaining stewardship
• AWS has committed long-term support and the DuckLabs team of 30+ people will remain together in Amsterdam, focusing on technical work rather than sales operations

This newsletter covers major developments across data and AI, including DoorDash’s new Write-Ahead Intent Log for scalable CDC, OpenAI’s Kepler AI analyst for massive data insights, and Lyft’s Metric Semantic Layer for unified business metrics. It also reviews ClickHouse ingestion optimizations, DuckDB’s latest release and agent use cases, Databricks’ Lakehouse//RT and LTAP pushes, plus tools for self-healing pipelines and data quality monitoring.

• DoorDash replaced Debezium with a Write-Ahead Intent Log (WAIL) that writes mutation intents to Kafka and the DB, enabling faster recovery and much higher CDC throughput at Cassandra scale
• MotherDuck claims a $2.40/hour DuckDB instance beats a $64/hour Snowflake 2XL cluster by ~5x on ClickBench, with median query latencies of 3ms, positioning DuckDB for fleets of lightweight agent workloads
• Lyft's Metric Semantic Layer assigns dual business/operational ownership to "Golden Metrics" defined in versioned JSON, auto-propagating changes across dashboards, APIs, and AI agents to prevent divergent calculations
• OpenAI's Kepler analyst now operates over 600 PB of data using Codex-based crawlers and AST-normalized LLM grading to infer lineage, freshness, and hidden semantics automatically

Rill lets you define BI workflows—models, metrics, dashboards and access policies—in YAML and SQL, and runs queries on engines like ClickHouse or DuckDB. It adds conversational and agent-driven interfaces for natural-language querying and automated dashboard generation, with real-time performance, embeddable dashboards, and Git-backed deployments.

• Full BI stack (models, metrics, dashboards, agent config) spins up from three prompts via `rill init`, then `rill start` gives an instant local UI.
• Metrics-as-code approach: SQL models plus YAML-defined dimensions/measures let Rill generate queries on the fly, returning sub-second results even at billions of rows.
• Same semantic layer powers both natural-language conversational BI and traditional SQL-based analyst workflows.
• Deployment is Git/CI-CD native via `rill deploy`, with dashboards embeddable and metrics exposed through a REST API.

The author shares their shift from using Excel and Google Sheets to DuckDB and SQL for handling CSV files, highlighting the efficiency of querying data directly. They discuss the benefits of using SQL for data manipulation and invite readers to share their own CSV handling tips.

• DuckDB lets you run SQL directly on CSV files (joins, window functions, dedup queries) without importing them into Excel/Sheets first
• You can query CSVs from URLs or cloud storage like S3 directly, and export results straight back to CSV
• Several users find SQL queries more intuitive than replicating the same logic with spreadsheet formulas or pivot tables
• Some commenters mix in Parquet files and other extensions alongside DuckDB to further speed up ad-hoc analysis

DuckDB has proven to be superior to Polars when handling large datasets, particularly 1TB of data. While DuckDB effectively manages memory and execution with a robust design, Polars struggles with large data processing, leading to out-of-memory errors.

• DuckDB handled the 1TB benchmark successfully while Polars crashed with out-of-memory errors
• DuckDB's memory management and execution engine scale more effectively to large datasets than Polars'
• Polars is optimized more for in-memory, smaller-scale workloads and breaks down at larger scales like 1TB