Click any tag below to further narrow down your results
Links
Databricks, Snowflake, Google, Amazon, and Microsoft have all built AI-facing products on PostgreSQL, and this piece explains why the open license, extension model, and developer familiarity made it the shared choice. It also warns that "Postgres" now covers four different levels of compatibility, so buyers need to check which one they're getting.
- Databricks paid around $1 billion for Neon and Snowflake paid around $165 million for Crunchy Data, showing the premium the market puts on real Postgres compatibility plus modern architecture.
- Extensions like pgvector, pg_lake, and pg_duckdb turned Postgres into a single engine for state, vector retrieval, and analytics, which suits most AI workloads below very large scale.
- "Postgres compatible" ranges from the real engine to a wire protocol over a different database, and the extensions you rely on determine which layer you need and what leaving will cost.
PostHog built its warehouse on DuckDB, then hit limits at scale and is now rebuilding it on a custom Trino engine with its own table format, HogLake. The post explains why DuckDB and DuckLake fell short for customers with very large, complex workloads.
- Single-node DuckDB was hard to run safely in the cloud because PostHog had to size each worker before queries arrived, and predicting CPU and memory per query was difficult.
- DuckLake's portable catalog schema limited optimizations: at PostHog's scale a single-table write could take over three minutes, and over 99% of catalog statistics rows belonged to deleted tables.
- At about 2 PB, roughly 1 trillion rows and 15.3 million snapshots, DuckLake broke in many ways, and PostHog ended up carrying patches in a fork instead of contributing upstream.
This a16z piece argues Databricks is quietly building the data infrastructure that agentic AI will need to run inside real companies, tracing its path from Spark to the lakehouse to Lakebase. It ties that infrastructure to a bigger claim: that front-line workers will regain decision-making power from the managerial hierarchy that dominated the 20th century.
- Databricks grew from Spark, an open-source engine Matei Zaharia wrote at Berkeley that kept data in memory and ran iterative workloads up to ten times faster than Hadoop.
- The "lakehouse" merged cheap data lake storage with warehouse reliability, and the company says its Lakehouse runs at a more than $1.5 billion revenue run-rate, growing over 100% year over year.
- Databricks bought Neon in 2025 to build Lakebase, which separates storage from compute and removes the pipelines between operational databases and analytics, a setup the author says agents will depend on.
Streamhouse is an architecture that organizes data workloads around shared, reusable tables rather than independently maintained copies. It unifies streaming and historical data through Lakestream, a storage foundation that treats fresh and old data as different layers of a single logical table.
- Most streaming systems duplicate data across multiple destinations, each maintaining its own copy with separate ingestion tracking and schema propagation. Streamhouse eliminates this by making tables platform resources with defined ownership and lifecycle that workloads share through supported APIs.
- The shift from topic-centric (event movement) to table-centric (shared datasets) architecture means the platform itself understands schemas, primary keys, and row semantics instead of requiring each consumer to independently interpret the same event stream into a usable form.
- Lakestream coordinates streaming storage (optimized for continuous fresh changes) and lakehouse storage (optimized for scans and retention) as layers of one logical table, so queries can access either committed historical data or union reads across both layers without duplicating data.
This article walks through setting up a future-proof data stack by outsourcing ingestion to turnkey tools, sticking with SQL-based transforms, and carefully integrating AI without ignoring fundamentals. It also covers key areas like data quality, storage and compute choices, and when to move from replicas to a full warehouse or lakehouse.
- Ingestion is a solved problem now—use managed connectors (Estuary, Fivetran, Glue, DLT) instead of building custom pipelines unless there's truly no supported option.
- SQL remains the backbone of transformation work regardless of tool (dbt, stored procedures, Airflow), with AI only starting to help via auto-generated schemas and template joins, not replacing hands-on cleanup.
- Flashy AI chatbots/analytics tools often demand as much setup as traditional dashboards, so lightweight CLI or skills-file integrations with tools like Claude or Codex are a more practical automation win.
- Don't over-architect storage/compute—a read replica and plain SQL can beat a lakehouse or warehouse if your team size and actual workload don't justify the complexity.