Databricks & the Future of the Firm
This a16z piece argues Databricks is quietly building the data infrastructure that agentic AI will need to run inside real companies, tracing its path from Spark to the lakehouse to Lakebase. It ties that infrastructure to a bigger claim: that front-line workers will regain decision-making power from the managerial hierarchy that dominated the 20th century.
- Databricks grew from Spark, an open-source engine Matei Zaharia wrote at Berkeley that kept data in memory and ran iterative workloads up to ten times faster than Hadoop.
- The "lakehouse" merged cheap data lake storage with warehouse reliability, and the company says its Lakehouse runs at a more than $1.5 billion revenue run-rate, growing over 100% year over year.
- Databricks bought Neon in 2025 to build Lakebase, which separates storage from compute and removes the pipelines between operational databases and analytics, a setup the author says agents will depend on.
Databricks started as a Spark company. Ali Ghodsi, Matei Zaharia and five other Berkeley researchers founded it in 2013 to wrap the distributed compute engine Spark in a managed platform, since Spark itself was roughly ten times faster than Hadoop for iterative workloads by keeping data in memory between rounds. The bigger shift came from the split between data warehouses, which were trustworthy but built for SQL and dashboards, and data lakes, which were cheap but became "data swamps" with nothing checked on the way in. Around 2017 to 2019, Databricks, Uber and Netflix independently built table formats (Delta Lake, Hudi, Iceberg) that added warehouse guarantees like all-or-nothing updates and schema management on top of lake storage. Databricks then productized this as the "lakehouse" in 2020, letting engineers, data scientists and analysts work on one copy of the data.
The article's argument is that the next layer, the "Lakebase," and the agent wave make this architecture matter more than it used to. The author's core claim is that what people actually do at work is constrained by the system of record a company uses, and that system is in turn constrained by its database technology. Agents need a system of record they can read, write and reason over reliably, and Databricks says it now sees more queries from agents than from people, with over 17 million databases spun up by agents each day. The author expects agentic automation and frontline employees taking initiative to reshape work, and frames the outcome less as "everyone is an engineer" and more as "everyone is a manager," a reversal of the centralized managerial model Burnham described in 1941.
The business backdrop is a company with a reported $7 billion revenue run rate growing 80% year over year, which the author calls one of the most under-the-radar stories in Silicon Valley despite its scale. The article is a promotional piece from a16z, an investor in Databricks, so its claims about the future of the firm are advocacy as much as analysis, and the specific growth and agent-traffic figures are the company's and the author's, not independently verified here. The useful takeaway is the architectural argument: the bottleneck on how work gets done is often the data layer, and whoever owns that layer shapes what agents can do inside a company.