S3 Is the Future, S3 Is the Past

Open original

S3 became the default storage layer for cloud systems because of disk constraints 20 years ago, but SSDs and modern networks have made those constraints irrelevant. Cloud providers won't build a faster replacement because they profit from the status quo, so someone else needs to.

  • S3's architecture is fundamentally a workaround for slow disk performance (100ms latency, <100MB/s per request), forcing every system built on it to add caching layers, batching logic, and separate metadata stores.
  • SSDs are now only 3× more expensive than disks while offering microsecond latency, millions of IOPS, and small random access—eliminating the need for most S3-based workarounds.
  • Amazon's S3 Express One Zone proves the company won't cannibalize its disk-based S3 revenue stream with a true SSD replacement, leaving the door open for competitors to build it.

S3 dominates cloud architecture because it solves real problems: infinite capacity, high durability, cheap storage per gigabyte, and a shared namespace that lets compute stay stateless. The entire modern data stack—Iceberg, Parquet, Kafka alternatives, vector databases—is built on top of it. But S3's design reflects 20-year-old hardware constraints. It's fundamentally a million hard disks behind an HTTP API, delivering less than 100 MB/s per request with tens of milliseconds of latency. Every serious system built on S3 has to work around these limitations: you need caching layers to hide latency, you batch small writes into large objects to amortize overhead, you maintain separate metadata stores because S3 can't handle small random lookups efficiently, and you can't update records in-place. The hardware assumptions have become obsolete. SSD prices have dropped consistently over two decades, narrowing the cost gap with hard disks to roughly 3×. SSDs offer access latency around 100 microseconds (200 times faster than S3), millions of I/O operations per second, and efficient 4 KB random read/writes. Datacenter networks now deliver 100+ Gbit links with sub-100-microsecond latency within a zone. An SSD-based storage system coupled to modern networks could eliminate most of S3's architectural tax: caching becomes optional, batching unnecessary, in-place updates possible, and you can unify metadata and data storage. Amazon released S3 Express One Zone in 2023 as an SSD option, but it's crippled by design—restricted to a single availability zone, still multi-millisecond latency, higher per-gigabyte costs, and steep bandwidth pricing. It's a niche product, not a replacement. The real obstacle isn't technology. Fast SSDs exist. Fast networks exist. Cloud providers simply have no incentive to cannibalize their hard disk inventory and the revenue streams built on current S3 pricing. We're now standardizing architectural patterns designed to work around hardware limitations that no longer exist, and we're locking ourselves into these patterns at exactly the wrong moment. The author argues we shouldn't wait for cloud providers to lead—we should build the infrastructure ourselves.

Ask about this article