More on the topic…
Andy Warfield got the idea for S3 Files while helping genomics researchers at UBC move massive sequencing datasets around. Working with botany professor Loren Rieseberg’s sunflower DNA lab, he saw scientists waste hours copying files between NFS shares and S3. Their pipeline—containers called “bunnies” built by PhD student JS Legare—ran GATK4 on Spark across serverless compute, delivering huge parallelism. But every tool still expected a POSIX filesystem, so data spent more time in motion than in analysis. That storage boundary friction showed up again and again—from media pipelines to machine-learning pretraining—and it’s exactly the sort of problem Amazon’s storage team wants to solve.
At the same time, Warfield points out, agent-driven development is slashing the cost and skill barrier of writing software. Domain experts in fields like finance or genomics can now spin up working apps without being expert coders. That accelerates the cycle from idea to prototype—and makes the division between application code and persistent data more critical. Data outlives every app; storage has to decouple data from any one tool or workflow while keeping access simple.
The S3 team saw these trends and dug into customer feedback around structured data. Every day S3 handles over 25 million Parquet requests, often via Hive tables. Open table formats like Apache Iceberg added mutations, schema evolution and snapshots, but still forced customers to wrestle with object-level APIs, security policies, compaction jobs and metadata services. S3 Files aims to bridge that gap: provide a filesystem-style interface backed by S3’s scale and durability, eliminate manual copy scripts, and bring table-format intelligence under one roof.
Questions about this article
No questions yet.