S3 Files and the changing face of S3

Open original

Andy Warfield recounts how genomic researchers struggled with moving large datasets between local filesystems and S3, spurring the creation of S3 Files to present S3 buckets as POSIX-compatible file stores. The post then covers how Amazon added S3 Tables for managed Iceberg tables and built-in vector indexing to cut storage friction and speed up analytics and search.

  • S3 Files was inspired by genomics researchers at UBC who wasted hours copying sequencing data between NFS and S3 because their GATK4/Spark pipeline still required POSIX filesystem semantics despite massive serverless parallelism.
  • Agent-driven development is lowering the barrier for domain experts to build apps, making it more critical to decouple persistent data from any single application or workflow.
  • S3 handles over 25 million Parquet requests daily, often via Hive tables, but open formats like Iceberg still force customers to manage object-level APIs, compaction jobs, and metadata services themselves.
  • S3 Files aims to provide a filesystem-style interface on top of S3's scale and durability, eliminating manual copy scripts and unifying table-format intelligence under one system.

Andy Warfield got the idea for S3 Files while helping genomics researchers at UBC move massive sequencing datasets around. Working with botany professor Loren Rieseberg’s sunflower DNA lab, he saw scientists waste hours copying files between NFS shares and S3. Their pipeline—containers called “bunnies” built by PhD student JS Legare—ran GATK4 on Spark across serverless compute, delivering huge parallelism. But every tool still expected a POSIX filesystem, so data spent more time in motion than in analysis. That storage boundary friction showed up again and again—from media pipelines to machine-learning pretraining—and it’s exactly the sort of problem Amazon’s storage team wants to solve. At the same time, Warfield points out, agent-driven development is slashing the cost and skill barrier of writing software. Domain experts in fields like finance or genomics can now spin up working apps without being expert coders. That accelerates the cycle from idea to prototype—and makes the division between application code and persistent data more critical. Data outlives every app; storage has to decouple data from any one tool or workflow while keeping access simple. The S3 team saw these trends and dug into customer feedback around structured data. Every day S3 handles over 25 million Parquet requests, often via Hive tables. Open table formats like Apache Iceberg added mutations, schema evolution and snapshots, but still forced customers to wrestle with object-level APIs, security policies, compaction jobs and metadata services. S3 Files aims to bridge that gap: provide a filesystem-style interface backed by S3’s scale and durability, eliminate manual copy scripts, and bring table-format intelligence under one roof.

Ask about this article