7 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
This article details the author's experience creating an object store called blobd, optimized for speed with sub-millisecond read times and high upload rates. It discusses design choices, including using a hash-based index and direct I/O to bypass traditional filesystems. The open-source project aims to enhance performance for small object storage.
If you do, here's more
The author shares their experience building "blobd," a single-machine object store optimized for random reads and small object retrievals. The project leverages NVMe disk capabilities, focusing on achieving sub-millisecond response times for reads and impressive upload speeds. Key innovations include using io_uring, async Rust, and atomic writes. By avoiding tree-based data structures and listing keys, the design allows for constant-time lookups, which is critical given the performance demands of user content delivery.
In the initial design, the author created a C implementation that utilized raw block devices without a filesystem, featuring a heap for storing objects, an index for key mapping, and a journal for state management. They opted for a hash map index over a tree structure to streamline lookups, aiming for O(1) performance. The article details the allocation strategy, where the heap is divided into 16 MB tiles, with smaller allocations handled by converting free tiles into "microtiles." The allocation state is tracked using a hierarchical bitmap, allowing efficient management of free and used tiles.
The author meticulously explains the locking mechanisms for read and write operations, ensuring thread safety without excessive overhead. This approach minimizes latency during reads while maintaining data integrity. The article includes performance comparisons with existing solutions like MinIO and XFS, showcasing blobd's superiority in read times across various object sizes. The project is open source, inviting others to explore the code and learn from the implementation.
Questions about this article
No questions yet.