1 link tagged with all of: rust + kafka-alternative + logging + object-storage
Click any tag below to further narrow down your results
Links
OpenData Log is an MIT-licensed Rust service that stores ordered logs directly on object storage with per-key indexing. It handles millions of individual logs efficiently by using a segmented LSM tree keyed by (key, sequence), which cuts read amplification compared to Kafka. Readers can scale by key-range without repartitioning or data reshuffling.
- Kafka's per-partition design makes "router" workloads (millions of small, addressable logs) expensive—read amplification can hit partition-size/record-size just to find a few records.
- OpenData Log replaces topic-partitions with a segmented LSM tree keyed by (key, sequence), enabling millions of independent logs per node with efficient per-key scans.
- Range-partitioned keys let you reshard (split/merge) without moving consumer offsets, and a bad key's issues stay isolated instead of stalling the whole log.
- Benchmarks on an m5n.xlarge against S3 show stable tail latencies even at millions of keys, claiming no loss of Kafka's durability or throughput.