Click any tag below to further narrow down your results
Links
LinkedIn built Northguard and Xinfra to replace Kafka's core design after hitting limits at 32 trillion records per day. The new system decouples logs from replication units, distributes metadata across independent state machines, and lets new brokers serve traffic without copying historical data.
- Kafka's partition couples ordering, replication, and consumer parallelism into one heavyweight unit; moving or repairing partitions at LinkedIn's scale wastes network and disk bandwidth. Northguard uses smaller segments with independent replica assignments, so adding capacity doesn't require copying old data.
- The centralized metadata controller became a bottleneck managing millions of partition replicas. Northguard shards metadata across 128+ coordinators using a hash ring, with each vnode managing its own Raft state machine for topic, range, and segment metadata.
- Northguard can enforce fsync on all replicas before acknowledging writes while meeting LinkedIn's performance SLOs—a durability guarantee that Kafka's default configuration doesn't require because it relies on replication instead.
Set up the Tansu broker using SQLite on an AWS t3.micro instance, taking advantage of the AWS Free Tier. The guide covers installation, configuration, and performance testing, demonstrating how to launch a low-cost Kafka-compatible broker for early-stage projects.
- Tansu with embedded SQLite runs a full Kafka-compatible broker on a free t3.micro (1GiB RAM), storing everything in a single tansu.db file for trivial backup/restore.
- Despite minimal hardware (~10MB/s EBS baseline throughput), the setup delivers usable, measurable throughput and latency, proving Kafka-compatible workloads are feasible on free-tier resources.
- Running below the t3 baseline CPU utilization accumulates burst credits, letting the broker handle demand spikes without extra cost.
- Tansu itself is lightweight on memory, leaving most of the 1GiB instance free for other system needs.