Northguard and Xinfra: What LinkedIn Changed When Kafka Was No Longer Enough
Short summary
LinkedIn built Northguard and Xinfra to replace Kafka's core design after hitting limits at 32 trillion records per day. The new system decouples logs from replication units, distributes metadata across independent state machines, and lets new brokers serve traffic without copying historical data.
- Kafka's partition couples ordering, replication, and consumer parallelism into one heavyweight unit; moving or repairing partitions at LinkedIn's scale wastes network and disk bandwidth. Northguard uses smaller segments with independent replica assignments, so adding capacity doesn't require copying old data.
- The centralized metadata controller became a bottleneck managing millions of partition replicas. Northguard shards metadata across 128+ coordinators using a hash ring, with each vnode managing its own Raft state machine for topic, range, and segment metadata.
- Northguard can enforce fsync on all replicas before acknowledging writes while meeting LinkedIn's performance SLOs—a durability guarantee that Kafka's default configuration doesn't require because it relies on replication instead.
Full summary
LinkedIn's Kafka infrastructure hit a wall. By 2025, they were running 32 trillion records and 17 petabytes per day across 150 clusters, and the system's fundamental design choices became liabilities. Kafka couples too many responsibilities into a single partition: it's an ordered log, a replication unit, a broker placement boundary, and a consumer workload divider all at once. This tight coupling meant that moving or repairing a partition affected huge amounts of data, and adding capacity required massive reassignment jobs that competed with production traffic. LinkedIn's response was Northguard, a new log storage layer, paired with Xinfra, a virtualization layer that lets applications use both systems transparently.
Northguard decouples the log from replication by introducing ranges (logical ordered logs) and segments (immutable chunks within those ranges). The segment becomes the replication unit instead of the partition, and different segments in the same range can have different replica sets—what LinkedIn calls log striping. When you add a broker, Northguard includes it in replica sets for new segments without copying old data over first. This means new capacity starts handling traffic immediately while historical data gradually redistributes as segments turn over. Failure recovery also splits into two paths: if a broker fails, the system seals the active segment and creates a new one with a different replica set, letting writes continue while old data repairs in the background.
The control plane got a complete overhaul too. Kafka's centralized controller became a bottleneck at this scale, so Northguard replaced it with a Dynamically-Sharded Replicated State Machine—128 or more independent coordinators, each managing a slice of metadata assigned by hash ring. Gossip handles membership and connection info, while Raft-backed vnodes hold authoritative topic and segment metadata. On the storage side, Northguard replaced Kafka's reliance on OS page caching with Direct I/O, a write-ahead log, and an application-managed cache, giving LinkedIn explicit control over what stays in memory. Important caveat: Northguard and Xinfra were closed-source at the time of the June 2025 report, and LinkedIn only said they'd consider open-sourcing them later.