Zalando Engineering Blog - Client-Side Load Balancing at a Million Requests Per Second

Open original

Zalando replaced shared edge routing for high-fan-out internal calls with an in-process load balancer that mirrors Skipper’s consistent-hash ring. They built a standalone JVM module for hash-parity routing, switched to a watch-based Kubernetes informer, and overhauled their CI/CD pipeline to safely roll out traffic ramps. The change cut latency spikes, reduced costs, and boosted deployment velocity.

  • Replacing shared Skipper routing with an in-process client-side load balancer for internal fan-out calls eliminated latency spikes and noisy-neighbor ambiguity from shared infrastructure.
  • The CSLB exactly replicates Skipper's consistent-hash ring (xxHash64, 100 virtual nodes) so both routers map product IDs identically, verified via unit tests and canary cache-hit parity.
  • A Kubernetes informer with a two-second debounce coalesces pod churn into single ring updates, avoiding control-plane overload while keeping routing current.
  • A refactored CI/CD pipeline cut build times from 21 to 12 minutes and removed 40+ manual steps, enabling faster iteration and safer traffic ramp-ups.

Zalando’s Product Read API used Skipper—an open-source Kubernetes ingress controller—for both edge traffic and internal fan-out calls. That meant every batch request, which splits into up to 100 parallel single-product calls, went through Skipper. Even though Skipper added only a couple hundred microseconds per hop, a batch’s latency was tied to its slowest call. On top of that, Skipper was shared infrastructure, so any hiccup in the cluster could look like PRAPI trouble. To regain control and clear that ambiguity, the team built an in-process client-side load balancer (CSLB) that replaces Skipper only for internal fan-out traffic. They replicated Skipper’s consistent-hash ring exactly, using xxHash64 and 100 virtual nodes, so both routers map product IDs to the same pods. Unit tests enforce parity—and canaries confirmed identical cache-hit ratios. The CSLB lives as a small JVM module with zero-allocation hashing, occupancy accounting, and bounded-load math, and it uses a Kubernetes informer to watch EndpointSlices. A two-second debounce coalesces pod churn into single ring updates, avoiding control-plane overload. Underneath it all, a refactored CI/CD pipeline cut build times from 21 to 12 minutes and removed 40-plus manual steps, making rapid, iterative improvements possible.

Ask about this article