6 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
This article details LinkedIn's transition from Zookeeper to a new scalable service discovery system designed to handle the demands of a growing number of microservices. The new system, which uses Kafka and a Service Discovery Observer, improves scalability, compatibility, and extensibility while supporting multiple programming languages.
If you do, here's more
LinkedIn has developed a new multi-language service discovery system to address limitations in its previous architecture, which relied on Zookeeper. For years, Zookeeper served as the control plane, allowing services to register their endpoint addresses and handle inbound and outbound traffic. However, issues arose as LinkedIn scaled up its operations, leading to significant read/write latencies and session closures. As of July 2022, Zookeeper faced an estimated 2.5 years of capacity left, with anticipated growth in cluster size and watch requests at 50-100% yearly. The reliance on custom schemas for D2 entities also presented compatibility problems with modern frameworks like gRPC.
The new Next-Gen Service Discovery system shifts to a model based on Kafka and a component called the Service Discovery Observer. This setup separates read and write requests, enhancing scalability and availability. The Observer can maintain 40,000 client streams and process 10,000 updates per second while consuming 11,000 Kafka events, with projections of fleet growth requiring around 100 Observers. The system sacrifices strict consistency for high availability, allowing for some short-term data inconsistency, which is acceptable in service discovery. Compatibility with gRPC and extensibility for advanced features, like centralized load balancing and security policies, are now possible due to the xDS protocol integration.
By centralizing the control plane, LinkedIn can better manage its massive network of services. The new architecture not only resolves scalability issues but also opens doors for modern service mesh capabilities. This transformation positions LinkedIn to handle the increasing complexity of its services while maintaining fast and reliable performance for its users.
Questions about this article
No questions yet.