This article details LinkedIn's transition from Zookeeper to a new scalable service discovery system designed to handle the demands of a growing number of microservices. The new system, which uses Kafka and a Service Discovery Observer, improves scalability, compatibility, and extensibility while supporting multiple programming languages.

Klaviyo successfully migrated its event processing pipeline from RabbitMQ to a Kafka-based architecture, handling up to 170,000 events per second while ensuring zero data loss and minimal impact on ongoing operations. The new system enhances performance, scales for future growth, and improves operational efficiency, positioning Klaviyo to meet the demands of over 176,000 businesses worldwide. Key design principles focused on decoupling ingestion from processing, eliminating blocking issues, and ensuring reliability in the face of transient failures.