This article discusses a system built for Wayfair that uses PostgreSQL as a Dead Letter Queue (DLQ) to manage failed event processing. Instead of using Kafka for failed events, the system stores them in a PostgreSQL table, allowing for better visibility and easier reprocessing. It also outlines a retry mechanism with exponential backoff to prevent flooding the DLQ with transient failures.

AWS Lambda now offers low latency processing for Kafka events, allowing sub-100ms event handling for Amazon MSK and self-managed Apache Kafka in Provisioned mode. By setting the MaximumBatchingWindowInSeconds parameter to 0, customers can achieve real-time processing, making it suitable for mission-critical applications across various industries. This feature is available in most AWS regions, enhancing the efficiency of latency-sensitive applications.

Klaviyo successfully migrated its event processing pipeline from RabbitMQ to a Kafka-based architecture, handling up to 170,000 events per second while ensuring zero data loss and minimal impact on ongoing operations. The new system enhances performance, scales for future growth, and improves operational efficiency, positioning Klaviyo to meet the demands of over 176,000 businesses worldwide. Key design principles focused on decoupling ingestion from processing, eliminating blocking issues, and ensuring reliability in the face of transient failures.