Netflix’s New Batch Compute ⚙️, Fix Data Early 🪟, Finding Topics Automatically 🔍
This roundup covers Netflix’s switch to Kueue for Kubernetes-native batch compute, an engineer’s workflow for long-running coding agents, and Zalando’s in-process client load balancer handling over a million requests per second. It also explains Zepto’s dual-sequence re-ranker for real-time personalization, strategies for catching data issues early, why technically strong teams still miss business impact, and a new storage/workload architecture taxonomy—plus a Databricks metrics webinar and SQL tools.
- Netflix moved its batch compute platform to Kueue, mapping its tenant hierarchy to Cohorts/ClusterQueues/LocalQueues and adding preemption-based fair sharing so critical jobs run without manual intervention.
- Zalando built an in-process client-side load balancer for its Product Read API that handles over a million requests per second, matching Skipper's consistent hashing via Kubernetes watch-based discovery and AZ-aware N-ring fade-in.
- Zepto's Dual Sequence ReRanker uses separate transformer encoders for long-term history and in-session behavior, rebuilding per-candidate user profiles with target-aware pooling and a learned fusion gate incorporating real-time signals like trending counters.
- Technically strong data teams still fail to drive business impact if they only deliver data without pushing decisions, regardless of skill level.
Netflix moved its batch compute platform onto Kueue, a Kubernetes-native scheduler, while keeping its old API intact. They translated their tenant hierarchy into Kueue Cohorts plus ClusterQueues and LocalQueues. On top of that, they added preemption-based fair sharing to let critical jobs run without manual intervention. At Zalando, engineers built an in-process client-side load balancer for their Product Read API, hitting over a million requests per second. They matched Skipper’s consistent hashing, used Kubernetes watch-based discovery, and rolled out N-ring fade-in with AZ-aware routing for smooth scale-ups.
On personalization, Zepto introduced a Dual Sequence ReRanker that feeds long-term user history and in-session behavior through separate transformer encoders. They rebuild user profiles per candidate item with target-aware pooling, blend signals via a learned fusion gate, and inject real-time context like trending counters and calendar data. In data quality, the “Broken Windows” approach pushes validation into modeling reviews, local tests, CI/CD pipelines, AI-assisted checks and monitoring to catch inconsistencies early. And even rock-star data teams can miss the mark if they only deliver data without driving decisions—tech skills alone won’t influence business outcomes.
Beyond those case studies, the newsletter lays out a taxonomy for storage architectures—OLTP, OLAP, HTAP and LTAP—evaluated on system count, workloads, visibility and copies. It sorts systems into Single Tier, Internal Tiering, Hybrid, Materializing and Shared Tiering (think Databricks LTAP or Apache Fluss). For practical tools, SQL Concepts Lab offers a DuckDB-WASM tutorial right in your browser. dbtrail streams MySQL binlogs for point-in-time recovery and time-travel queries. SQLBuild bolsters dbt with change-aware execution through fingerprinting and lineage tracking, plus support for Python nodes and DuckDB CI.