2 links tagged with all of: data-architecture + data-quality
Click any tag below to further narrow down your results
Links
This roundup covers Netflix’s switch to Kueue for Kubernetes-native batch compute, an engineer’s workflow for long-running coding agents, and Zalando’s in-process client load balancer handling over a million requests per second. It also explains Zepto’s dual-sequence re-ranker for real-time personalization, strategies for catching data issues early, why technically strong teams still miss business impact, and a new storage/workload architecture taxonomy—plus a Databricks metrics webinar and SQL tools.
- Netflix moved its batch compute platform to Kueue, mapping its tenant hierarchy to Cohorts/ClusterQueues/LocalQueues and adding preemption-based fair sharing so critical jobs run without manual intervention.
- Zalando built an in-process client-side load balancer for its Product Read API that handles over a million requests per second, matching Skipper's consistent hashing via Kubernetes watch-based discovery and AZ-aware N-ring fade-in.
- Zepto's Dual Sequence ReRanker uses separate transformer encoders for long-term history and in-session behavior, rebuilding per-candidate user profiles with target-aware pooling and a learned fusion gate incorporating real-time signals like trending counters.
- Technically strong data teams still fail to drive business impact if they only deliver data without pushing decisions, regardless of skill level.
This article walks through setting up a future-proof data stack by outsourcing ingestion to turnkey tools, sticking with SQL-based transforms, and carefully integrating AI without ignoring fundamentals. It also covers key areas like data quality, storage and compute choices, and when to move from replicas to a full warehouse or lakehouse.
- Ingestion is a solved problem now—use managed connectors (Estuary, Fivetran, Glue, DLT) instead of building custom pipelines unless there's truly no supported option.
- SQL remains the backbone of transformation work regardless of tool (dbt, stored procedures, Airflow), with AI only starting to help via auto-generated schemas and template joins, not replacing hands-on cleanup.
- Flashy AI chatbots/analytics tools often demand as much setup as traditional dashboards, so lightweight CLI or skills-file integrations with tools like Claude or Codex are a more practical automation win.
- Don't over-architect storage/compute—a read replica and plain SQL can beat a lakehouse or warehouse if your team size and actual workload don't justify the complexity.