6 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
Uber improved its data observability by implementing a system that tracks I/O patterns across its cloud and on-prem infrastructure. This allows for real-time insights into application performance, network usage, and data access, aiding in migration to a hybrid cloud model. The solution aggregates metrics without requiring code changes, benefiting various workloads.
If you do, here's more
Uber's data infrastructure is shifting to a hybrid cloud setup, which requires better tracking of data access patterns. This is where I/O observability becomes essential. The transition to CloudLake, Uber's hybrid cloud architecture, presents challenges like network bottlenecks and efficient workload management. Previously, Uber lacked visibility into important metrics such as data read/write throughput and network egress, which are critical for optimizing performance and reducing costs.
To address these gaps, Uber developed a real-time, vendor-agnostic observability system. By enhancing their Hadoop-compatible file system clients, they can now track all file stream operations and capture detailed metrics, including runtime metadata. This change allows Uber to gather data without needing to alter application code. The system aggregates metrics from millions of workloads daily, including those from Spark and Presto, which provide insights into data usage across different cloud providers and on-prem systems.
Managing the massive volume of generated metrics posed another challenge. Uber implemented HiCam, an in-house metrics aggregator that reduces the number of metrics sent to storage by consolidating data from thousands of containers. This approach minimizes write amplification and storage costs while maintaining near-real-time observability. HiCam, backed by Apache Flink and Zookeeper for high availability, has become central to Uber's observability strategy, enabling detailed analytics and efficient resource management across their expansive data ecosystem.
Questions about this article
No questions yet.