More on the topic…
Grab's data mesh relies on certification to establish trust in data products, but that's only half the battle—certified data still fails in production. The company built Data Production Issues (DPIs) as an operational workflow that automatically detects, diagnoses, and resolves failures. When Kinabalu, their incident orchestrator, observes that a data asset violates its contract (checked through Test Health APIs), a DPI gets created. The system distinguishes between a single test failure and the overall health of a test, deduplicates alerts around the same breach, and gathers context about affected assets, dependencies, and which consumers are impacted. This filtering matters because it keeps the workflow focused on real contract breaches rather than drowning responders in noise.
The tricky part is figuring out who should fix what. Not every unhealthy dataset should land on the asset owner's desk—sometimes the problem is an upstream dependency, platform infrastructure, or the producing job itself. Grab uses a Data Health API with a simple four-category taxonomy (UPSTREAM_ERROR, PLATFORM_ERROR, JOB_ERROR, DATA_ERROR) that lets the orchestrator assign DPIs to the right team without needing to understand the internals of every platform. When table_A fails because table_B didn't arrive, the system traces back to the upstream DPI and groups both issues under the team that owns the root cause. This keeps the workflow stable across different platforms—ingestion systems, streaming, metrics, ML—because the orchestrator only consumes a consistent contract rather than platform-specific error catalogs.
Grab's ingestion platform Hugo demonstrates how this works in practice. It consolidates signals from multiple sources into a unified diagnostic workflow instead of treating each alert independently. The accuracy of these diagnostics matters because noisy or duplicated DPIs train people to ignore them. If the system can't reliably point teams toward fixable problems, it becomes noise. This creates pressure on each platform to improve its own diagnostics and incorporate signals from dependencies, not just its own local failure state.
Questions about this article
No questions yet.