Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering

Open original

Spotify built an AI data assistant, Vedder, to let anyone query its 70,000+ datasets in plain English. It uses domain-specific clusters—each with selected tables, vetted question-SQL pairs, and docs—curated and maintained by experts to ensure accuracy and trust. A continuous health score and feedback loop keep clusters up to date as data and schemas evolve.

  • Spotify rejected 87.5% of auto-generated example query pairs, keeping only human-vetted ones to avoid encoding bad query patterns
  • Domain experts curate "clusters" (tables, vetted Q&A examples, docs) rather than dumping full schemas into the LLM, since context windows and raw schemas can't capture business nuance
  • Each cluster gets a continuous health score tracking schema drift, example validity, and query reproducibility, auto-flagging maintenance tasks when things break
  • User interactions feed back to cluster owners, turning experts' role from answering one-off questions into maintaining a shared context layer that scales to thousands of users

At Spotify, teams struggled to get reliable data because there are over 70,000 datasets across petabytes of storage and no single expert knows it all. Traditional dashboards couldn’t keep up, and Slacking a data guru introduced long waits. Simply dumping warehouse schemas into an LLM failed: context windows top out, and raw table definitions don’t capture quirks like legacy test data or nuanced business terms. Spotify solved this with Vedder, a data assistant that routes questions to the right “cluster” of domain-owned context, generates SQL, runs it, and returns results plus the query and sources. Each cluster groups relevant tables (with schema, column cardinality, sample values), hand-picked question-and-SQL examples, and written docs on terminology and gotchas. Domain experts curate these three components. They rejected 87.5% of auto-generated example pairs drawn from historical queries, because most past queries are one-offs or teach bad patterns. By keeping only vetted examples, clusters encode true business knowledge. When a question arrives, Vedder uses a ReAct loop—reasoning, acting, checking tool outputs—to build and refine its SQL before returning a final answer. Clusters earn a health score based on signals like schema drift, validity of curated pairs after changes, reproducibility of generated queries, and coverage of actual user questions. If a column is renamed or a table deprecated, that cluster’s score drops and suggests specific maintenance tasks on its dashboard. Every user interaction—questions, SQL, feedback—feeds back to cluster owners, helping them spot gaps and update examples or docs. That feedback loop scales each expert’s reach, shifting their work from answering ad-hoc requests to stewarding a shared context layer for thousands of users.

Ask about this article