The context engineering playbook (Claire Gouze)

Open original

Claire Gouze lays out a practical process to build and test a context layer your data agents can trust. She shows how to start small with dbt docs and markdown files, write evals, clean up data models, and iterate until you reach around 90% reliability.

  • Adding query logs and profiling alone got the agent stuck around 40% accuracy; reworking the data model and documentation pushed it to 90%.
  • Treat context like code: store it in files/markdown, version in Git, add CI/CD evals, and iterate on failure cases rather than plugging agents directly into raw databases.
  • Start small—10 to 20 core tables with existing dbt docs—rather than trying to cover the whole warehouse at once.
  • Evals should come from real BI query history or autogenerated question sets, reviewed by teams and tracked over time to build trust in the agent.

Claire Gouze built nao Labs after talking to 80 data teams and realizing they all struggled to ship reliable agents because their context layers were messy. She defines context engineering as the process of gathering business knowledge—docs, data models, semantic layers—and turning it into a structured, testable layer that an AI agent can query. Instead of plugging an agent directly into raw databases (the 2010s mistake of BI tools), she recommends treating context like code: store it in files or markdown, version it in Git, add CI/CD tests, and iterate based on failure cases. Her playbook starts small. Pick a core team or a handful of key metrics—10 to 20 tables—and pull in whatever documentation you already have, usually dbt docs. Run an initial round of evals to get a baseline reliability score. Then look at where the agent trips up: ambiguous column names, inconsistent metrics, missing definitions. Tidy your data model and write clear docs. In Gouze’s experiment, adding query logs and profiling got her agent stuck around 40% accuracy. Reworking the model and fleshing out documentation pushed it up to 90%. Evaluations come from two sources: existing BI queries or autogenerated question sets based on your table metrics. Teams review and refine those questions, then track accuracy over time. That reliability score is what gives business users confidence to adopt the agent. On tooling, nao provides a UI for permissions and governance, but the core remains a simple file-based context layer you own, avoiding vendor lock-in. Finally, context engineer is already a job title at some companies. Gouze argues data people have always done this work—translating stakeholder knowledge into technical artifacts—but now they need to optimize it for NLP agents, balancing thoroughness with token costs. The goal isn’t flashy new data sources; it’s keeping your “house” in order so agents give consistent, trustworthy answers.

Ask about this article