How we built an internal data analytics agent

Open original

GitHub built Qubot, an internal Copilot-powered analytics agent that lets employees query their data warehouse in plain English via Slack, VS Code, or the CLI. It pulls federated context from raw, conformed, and curated data layers, then runs queries on Kusto or Trino to deliver SQL-backed insights within seconds. A continuous evaluation framework and community contributions to the context layer keep responses accurate and fast.

  • Qubot lets any employee query GitHub's data warehouse in plain English via Slack, VS Code, or CLI, cutting analytics team question load and freeing analysts for harder work
  • It uses a three-tier federated context layer (bronze/silver/gold) maintained collaboratively by product, analytics, and business teams, turning scattered knowledge into a single source of truth
  • Investing in well-structured context both improved accuracy and cut answer time by two-thirds
  • A continuous offline evaluation framework tests every context or agent update against known SQL answers to catch regressions before release

At GitHub, the analytics team built Qubot, an internal Copilot-powered agent that turns plain-language questions into SQL and returns data in seconds. Users ask a question in Slack, VS Code or the Copilot CLI, and Qubot spins up on GitHub’s cloud agent. Results land as a Slack reply and a markdown report in a pull request, which teams can tweak or plug into dashboards. It isn’t meant to replace dashboards; it’s for quick exploratory queries like “Which user cohort shows highest retention?” or “What drove this metric last week?” With zero-maintenance and no setup, even employees who’ve never touched the data warehouse can dive in immediately. Under the hood, Qubot has three layers: the user interface, a federated context layer, and a query engine. The context layer pulls metadata from three tiers of data—raw events (bronze), conformed facts (silver) and curated business datasets (gold). Product teams feed telemetry schemas for bronze data; the analytics team maintains query examples and mandatory filters for silver; and business owners contribute metric definitions for gold. This context gets loaded at runtime via GitHub’s MCP server and is continually enriched through ETL pipelines and a context-agent that standardizes new contributions via pull requests. Every update to either context or the agent itself goes through an offline evaluation framework. A suite of test cases—prompts with known SQL answers—runs multiple parallel trials via the GitHub CLI. The framework measures completion rate, accuracy and latency, catching regressions before they reach users. Query execution happens over two engines: Kusto for fast, recent-data exploration and Trino for complex joins and long-term history. Qubot defaults to Kusto, then switches to Trino when needed, so users don’t have to choose. Since launch, hundreds of GitHub employees have run thousands of queries through Qubot. Questions in the central analytics Slack channels have plummeted, freeing up data analysts for tougher tasks. The team found that well-structured context not only boosted Qubot’s accuracy but also cut answer-time by two-thirds. By centralizing dataset rules and metadata, Qubot turns distributed knowledge into a single source of truth—and gives every product and business team a stake in keeping that knowledge current.

Ask about this article