Can Your LLM Vibe Code Your Semantic Layer? - Luke Kranz
Short summary
A researcher built a fictional billing database and had five AI models create semantic layers (business logic definitions) for it, then tested how well those layers helped the models answer finance questions. The results show that a well-built layer can dramatically improve accuracy, but models often make confident mistakes about scope and conventions that human finance teams would catch.
- Haiku failed to build a usable layer at all; the four models that succeeded got the math right but disagreed on scope—which accounts count as customers, whether subsidiaries roll up to parents, and how to handle QA test data.
- A reference layer written by a human improved Opus from 17 of 30 correct answers to 30 of 30, while a flawed layer built by Sonnet dropped Fable from 18 correct to 12, showing that semantic layers can either unlock or degrade model performance.
- Models built layers with high confidence regardless of accuracy: Muse matched finance definitions on all but one rule, while Opus and Fable made the same incorrect assumption about subsidiary rollups that finance never used, and Sonnet inconsistently filtered QA accounts across different metrics.
Full summary
A semantic layer sits between your data warehouse and the people querying it, defining what metrics mean so everyone gets the same answer to the same question. It specifies what counts as a customer, how revenue gets recognized, which accounts to exclude—the business logic that lives nowhere else. The author tested this by building a fictional B2B SaaS warehouse with 14 metrics and having five Claude models (Haiku, Sonnet, Opus, Fable) plus Muse build their own semantic layers, then grading how well each layer helped those models answer 30 finance questions. A strong reference layer built by a human from actual finance definitions got 97% of answers right. The layers built by AI ranged from 41% to 91% accuracy, and here's the catch: every model nailed the arithmetic but fumbled on scope—which accounts count, what defines a customer, how to handle subsidiaries. They stated their wrong choices with the same confidence as their right ones.
The results show semantic layers matter most when LLMs do the querying, since models follow the layer's definitions about 93% to 99% of the time. Haiku failed entirely, producing invalid metrics and actually making things worse—queriers got 30% right with its broken layer versus 39% with no layer at all. Sonnet filtered QA accounts from some metrics but not others, creating inconsistency. Opus and Fable both independently decided subsidiaries should roll up to parents (finance disagrees) and excluded QA accounts everywhere, changing every customer-based metric. Muse came closest to finance's actual definitions, only slipping on one edge case with multi-subscription accounts. The real problem: building a semantic layer can't be a one-shot task, even for frontier models. Each definition carries weight, errors compound across the pipeline from bronze to silver to gold tables, and the layer needs human verification.
The author published the full test setup—the generator, dbt project, all recorded sessions, and a grading script—on GitHub so others can review the results. The takeaway isn't that AI can't build semantic layers. It's that AI-built layers need the same scrutiny as human ones, and the stakes are high because downstream users and systems will trust whatever definitions the layer provides. A flawed layer actively drags down even strong models: Fable's broken layer dropped Sonnet from 18 correct answers to 12. When the layer is good, it lifts everyone. Opus went from 17 correct with no layer to a perfect 30 with the human-built reference layer.