The article argues that as AI automates data queries, pipelines, and models, the real value shifts to “measurement engineers” who decide if we’re measuring the right things and interpret ambiguous results. It breaks down why judgment—construct validity, reliable metrics, and decision theory—is a teachable skill that organizations must build into hiring, training, and structure.

• As AI automates SQL, pipelines, and modeling, the bottleneck shifts to judgment: deciding whether you're measuring the right thing.
• Teams that track hundreds of metrics tend to cherry-pick supporting ones instead of narrowing to a few that predict real outcomes.
• Rising internal model evals can coexist with falling user satisfaction because evals often measure fluency, not usefulness.
• Borderline A/B test results demand power analysis and decision theory, not just p-values, to judge if effects like a 1.5% retention drop are real.

The article discusses the shifting landscape for data scientists and machine learning engineers in the age of large language models (LLMs). It emphasizes the importance of data science fundamentals in evaluating AI systems, addressing common pitfalls in metrics, experimental design, and data quality. The author argues that the core work of data scientists remains vital, even as their roles evolve.

• Off-the-shelf eval framework metrics often mislead teams; digging into your own data to find relevant metrics is what data scientists actually do.
• Using LLMs as judges without validating them against human labels is a common, risky shortcut.
• Synthetic test data that isn't grounded in real production logs leads to flawed experimental design and misleading results.
• Outsourcing labeling away from domain experts degrades data quality and undermines the whole evaluation process.