Claire Gouze lays out a practical process to build and test a context layer your data agents can trust. She shows how to start small with dbt docs and markdown files, write evals, clean up data models, and iterate until you reach around 90% reliability.

• Adding query logs and profiling alone got the agent stuck around 40% accuracy; reworking the data model and documentation pushed it to 90%.
• Treat context like code: store it in files/markdown, version in Git, add CI/CD evals, and iterate on failure cases rather than plugging agents directly into raw databases.
• Start small—10 to 20 core tables with existing dbt docs—rather than trying to cover the whole warehouse at once.
• Evals should come from real BI query history or autogenerated question sets, reviewed by teams and tracked over time to build trust in the agent.

The article discusses the shifting landscape for data scientists and machine learning engineers in the age of large language models (LLMs). It emphasizes the importance of data science fundamentals in evaluating AI systems, addressing common pitfalls in metrics, experimental design, and data quality. The author argues that the core work of data scientists remains vital, even as their roles evolve.

• Off-the-shelf eval framework metrics often mislead teams; digging into your own data to find relevant metrics is what data scientists actually do.
• Using LLMs as judges without validating them against human labels is a common, risky shortcut.
• Synthetic test data that isn't grounded in real production logs leads to flawed experimental design and misleading results.
• Outsourcing labeling away from domain experts degrades data quality and undermines the whole evaluation process.

This article explores how advanced AI models can generate detailed image descriptions and reasoning without actual image input, a phenomenon called mirage reasoning. It highlights vulnerabilities in these models, particularly in medical contexts, and introduces B-Clean, a method for better evaluating multimodal AI systems by minimizing non-visual inference.

• A model topped a chest X-ray QA benchmark while being completely blind to the actual images, exposing "mirage reasoning" where models fabricate plausible visual descriptions from text priors alone.
• Explicitly prompting models to guess without assuming an image was present caused performance to drop sharply, showing benchmark scores were inflated by non-visual inference rather than real image understanding.
• The proposed B-Clean method strips out textual cues that let models cheat their way to correct answers, aiming to force genuine visual grounding in evaluations—critical for high-stakes fields like medical AI.

The article discusses the shortcomings of achieving high accuracy in Text-to-SQL systems, emphasizing that 90% accuracy is insufficient for enterprise applications. It highlights the need for rigorous evaluation frameworks, like Spider 2.0, to ensure reliability and trust in AI-driven analytics.

• 90% accuracy still means 1 in 10 SQL queries is wrong, which is enough to destroy user trust and stall enterprise adoption.
• Standard benchmarks like Spider 1.0 don't reflect enterprise reality; Spider 2.0 tests against schemas with 3,000+ columns to simulate real-world messiness.
• Simple accuracy scores are misleading — metrics like Execution Accuracy (EX) and Soft-F1 are needed to actually capture whether generated SQL is reliable.

The article discusses how the rise of AI tools, particularly LLMs, has affected software engineering and data work. While some engineers are concerned about the declining quality of code, data professionals find value in these tools for generating quick, low-maintenance solutions. It emphasizes the need for careful evaluation of the new data generated by these systems.

• Fast, disposable code from LLMs isn't a problem for data work the way it is for engineers building long-term systems, since data scientists often throw away code after one use anyway
• LLMs are genuinely useful for building data extraction tools (e.g., pulling usable data from messy PDFs), unlocking previously inaccessible data sources with minimal effort
• Using LLMs to generate synthetic data is risky and shouldn't replace real human data, especially in user experience research
• As LLMs improve at generating and explaining code, data professionals need less hand-holding from software engineers for low-risk coding tasks, shrinking the traditional divide between the two roles

Andrei Kaparthy's insights on AI's role in work resonate with many, prompting a reflection on how to integrate these ideas into data engineering practices. The article emphasizes the importance of mastering fundamentals to effectively evaluate AI-generated work and encourages active participation in the evolving landscape of technology.

• Deep fundamentals in data engineering remain essential for judging whether AI-generated code, queries, or pipelines are actually correct
• Passively consuming AI outputs without understanding the underlying systems leaves practitioners unable to catch subtle errors
• Staying engaged with hands-on practice, rather than just watching AI do the work, is key to keeping pace with the field's evolution