This issue covers the latest in data tooling and AI agents—from Databricks’ Agent Orchestrator and Spotify’s Vedder assistant to Feldera’s incremental view engine and LinkedIn’s MUSE semantic search. It also dives into managing agentic AI costs, building stable model ecosystems, and new open-source releases like Omnigent and Apache DataFusion 54.0.0.

• Spotify's Vedder only kept 12.5% of automatically mined query pairs after expert review, relying on human-curated clusters over the other 70,000-dataset schema-only retrieval
• Uber's 5,000 engineers burned through the annual AI budget by mid-April using Claude Code, showing token pricing becomes a task-economics problem once retries, orchestration, and governance overhead are counted
• DataFusion 54.0.0 added LATERAL joins, SQL lambdas, and spill-to-disk nested loops, pushing join speeds up to 50x faster in some cases
• Databricks' Omnigent unifies Claude Code, Codex, Pi, and in-house agents under one harness with shared security, cost limits, and session sharing

This article discusses a live session with Alejandro Aboy, a Senior Data and AI Engineer, about the evolving role of data engineers in the context of AI. Alejandro emphasizes that strong data engineering skills, particularly data modeling and orchestration, are crucial for successful AI work and shares practical steps for integrating AI into existing data workflows.

• Data modeling accounts for roughly 80% of the skill transfer from data engineering to AI engineering, according to Aboy.
• Good orchestration/workflow design can handle most AI tasks without needing complex agents.
• Precise column/model descriptions function as prompts for AI, so metadata and documentation quality directly determine AI performance.
• AI still lacks common-sense judgment (illustrated by a debugging agent missing context), so critically reviewing outputs rather than accepting them blindly remains essential.