Databricks’ Agent Orchestrator 🕹️, Ecosystems Beat Models 🔁, LinkedIn’s Search Brain 🔍

Spotify’s Vedder puts domain experts at the center of its data assistant. Instead of relying on schema-only retrieval, Spotify divided its 70,000 datasets into 177 clusters. Experts curate each cluster with vetted question-to-SQL pairs and relevant business documents. Only 12.5 percent of automatically mined query pairs passed expert review, so Vedder uses health scores—tracking drift, validity, coverage and reproducibility—to keep its context accurate for 2,100 users. Feldera treats streaming data as incremental SQL views, using DBSP to push only row-level deltas through joins and aggregations. Inserts, updates and deletes map to Z-set changes, cutting CPU use and memory pressure compared with full recomputations. At LinkedIn, MUSE powers semantic search in the Hiring Assistant. A dual-tower embedding model learns from millions of examples labeled by an LLM teacher tied to product policy. Embedding-based retrieval narrows candidates; a downstream ranker then optimizes for engagement. AI coding agents are speeding up routine work but they can balloon technical debt if left unchecked. Teams at Uber burned through their annual AI budget by mid-April after 5,000 engineers adopted Claude Code. Hidden costs—from redundant context sent in retries, orchestration overhead and governance checks—turned token pricing into a task-economics problem. Experts now measure value per task, clamp context size and build stateful agent infrastructure to contain spending. On the tooling side, Databricks’ Omnigent meta-harness stitches together Claude Code, Codex, Pi and in-house agents under one layer, adding security, cost limits and live session sharing. Apache DataFusion 54.0.0 rolled out LATERAL joins, SQL lambdas for arrays, an Arrow-based Avro reader and spill-to-disk for nested loops, boosting join speeds by up to 50× in some cases. MotherDuck’s Flights feature lets agents launch Python-based ingestion and transformation jobs with built-in logging and versioning. The Linux Foundation’s OpenSharing project extends Delta Sharing into a standard API for models, skills and unstructured data across clouds. Meanwhile, Databricks warns that ai_parse_document pipelines can rack up LLM costs and duplicate work; adding checkpoints, versioned prompts and dedup filters—or switching to deterministic parsers when templates are stable—keeps costs and reprocessing in check.