Click any tag below to further narrow down your results
Links
OpenAI is pitching agents that operate EHRs and payer portals by clicking and typing, so they can handle tasks like MRI authorization that APIs don't reach. The post shares evaluation results on helpfulness, safety, and cost, plus early deployments with Cigna, CVS Health, Abridge, and Ambience.
- A demo agent takes a patient's message and visit notes, pulls prior imaging and PT records from the EHR, and drafts an MRI order, encounter note, and patient reply for clinician review.
- CVS Health reports an agent cleared a months-long document-sending backlog over a weekend and handles system steps for tens of thousands of consults a day, with validation checks and human oversight.
- Cigna case managers previously had to navigate more than 10 separate systems to gather member context; computer use now pulls that together in seconds.
Thomas Wolf, a startup founder, argues that AI's strong benchmark scores don't match its limited economic impact because real work depends on judgment and agency, which benchmarks rarely measure. He ties this to a drop in entry-level hiring, since execution is the part of junior work AI handles best, and that junior work is where people usually build the judgment that later makes them senior.
- A Stanford analysis found employment for workers aged 22–25 in the most AI-exposed occupations fell about 6% between late 2022 and mid-2025, while older workers in those jobs gained 6–9%.
- Models now solve around 75% of real-world coding tasks on SWE-bench, yet the Remote Labor Index shows the strongest agents succeed on only about 2.5% of paid freelance projects.
- Wolf's hiring framework splits workers into execution, judgment, and agency; AI is advancing fastest on execution, so the entry layer gets squeezed first, weakening the pipeline for future senior people.
The author argues that AI makes cheap to fake the signals we use to judge quality, such as GitHub stars, polished writing, and homework passing autograders, so good work stops getting rewarded and a market for lemons forms. He uses signaling theory to explain why this change is so fast and disorienting compared with earlier, slower erosion.
- Shortcuts like stars, clean READMEs, and passing tests worked because faking them used to cost nearly as much as doing the real work; AI removes that cost gap.
- Buying GitHub stars already cost about $0.07–0.20 each, and AI now makes fake accounts with realistic histories easy to produce, hiding the fakery better.
- When signals collapse, evaluation gets more expensive, and buyers, employers, and reviewers stop paying a premium for good work, so poor work dominates, as in Akerlof's 1970 used-car lemons paper.
- Past signal erosion (biological mimicry, CAPTCHAs, pass/fail grading) was usually slow, but AI is breaking many signals across open source, education, and research at once.
TypeSafe released Jev 1.14.0 with a split approach: stable improvements for production use and experimental "danger zone" features (image support, bounding box type, longer context) for developers willing to accept less reliability. The company is also sharing internal intelligence metrics while acknowledging their limitations.
- Image support is now composable across Jev's JSON interface (state, instructions, criteria), though multimodal performance is weaker than text-only and requires careful prompt decomposition.
- New "box" type returns bounding boxes without type errors, representing the first Jev type designed beyond binary decisions and signaling exploration into broader System One Model capabilities.
- Jev supports longer context lengths than previously enabled in production; developers can opt in via a danger zone flag, but TypeSafe intentionally limits context by default because reliability degrades at scale.
- Internal benchmarking data is being shared for the first time, limited to open-weight models only, with explicit caveats that the methodology will evolve and developers should run their own evals.
Redis published benchmark results showing its memory system outperforms other production systems on LongMemEval, a standard test for long-term agent memory. The report also flags that published memory benchmarks often don't replicate when re-tested independently.
- Redis' Remis + Instruct strategy achieved 86.5% task-averaged accuracy on LongMemEval, the highest among production systems tested on GPT-4o, at lower cost than competitors
- Oracle's managed service scored slightly higher (within 0.5 points) but uses a stronger answer model
- Vertex Memory Bank's actual performance (42.4%) significantly underperformed its published results (~70%), suggesting benchmark results don't always hold up in real-world setups
A researcher at Arize built a rigorous evaluation system to test whether Claude Opus 5.5 actually fixed the distinctive writing quirks that plague Claude outputs. The answer: it improved by 50%, but "fixed" overstates it — the model still relies heavily on rhetorical moves that make AI-generated text feel artificial.
- Em dashes dropped dramatically (from 12.9 to 0.035 per 1,000 words), but other Claudisms only halved across six categories: salience flags, verdict intensifiers, signposts, contrast reframes, stock metaphors, and gotcha framing.
- The real problem isn't specific phrases—it's rhetorical moves. Claude doesn't say "load-bearing" that often, but it constantly tells readers something matters instead of showing why, which no phrase list can catch.
- Evaluating writing quality requires human annotation first, then building automated checks against those annotations. The researcher manually flagged 153 instances across 53,000 words to create ground truth before training an LLM judge.
This article introduces PaperOrchestra, a multi-agent system that transforms raw idea summaries and experimental logs into submission-ready AI research papers using agents for outlining, plotting, literature review, writing, and refinement. It outperforms single-agent and state-of-the-art baselines on PaperWritingBench, a new benchmark of 200 CVPR and ICLR papers, in both literature review and overall manuscript quality.
- PaperOrchestra uses a multi-agent pipeline (outline, plots, citation graph via Semantic Scholar, LaTeX writing, peer-review-style refinement) to turn raw idea summaries and experiment logs into full draft papers.
- Tested on a new 200-paper benchmark (100 CVPR 2025, 100 ICLR 2025), it beat a monolithic LLM pipeline and AI Scientist-v2 by 50–68 points on literature-review quality and 14–38 points on overall manuscript quality in blind human evaluations by 11 AI researchers.
- Grounding citations in real API data and using iterative self-reflection agents reduced hallucinated references and shallow reviews compared to baselines.
- The tool is positioned as a drafting accelerator, not a replacement for authors—humans must still verify facts, originality, and ethics despite built-in flags for dubious content.
Andrei Kaparthy's insights on AI's role in work resonate with many, prompting a reflection on how to integrate these ideas into data engineering practices. The article emphasizes the importance of mastering fundamentals to effectively evaluate AI-generated work and encourages active participation in the evolving landscape of technology.
- Deep fundamentals in data engineering remain essential for judging whether AI-generated code, queries, or pipelines are actually correct
- Passively consuming AI outputs without understanding the underlying systems leaves practitioners unable to catch subtle errors
- Staying engaged with hands-on practice, rather than just watching AI do the work, is key to keeping pace with the field's evolution