Click any tag below to further narrow down your results
Links
OpenAI introduces LifeVal, an evaluation of everyday practical tasks like finding housing, managing budgets, accessing government services, and shopping, which the company used to track ChatGPT's progress over the past year. The goal is to measure whether AI can take actions that actually get things done, not just give advice.
- The tasks were chosen from areas where users most want help: caregiving, health, education, household administration, personal finances, shopping, and government services.
- The benchmark tests action-taking, such as adding a deed to a cart for review, rather than only chat-style answers.
- OpenAI says it has used LifeVal to track and drive model improvement on everyday tasks over the past year, and frames it as a step beyond academic knowledge tests.
This arXiv paper introduces FrameBench, a multiple-choice benchmark that checks whether language models can tell apart the semantic frames a verb evokes in different sentences, the implicit background inferences humans make when reading. It covers English and Japanese, built from FrameNet-style resources with native-speaker verification.
- The benchmark tests whether models distinguish the frames the same verb evokes across contexts, probing implicit enrichment rather than surface-level language skill.
- Small models struggle noticeably, while several large models score above the human reference baseline.
- The dataset and construction and evaluation code are publicly released, so the results can be reproduced and extended.
Redis published benchmark results showing its memory system outperforms other production systems on LongMemEval, a standard test for long-term agent memory. The report also flags that published memory benchmarks often don't replicate when re-tested independently.
- Redis' Remis + Instruct strategy achieved 86.5% task-averaged accuracy on LongMemEval, the highest among production systems tested on GPT-4o, at lower cost than competitors
- Oracle's managed service scored slightly higher (within 0.5 points) but uses a stronger answer model
- Vertex Memory Bank's actual performance (42.4%) significantly underperformed its published results (~70%), suggesting benchmark results don't always hold up in real-world setups
Researchers at Latch Bio created a benchmark of 100 experimentally grounded tasks to test whether AI agents can make defensible decisions in antibody discovery. Claude Opus 5 performed best at 53% pass rate, but all models frequently failed by answering scientifically adjacent questions rather than the actual problem at hand.
- Opus 5 with Claude Code achieved the highest pass rate at 53%, with xAI and Google models within 3 percentage points, but no model was reliably accurate
- Model rankings shifted depending on the specific competency and decision type—higher cost and token usage didn't correlate with better performance
- Most failures came from scientific framing errors, not computational mistakes: agents performed internally consistent calculations on the wrong question
- The benchmark spans 10 discovery stages from target assessment through engineering and candidate de-risking, with diverse evidence types (binding kinetics, dose-response, sequence data, structural info)