Click any tag below to further narrow down your results
Links
Redis published benchmark results showing its memory system outperforms other production systems on LongMemEval, a standard test for long-term agent memory. The report also flags that published memory benchmarks often don't replicate when re-tested independently.
- Redis' Remis + Instruct strategy achieved 86.5% task-averaged accuracy on LongMemEval, the highest among production systems tested on GPT-4o, at lower cost than competitors
- Oracle's managed service scored slightly higher (within 0.5 points) but uses a stronger answer model
- Vertex Memory Bank's actual performance (42.4%) significantly underperformed its published results (~70%), suggesting benchmark results don't always hold up in real-world setups
Researchers at Latch Bio created a benchmark of 100 experimentally grounded tasks to test whether AI agents can make defensible decisions in antibody discovery. Claude Opus 5 performed best at 53% pass rate, but all models frequently failed by answering scientifically adjacent questions rather than the actual problem at hand.
- Opus 5 with Claude Code achieved the highest pass rate at 53%, with xAI and Google models within 3 percentage points, but no model was reliably accurate
- Model rankings shifted depending on the specific competency and decision type—higher cost and token usage didn't correlate with better performance
- Most failures came from scientific framing errors, not computational mistakes: agents performed internally consistent calculations on the wrong question
- The benchmark spans 10 discovery stages from target assessment through engineering and candidate de-risking, with diverse evidence types (binding kinetics, dose-response, sequence data, structural info)