How Redis Agent Memory stacks up | Redis
Short summary
Redis published benchmark results showing its memory system outperforms other production systems on LongMemEval, a standard test for long-term agent memory. The report also flags that published memory benchmarks often don't replicate when re-tested independently.
- Redis' Remis + Instruct strategy achieved 86.5% task-averaged accuracy on LongMemEval, the highest among production systems tested on GPT-4o, at lower cost than competitors
- Oracle's managed service scored slightly higher (within 0.5 points) but uses a stronger answer model
- Vertex Memory Bank's actual performance (42.4%) significantly underperformed its published results (~70%), suggesting benchmark results don't always hold up in real-world setups
Full summary
Redis tested its memory system on LongMemEval, a standard benchmark for how well AI agents retain and use long-term information. Their combined Remis + Instruct approach hit 86.5% accuracy—the highest among production systems they tested on the same GPT-4o model. Only Oracle's managed service beat it, scoring slightly higher, but Oracle uses a more capable answer model. The real win here is that Redis achieved near-Oracle performance at significantly lower cost, which matters for anyone actually deploying this stuff at scale.
The benchmark results themselves tell a cautionary tale. When Redis reran published memory scores from other systems through the same testing setup, the numbers didn't match what was originally reported. Vertex Memory Bank, for instance, scored 42.4% in their test—roughly 40 percentage points below the 70% claimed in the original paper. The gap exists because the shipping product's actual configuration differs from what the researchers tested in their published work. This gap matters because it means you can't just trust headline numbers without understanding what's actually running in production.
The takeaway isn't complicated: Redis built something competitive on a public benchmark and was transparent about why other systems' published results fell apart under scrutiny. Whether 86.5% is good enough depends on what you're trying to do, but the cost-to-performance ratio and the willingness to show their work puts them in a different category from vendors who just cite their own papers.