Dream-RSI: Recursive Self-Improvement through Evolving Worlds | Papers with Code
This paper presents Dream-RSI, a framework that lets AI agents improve their exploration strategies by learning from past discovery attempts without constantly running expensive new experiments. The system uses historical search data as a simulator to test and refine exploration policies cheaply before deploying them back into the real search space.
- Solves the exploration bottleneck in recursive self-improvement by creating a replay simulator from accumulated discovery history, enabling off-policy feedback without costly online re-evaluation
- Keeps the exploration layer separate from the underlying agent, making strategies explicit and programmable while maintaining compatibility with existing systems
- Demonstrates competitive or better results across algorithm engineering, mathematical optimization, and GPU kernel engineering while substantially cutting discovery costs
Dream-RSI Summary
The core problem here is that autonomous AI agents need to explore search spaces effectively to find good solutions, but exploration strategies themselves become a bottleneck as problems get harder. Standard approaches hit a wall: you can use fixed exploration tactics, but they don't adapt when the search space grows. Alternatively, you can optimize exploration policies online, but that means running expensive trials repeatedly with slow feedback from long-horizon attempts. The researchers behind Dream-RSI spotted the real issue — we're treating exploration as something that happens in a vacuum rather than something we can learn from systematically.
Their solution uses a replay simulator built from the agent's own discovery history. Instead of running new expensive trials every time you want to test a better exploration strategy, you can "dream" — simulate what would happen using the tree of past discoveries you've already accumulated. This gives you instant, cheap feedback on whether a new exploration policy would have worked better on problems you've already seen. Once you've refined the policy this way, you deploy the improved version to find new solutions, which then expands your simulator pool. It's a self-reinforcing loop where each round of dreaming produces a better exploration strategy, which finds more discoveries, which builds a richer simulator for the next round.
They tested this across three different domains: algorithm engineering, mathematical optimization, and GPU kernel engineering. Dream-RSI matched or beat existing approaches while cutting discovery costs substantially. The framework keeps the underlying coding agent unchanged and just adds a lightweight layer on top that makes exploration explicit and programmable. The practical payoff is clear — you get better solutions without burning through as many expensive evaluations.