@nikogrupen - Wake-Sleep Learning for Legal

Open original

@nikogrupen on X This post describes applying the wake-sleep learning algorithm to long-horizon legal agents, where the agent works on tasks online and a review model writes reusable lessons offline. Tested on Harvey's Legal Agent Benchmark, the approach raised all-pass rates and cut cost per task by letting the agent retrieve only relevant lessons.

  • Wake-sleep lifted the all-pass rate from 2.9% for a vanilla agent to 15.7% across 196 Corporate M&A and Capital Markets tasks, with the biggest gains in early cycles.
  • Lessons written offline generalized to new matter types the agent never saw in training, not just familiar ones.
  • Retrieving only task-relevant lessons halved cost per task while keeping rubric pass rates roughly the same.

Wake-sleep is an old learning idea that alternates between an online "wake" phase, where a system acts on real data, and an offline "sleep" phase, where it reviews what happened and generates examples or lessons to improve. The authors argue this pattern is moving into agent design, since long-horizon agents leave a record of actions and deliverables that can be graded and reviewed after the fact. They point to recent work like Reflexion, Anthropic's Dreaming in Claude Managed Agents, and Cognition's Agent Memory Repo as signs the approach is catching on. Their own test applies it to legal work using Harvey's Legal Agent Benchmark, focusing on tasks like issues lists, memos, and marked-up drafts drawn from synthetic client matters. The pipeline runs in six steps: the agent works through tasks, dual LLM judges grade the deliverables against expert rubrics without the agent seeing the grades, a review model writes candidate lessons from the graded trajectories, a merge step consolidates overlapping lessons and judges which older ones helped, a commit gate rejects lessons that are too narrow, tied to one training task, or contain client-specific details like a party name or purchase price, and the surviving lessons are stored as text grouped by analysis, drafting and review, and general learnings. The overfitting gate is the piece the authors clearly worry about most. In a 10-cycle run across 196 Corporate M&A and Capital Markets tasks (110 for training, 86 for validation), with GPT-6 Luna as the agent model, the wake-sleep agent reached a 15.7% all-pass rate against 2.9% for a vanilla agent with no memory. Gains were larger on familiar matters, with over 10 percent more rubric criteria completed per task, and smaller but still present on genuinely new matters, at nearly 5 percent more. Most of the improvement came in the first few cycles. The qualitative findings are where the mechanism gets interesting. The first sleep cycle added 122 lessons, mostly checklist items saying what a deliverable must contain. Over later cycles these were refined into behavioral rules and tactical instructions, such as "recompute every figure yourself to verify it," while unhelpful or overly specific ones were pruned. One example follows a deadline lesson that started as a rule to state contractual time terms as durations and then picked up cases like weekend deadlines and conflicting windows across documents. The behavioral change is measurable: the agent made about 2.5 times as many tool calls per task, mostly for running code, checking work, and drafting, while its document reading and searching stayed roughly flat. A separate finding is that retrieving only the memories relevant to each task, instead of loading all of them, roughly halves the cost per task at about the same quality. The authors note that identifying which lessons drive which behavior is still open work.

Ask about this article