A good teacher meets the student where they are
Short summary
JOLT is a training method where one model plays both teacher and student, with the teacher trained to give feedback the student can actually learn from. The authors report gains in accuracy and token efficiency over GRPO on math, coding, and agentic tasks.
- Outcome-reward RL gives sparse feedback, and on long agentic tasks with rare successes that makes learning slow and sample-hungry.
- Privileged-information teachers (like a model that sees the verified solution) can be good at the task but poor at teaching it, since their guidance may exceed what the student can absorb.
- JOLT trains the teacher to be "reward-consistent," meaning its guidance pushes the student in the same direction as RL on the student's own reward. Reported gains over GRPO include +4.2pp on MATH, +3.0pp on LCBv6, and +9.9pp on AppWorld.
Full summary
Reinforcement learning after pretraining usually gives a language model one sparse signal per attempt: did the whole thing succeed or not. On long agentic tasks, where successes are rare, that tells the model almost nothing about which intermediate steps helped. On-policy distillation (OPD) fixes the density problem by having a stronger teacher score every token of the student's own attempt, but the authors point out a catch: a model that solves a task well is not automatically good at teaching it, since its guidance can sit beyond what the student can use. JOLT, short for Joint On-policy Learning and Teaching, is their answer. It builds on on-policy self-distillation (OPSD), where one model plays two roles, a teacher that sees privileged information like verified solutions and a student that doesn't, and trains both roles at once.
The interesting part is how the teacher is judged. Rather than rewarding the teacher for solving the problem, JOLT asks whether the update the teacher induces in the student matches the policy gradient of the student's own expected reward. The authors call such a teacher "reward-consistent" and give a closed-form target based on student advantages, with a penalty term that keeps the teacher near the student's own behavior. The student side is the standard OPSD update, penalizing reverse KL to the privileged teacher at every state it visits. The point, as they frame it, is that a teacher can exploit shortcuts the student can't reach or put probability on trajectories the student would never produce, and neither of those helps the student learn. Their closest comparison, π-Distill, also trains both roles jointly but uses a different teacher target and an off-policy student update.
On the numbers, JOLT+ posts the highest mean score in every setting they tested. Against GRPO, the strongest baseline, it gains 1.5 points on GSM8K, 4.2 points on MATH, 3.0 points on LCBv6, and 9.9 points on the AppWorld state-test pass rate. The experiments span math, code, tool use, and terminal tasks, with Qwen3-30B-A3B-Instruct trained on the TMax terminal set and evaluated on Terminal-Bench 2.1 among the setups. They also fit accuracy against log training tokens and report that JOLT's token advantage holds across roughly 55 to 65 percent accuracy, which is the efficiency claim that matters most for expensive agentic training runs. The authors close by suggesting the same joint setup could eventually train a teacher that generates new tasks for the student, though that remains future work.