Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
The paper shows that typicality bias in human preference data drives mode collapse in aligned LLMs, narrowing their output diversity. It introduces Verbalized Sampling, a prompt that asks the model to list multiple responses with their probabilities, and demonstrates significant diversity gains across writing, dialogue, QA, and synthetic data tasks without retraining.
- Mode collapse comes from "typicality bias" in human preference data—annotators favor typical/safe responses, so even a perfect reward model trained on that data would still collapse diversity.
- Verbalized Sampling (prompting the model to list multiple responses with probabilities) recovers pre-training diversity at inference time, no retraining needed.
- This boosts creative writing diversity 1.6–2.1x over direct prompting, with human judges rating outputs 25.7% better on average, and it also improves downstream math solver accuracy when used to generate training data—all without hurting factual accuracy or safety.
Mode collapse—where a tuned language model keeps spitting out the same safe, familiar text—turns out to be driven not just by faulty algorithms but by the data humans supply. Annotators naturally pick typical, “safe” responses when ranking model outputs. The authors call this “typicality bias.” They build a simple mathematical model showing how preference labels skew toward common phrases, then confirm it across real RLHF datasets. Even a perfect reward model would still collapse if its training labels favor the usual over the unusual.
To push back on that, they propose Verbalized Sampling (VS). Instead of asking the model for a single best answer (“Tell me a coffee joke”), you prompt it to generate, say, five responses plus their probabilities. By making the model reveal its own internal distribution, VS recovers much of the pre-training diversity that alignment steps had suppressed. It’s a zero-train, inference-only tweak.
They test VS on poems, jokes, short stories, dialogue simulations, open-ended QA and synthetic data creation. Diversity jumps by 1.6–2.1× in creative writing compared to direct prompting. Human judges rate VS outputs 25.7% better on average. In dialogue tasks, VS outputs behave as naturally as those from a fine-tuned model. And when you use VS-generated data to train math solvers, downstream accuracy climbs. All that without hurting factual accuracy or violating safety constraints.
The key takeaway: aligned LLMs still hold latent variety. By extracting it with a distributional prompt, you undo much of the narrowing effect of human preferences. This moves the needle on the trade-off between safe, high-quality answers and creative, varied ones—without retraining or changing the model itself.