ConvApparel: Measuring and bridging the realism gap in user simulators
Short summary
ConvApparel is a human-AI conversation dataset and evaluation framework designed to quantify and reduce the realism gap in LLM-based user simulators. Using dual-agent shopping dialogues and three validation pillars—statistical alignment, human-likeness scoring, and counterfactual tests—researchers show that data-driven simulators adapt more like real users but still exhibit detectable artifacts.
- Google researchers built ConvApparel by secretly routing users to a "Good" or "Bad" recommender across 4,000+ sessions and 15,000 turns, capturing real turn-by-turn satisfaction/frustration ratings.
- They tested prompting, in-context learning, and fine-tuning (on Gemini 2.5 Flash) as user simulators, validating each against stats alignment, a human-likeness discriminator, and counterfactual "Good vs Bad" agent tests.
- Prompt-based and in-context simulators failed to show genuine frustration when switched to the "Bad" agent, staying too smooth and helpful.
- Only the fine-tuned simulator closely matched real humans' shifts in frustration and satisfaction, showing targeted training on real interactions beats prompting or few-shot examples for realistic simulation.
Full summary
LLM-based user simulators often act more patient, knowledgeable, or consistent than real people. ConvApparel tackles this by measuring the “realism gap” in user simulators and guiding their improvement. Google researchers Ofer Meshi and Sally Goldman built a human-AI conversation dataset around apparel shopping: over 4,000 sessions and nearly 15,000 turns. They secretly routed participants to either a helpful “Good” recommender or a confusing “Bad” one. After each turn, users rated satisfaction, frustration, and purchase intent. Those turn-by-turn annotations reveal genuine human reactions you don’t get from scripted bots.
The team tested three simulator styles against that data. One used plain prompting, another relied on in-context learning with example snippets, and a third fine-tuned a Gemini 2.5 Flash model on ConvApparel transcripts. They then applied a three-part validation. First, they compared population-level stats—things like average words per turn or how often users reject a suggestion. Next, they trained a discriminator to score how “human” each simulated conversation felt. Finally, they ran counterfactual tests: simulators saw only “Good” agent data during training, then had to handle a “Bad” agent. A realistic simulator should mirror the jump in frustration and drop in satisfaction that real humans showed.
Results showed clear gaps. The prompt-based model swung too little on frustration and kept offering smooth, helpful responses. The in-context learner did better on style but still lacked genuine annoyance when faced with the “Bad” agent. Only the fine-tuned simulator came close to human-like shifts across all three metrics. That tells us: if you want a user simulator that adapts to novel, annoying behaviors, you need targeted training on real human interactions, not just clever prompting or example snippets.