Muratcan Koylan (@muratcan) on X
Short summary
Researchers tested an AI system called Jev on 2,029 real phone calls stripped down to pure call structure—no audio or transcripts—and it learned to predict which calls would result in bookings with 94% accuracy by the end, all for $3 in compute costs.
- Jev made 38,012 turn-level forecasts across the calls at 118ms median latency, processing thousands of calls in seconds without any fine-tuning or examples from the company's data.
- The system achieved 0.78 AUC by the halfway point and reached 94% ranking accuracy near the end, learning to separate bookable from non-bookable calls based solely on call structure, timing, and workflow patterns.
- The experiment revealed the AI could extract predictive signal from call metadata alone—turns, tool calls, workflow stages—demonstrating that meaningful patterns exist independent of what's actually said.
Full summary
Researchers tested an AI receptionist called Jev on 2,029 real phone calls without giving it access to transcripts or audio. Instead of hearing what was said, Jev worked with stripped-down structural data—conversation turns, tool calls, workflow stages, and timing information. During the calls, it made 38,012 predictions at a median latency of 118 milliseconds, asked five typed questions about each call, and generated 10,145 answers within 26 seconds while managing 256 simultaneous requests.
The test ran zero-shot, meaning Jev had no fine-tuning or examples from the team's own data to learn from. Researchers then checked Jev's forecasts against actual outcomes recorded in the EHR (electronic health record). By the midpoint of testing, it could meaningfully distinguish calls that would result in bookings from those that wouldn't, hitting an AUC score of 0.78. By the end, it ranked calls correctly 94% of the time.
The system did overweight certain visible errors that the agent typically handles well anyway, but the core result remains striking: analyzing thousands of real calls in seconds cost only $3. The experiment shows how much predictive signal exists in call structure alone, independent of the actual conversation content.