Self-correction, multi-step instructions, Pipecat Flows — and a separate count of the mistakes that break a call.
PhoneBench evaluates LLMs on their fitness for phone agents: multi-turn tool calling and dialogue across realistic customer-service scenarios, graded by a panel of LLM judges — calibrated against human labels — that compare every response to a high-quality reference. The judges measure points of quality that would normally require human labeling: telephone speaking style, tool-call accuracy, say/do consistency, factual grounding, conversation coherence, authentication and escalation discipline, and caller outcome. The scenarios, tool lists, and prompts are kept separate from the data PhoneLLM was trained on. Alongside accuracy, the bench measures time-to-first-answer-token under production-style serving and estimates cost per minute of conversation.
In Alpha 2, we’ve updated PhoneBench to cover several new cases the previous model struggled with. Among other things, PhoneBench now checks whether the model can correct mistakes from earlier in the conversation, follow sequences of instructions, and handle Pipecat Flows scenarios. We also have separate measurements for how often the model makes a mistake when calling a tool, and how often the model makes a major mistake of any kind during the conversation.
PhoneLLM Max Alpha 2 takes the top overall score and the lowest major-error rate of any model, and PhoneLLM Mini Alpha 2 outscores every other model with reasoning turned off.
| Model | PhoneBench scorehigher is better | Tool-call failureslower is better | Major-error turnslower is better |
|---|---|---|---|
| 01PhoneLLM Max Alpha 2microthinking on | 83.0% | 3.1% | 4.9% |
| 02Claude Sonnet 5.5thinking minimal | 80.3% | 1.6% | 8.5% |
| 03PhoneLLM Mini Alpha 2thinking off | 75.6% | 5.9% | 7.9% |
| 04Gemini 3.6 Flashthinking minimal | 73.9% | 3.5% | 6.8% |
| 05Qwen3.8 27Bthinking off | 71.4% | 8.0% | 11.5% |
| 06GPT-5.6 Lunathinking off | 69.5% | 10.2% | 9.5% |
| 07GPT-5.6 Terrathinking off | 69.1% | 11.9% | 11.2% |
| 08Claude Haiku 5.5thinking off | 66.3% | 4.9% | 11.4% |
| 09Nemotron 3.5 Super VL 120B basethinking off | 65.1% | 14.5% | 17.9% |
| 10DeepSeek V4 Flash 0731thinking off | 65.0% | 5.7% | 14.9% |
| 11GPT-4.1thinking off | 63.3% | 7.0% | 11.6% |
| 12PhoneLLM Alpha 1thinking off | 57.2% | 15.3% | 12.5% |
| 13Gemma 4 31Bthinking off | 56.9% | 8.9% | 24.5% |
| 14Nemotron 3 Nano 30B basethinking off | 29.7% | 20.3% | 48.5% |
Rows ordered by PhoneBench score — the judge panel’s aggregate grade of each model’s responses against the reference set. The label beside each model is the reasoning setting it ran with. Tool-call failures is the share of tool calls the model gets wrong; major-error turns is the share of conversation turns with a major mistake of any kind. Base rows are untuned Nemotron base models. See the announcement for how PhoneLLM Alpha 2 was trained. The full harness and methodology will be published as PhoneBench matures.