PhoneBench Alpha 1: phone-assistant tool calling and dialogue

PhoneBench evaluates LLMs on their fitness for phone agents: multi-turn tool calling and dialogue across realistic customer-service scenarios, graded by a panel of LLM judges — calibrated against human labels — that compare every response to a high-quality reference. The judges measure points of quality that would normally require human labeling: telephone speaking style, tool-call accuracy, say/do consistency, factual grounding, conversation coherence, authentication and escalation discipline, and caller outcome. The scenarios, tool lists, and prompts are kept separate from the data PhoneLLM was trained on. Alongside accuracy, the bench measures time-to-first-answer-token under production-style serving and estimates cost per minute of conversation.

Gemini 3.6 Flash takes the top score. PhoneLLM 30B Alpha 1 matches GPT-5.6 Terra to within a tenth of a point — roughly 94% cheaper per minute, with P95 first-token latency about 1,300 ms faster — and lifts the untuned base model it was trained from off the bottom of the board, from 28.6% to 72.3%.