Forty-six configs, one 30-turn script — and accuracy only counts on the clock.
The aiwf_medium_context benchmark runs every model through the same scripted 30-turn conversations: a Pipecat agent that has to call tools, follow standing instructions, and stay grounded in a knowledge base, with every turn judged on all three at once. A turn passes only if nothing fails, and rows are ranked by that strict pass rate. Alongside accuracy the harness measures TTFAT — time from request start to the first user-visible answer token or tool-call output, with separately-streamed reasoning excluded. The repo’s rule of thumb: an optimized voice pipeline adds roughly 500 ms on top of the LLM, so a model much above 700 ms TTFAT is too slow for most voice agents.
Rank on accuracy alone and four configurations tie at 100% — but a voice agent only gets credit for answers that arrive on time. Judge accuracy inside the ~700 ms budget and the leaders change: qwen3.8-27b (98.2% at a 649 ms median) and claude-haiku-4-5 (98.0% at 637 ms) are the strongest hosted configs that actually fit, and the NVFP4 quantization of qwen3.8-27b hits 97.8% at a 101 ms median on a single RTX 5090 — though over localhost, with no network hop. Of the perfect scorers, only claude-sonnet-4-6 comes near the clock at an 850 ms median; claude-fable-5’s always-on thinking puts its 100% past 3.5 s, off the board for voice. The inverse failure is just as real: gpt-oss-120b on Groq answers in 98 ms but passes 86.3%, and the fastest row in the field, nemotron-3.5-lightning with thinking off at 62 ms, fails half its turns.
- Configs scored
- 46 · 7 providers · 30 scripted turns each
- Best inside 700 ms
- 98.2%
- qwen3.8-27b (FP8)98.2% · 649ms
- claude-haiku-4-598.0% · 637ms
- qwen3.8-27b (NVFP4, local)97.8% · 101ms
- Perfect score, off the clock
- 100.0%
- nemotron-3-ultra (128)541ms †
- claude-sonnet-4-6850ms
- claude-fable-5 (low)3535ms
- claude-fable-5 (default)3956ms
- Fastest at 97%+
- 101msqwen3.8-27b NVFP4 · local RTX 5090
All 46 configs · ranked by strict pass rate, then TTFAT P50 · each conversation's first scripted response excluded · log axis
| Model | Pass rate | Tool err | Instr err | KB err | P50 (ms) | P95 (ms) | Max (ms) | Provider |
|---|---|---|---|---|---|---|---|---|
| 01nemotron-3-ultra (128) † | 100.0% | 0.0% | 0.0% | 0.0% | 541 | 712 | 1,302 | Baseten |
| 02claude-sonnet-4-6 | 100.0% | 0.0% | 0.0% | 0.0% | 850 | 4,126 | 9,396 | Anthropic |
| 03claude-fable-5 (low) | 100.0% | 0.0% | 0.0% | 0.0% | 3,535 | 5,148 | 8,815 | Anthropic |
| 04claude-fable-5 (default) | 100.0% | 0.0% | 0.0% | 0.0% | 3,956 | 6,496 | 13,602 | Anthropic |
| 05glm-5.2 (none) | 99.7% | 0.2% | 0.2% | 0.0% | 936 | 2,140 | 7,567 | Baseten |
| 06nemotron-3-ultra (96) † | 98.3% | 1.3% | 1.3% | 0.3% | 529 | 655 | 1,259 | Baseten |
| 07kimi-k2.6 (thinking on) | 98.3% | 1.4% | 1.7% | 0.0% | 1,560 | 5,404 | 13,622 | Baseten |
| 08qwen3.8-27b (thinking off, FP8) | 98.2% | 1.8% | 1.8% | 0.0% | 649 | 801 | 2,161 | Baseten |
| 09claude-haiku-4-5 | 98.0% | 0.7% | 2.0% | 0.0% | 637 | 1,615 | 3,152 | Anthropic |
| 10gpt-5.1 | 98.0% | 2.0% | 2.0% | 0.0% | 739 | 1,492 | 4,244 | OpenAI |
| 11qwen3.8-27b (thinking off, NVFP4) | 97.8% | 1.9% | 2.2% | 0.1% | 101 | 318 | 592 | Local RTX 5090 |
| 12gpt-5.6-terra (medium) | 97.8% | 1.9% | 2.2% | 0.0% | 927 | 2,149 | 4,167 | OpenAI |
| 13gpt-5.5 (none) | 97.4% | 2.0% | 2.6% | 0.0% | 875 | 2,177 | 5,623 | OpenAI |
| 14qwen3.6-27b (thinking off) | 97.3% | 2.7% | 2.7% | 0.0% | 667 | 769 | 1,920 | Baseten |
| 15deepseek-v4-pro-0813 (low) | 97.3% | 2.7% | 2.6% | 2.0% | 752 | 1,477 | 3,545 | Baseten |
| 16gemini-3.6-flash (minimal) | 97.1% | 2.4% | 2.8% | 0.1% | 798 | 984 | 1,472 | AI Studio |
| 17nemotron-3-super-120b (512) † | 97.0% | 1.0% | 3.0% | 0.3% | 687 | 1,210 | 2,254 | Baseten |
| 18gpt-5.4 (low) | 97.0% | 3.0% | 3.0% | 0.0% | 782 | 1,706 | 2,698 | OpenAI |
| 19deepseek-v4-flash-0731 (low) | 96.7% | 2.8% | 3.2% | 0.7% | 677 | 1,452 | 4,687 | Baseten |
| 20gemma-4-31b-it (thinking off) | 96.6% | 3.3% | 3.4% | 0.0% | 489 | 609 | 38,250 | Baseten |
| 21gpt-5.6-sol (none) | 96.6% | 3.3% | 3.3% | 0.1% | 1,098 | 2,625 | 6,344 | OpenAI |
| 22gpt-4.1 | 96.3% | 3.7% | 3.3% | 0.0% | 536 | 1,771 | 5,056 | OpenAI |
| 23gpt-5.4 (none, +96 dots) | 95.2% | 4.7% | 4.6% | 0.1% | 694 | 2,273 | 17,264 | OpenAI |
| 24inkling (none) | 94.8% | 5.1% | 4.8% | 1.3% | 447 | 727 | 1,813 | Baseten |
| 25gpt-4o | 94.7% | 3.0% | 5.0% | 0.3% | 546 | 1,369 | 4,897 | OpenAI |
| 26kimi-k2.6 (thinking off) | 93.9% | 6.0% | 3.9% | 0.0% | 475 | 842 | 4,458 | Baseten |
| 27deepseek-v4-flash-0731 (high) | 93.9% | 5.9% | 6.1% | 4.8% | 763 | 1,871 | 8,702 | Baseten |
| 28nemotron-3.5-lightning (thinking on, NVFP4) | 93.6% | 2.8% | 5.7% | 0.9% | 1,464 | 5,787 | 29,869 | Local RTX 5090 |
| 29gemini-3.5-flash (minimal) | 93.3% | 5.3% | 6.7% | 4.9% | 892 | 1,183 | 1,721 | AI Studio |
| 30claude-sonnet-5 | 93.0% | 7.0% | 7.0% | 0.0% | 1,204 | 2,465 | 6,955 | Anthropic |
| 31qwen3.6-35b-a3b (thinking off, FP8) | 91.6% | 6.8% | 7.8% | 0.4% | 764 | 1,233 | 35,664 | Baseten |
| 32gpt-5.6-terra (none) | 91.3% | 8.1% | 8.6% | 0.3% | 621 | 1,870 | 5,665 | OpenAI |
| 33nemotron-3-nano-30b (512) † | 90.6% | 5.0% | 6.1% | 4.0% | 940 | 1,912 | 2,821 | Baseten |
| 34gpt-5.4 (none) | 90.2% | 9.4% | 9.7% | 0.1% | 689 | 1,723 | 6,571 | OpenAI |
| 35gemini-2.5-flash (thinking off) | 89.9% | 9.1% | 10.1% | 0.0% | 550 | 850 | 2,352 | AI Studio |
| 36gpt-5.2 | 89.3% | 10.0% | 10.7% | 0.7% | 624 | 1,171 | 2,509 | OpenAI |
| 37gpt-5.6-luna (none) | 88.3% | 11.7% | 11.7% | 0.0% | 671 | 2,304 | 12,017 | OpenAI |
| 38gpt-oss-120b (groq) | 86.3% | 9.3% | 13.0% | 0.7% | 98 | 217 | 2,117 | Groq |
| 39muse-glimmer-30b (thinking low, GGUF) | 86.1% | 13.0% | 13.7% | 0.0% | 231 | 1,752 | 5,474 | Local RTX 5090 |
| 40poolside/laguna-s-2.1 (thinking off) | 85.6% | 13.7% | 11.2% | 5.7% | 295 | 620 | 21,032 | OpenRouter |
| 41gpt-4.1-mini | 85.3% | 14.7% | 14.7% | 0.0% | 851 | 2,135 | 5,945 | OpenAI |
| 42gpt-5-mini | 83.7% | 14.0% | 16.3% | 1.0% | 682 | 1,132 | 1,904 | OpenAI |
| 43gpt-4o-mini | 82.7% | 10.3% | 13.7% | 2.3% | 553 | 1,947 | 6,497 | OpenAI |
| 44gemma-4-26b-a4b-it (thinking off) | 80.7% | 13.9% | 19.3% | 0.9% | 578 | 634 | 31,574 | Baseten |
| 45gemini-3.5-flash-lite (minimal) | 68.6% | 30.8% | 31.4% | 28.1% | 591 | 679 | 928 | AI Studio |
| 46nemotron-3.5-lightning (thinking off, NVFP4) | 50.9% | 49.0% | 47.9% | 38.9% | 62 | 70 | 80 | Local RTX 5090 |
Rows ordered by strict turn pass rate, then TTFAT P50, exactly as published in leaderboard-medium-context.md. A turn fails if any of tool use, instruction following, or KB grounding fails; the three error columns overlap, so they don’t sum to the pass-rate complement. TTFAT is measured by the Pipecat service and excludes each conversation’s first scripted response. † The four Nemotron 3 rows are historical Baseten deployments whose latencies are legacy TTFT summaries the repo marks as not comparable with the corrected TTFAT campaigns — they sit out every latency-based cut above (nemotron-3-ultra (96), at 98.3% with a legacy 529 ms TTFT, would otherwise lead the budget tile). Local RTX 5090 rows measure wall-clock over localhost to a dedicated batch-one GPU, with no provider network time. Per-row campaign reports — e.g. the Nemotron 3.5 Lightning binary campaign and the Muse Glimmer reasoning-strength sweep — are linked from the repo README.