Introducing PhoneLLM Alpha 2

Today we’re releasing Alpha 2 of Pipecat PhoneLLM: a blazingly fast, open-weights text model designed to be integrated into voice agent pipelines.

This time, we’re releasing two variants of the model:

The key improvements in Alpha 2, aside from some major accuracy enhancements due to our expanded dataset, are the addition of microthinking to PhoneLLM Max, support for Pipecat Flows integration, and reduced sensitivity to different prompting styles, making the model more flexible.

Microthinking

Most LLMs which emit reasoning tokens generate several paragraphs (or even pages) of thought for each request. This increases cost and latency, and high accuracy with reasoning disabled was a major goal for PhoneLLM Alpha 1.

In Alpha 2, we’ve found that allowing PhoneLLM Max to output a single sentence of reasoning has a measurable impact on whether it calls the correct tools. We refer to this as “microthinking”.

Microthinking is a good compromise between the accuracy of full reasoning mode, and the latency/performance of having reasoning disabled. A single sentence, when emitted at PhoneLLM’s high inference speed, has a negligible impact on response time.

Pipecat Flows support

Pipecat Flows is a framework for structured conversations with voice agents. LLMs are nondeterministic and often make mistakes when following instructions, and a programmatic harness like Pipecat Flows can reproducibly guide the LLM through a complex sequence of steps, while keeping each individual step simple.

Smaller, cheaper models like PhoneLLM are a perfect fit for this: the constraints and guidance of the harness reduce the likelihood of mistakes, as the model doesn’t need to make complex decisions on its own.

We’ve trained PhoneLLM Alpha 2 on synthetic Pipecat Flows sessions, meaning it already understands the prompting format and conversation structure, and can be used as a drop-in replacement for larger models.

In our testing with real customers, we’ve found that PhoneLLM Max brings calls to a successful resolution at a higher rate than GPT-4.1 when integrated into Flows, with around half the latency and a fraction of the cost.

More flexible prompting

For Alpha 2 we’ve expanded our training dataset to cover a wider range of use cases and prompting styles.

This means less time spent tweaking your prompts to get the most out of PhoneLLM.

PhoneBench updates

For this release we’ve improved our PhoneBench benchmark to cover several new cases the previous model struggled with.

Among other things, PhoneBench now checks whether the model can correct mistakes from earlier in the conversation, follow sequences of instructions, and handle Pipecat Flows scenarios.

We also have separate measurements for how often the model makes a mistake when calling a tool, and how often the model makes a major mistake of any kind during the conversation.

Our training process

Like Alpha 1, the new PhoneLLM Mini is based on NVIDIA’s Nemotron 3 Nano, with 30B total parameters, and 3.5B active parameters. PhoneLLM Max is built on Nemotron 3.5 Super VL, a 120B model with 12B active parameters.

We use NVIDIA’s NeMo Framework for training, specifically Megatron Bridge and NeMo RL. The NeMo framework has great support for the Nemotron models, and takes the hassle out of finding a working training configuration.

It also makes it easy to scale training across multiple GPUs and nodes: to train PhoneLLM Max, we need 16x B300 GPUs across two machines.

PhoneLLM is trained on fully synthetic datasets, designed to mimic real-world voice agent use cases. We use a variety of open-weights models as teachers — these teacher models generate high quality responses which the student model is then taught to imitate.

To make the model as flexible as possible, we train on a variety of data shapes: different prompting styles, tool sets, harnesses such as Pipecat Flows, and so on.

So far, our PhoneLLM releases have been full-weights fine-tunes using SFT, which we’ve found is fast, relatively inexpensive, and gives predictable improvements. We have ongoing experiments with reinforcement learning — for example, we’ve seen some improvements by applying a final GRPO stage to our training pipeline, and it’s likely that we’ll enable this for future releases of the model once we’re confident in the quality.

Running the model

PhoneLLM is an open-weights model. Our partners AWS, Baseten, Cloudflare, and Modal offer PhoneLLM as click-to-deploy models and hosted API endpoints.

Inference partners

• 

  AWS SageMaker AI

  Deploy to AWS SageMaker inference clusters. Your coding agent can guide you through deploying to SageMaker.
• 

  Baseten

  PhoneLLM is a supported model in Baseten’s Dedicated Inference feature, allowing you to deploy an optimized PhoneLLM instance from the Baseten CLI or dashboard.
• 

  Cloudflare

  PhoneLLM Mini on Cloudflare’s global Workers AI serverless inference platform, priced at $0.06 per million input tokens and $0.24 per million output tokens.
• 

  Modal

  Start an optimized endpoint for Mini or Max using Modal’s Endpoints service. See the PhoneLLM entries in Modal’s model library.

Weights

You can run the PhoneLLM models on your own hardware, or set up PhoneLLM inference clusters from scratch in the cloud. Links to model weights:

vLLM

When making requests, set temperature to 0.0. For Mini, turn reasoning off, and for Max, turn reasoning on to enable microthinking.

vllm serve pipecat-ai/phonellm-mini-alpha-2 \
  --served-model-name phonellm-mini-alpha-2 \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-ssm-cache-dtype float32 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

vllm serve pipecat-ai/phonellm-max-alpha-2 \
  --served-model-name phonellm-max-alpha-2 \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-ssm-cache-dtype float32 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3