Experimenting with tiny classifiers for tone and intent

Pipecat Tone is a pair of tiny text classifiers that sit either side of the LLM in a voice pipeline. The tone reader reads how the caller meant each turn — their emotion, sarcasm and intent — and the tone tagger marks how the agent should say its reply, for the TTS. They’re 48.2 MB and 63.4 MB, run on a CPU in about 5 ms a turn and 14 ms a sentence, work with any LLM, and speak the Real-time Context Standard (RCS). We’re still experimenting with these, but they’re ready for people to try.

A text cascade pipeline is still the best way to build a voice agent: you pick the best model for each job, tool calls are reliable, every turn is text you can log and test, and it costs less. But passing text between the steps loses how something was said. An exasperated “oh, great” reaches the LLM looking like a delighted one, and the reply reaches the TTS as flat words.

These models work from transcripts, so they still miss anything that isn’t captured in the words. We think pairing them with small audio models will help: the audio model hears how the user said it, while the text classifiers check that reading against the words and shape the reply.

Why we built them

RCS carries context beyond the words inline, in the same text stream. Perception, how the user said it, heads the user’s message. Expression, how the agent should say its reply, reaches the TTS as tags:

LLM reads:  ([emotion angry=0.90, disgusted=0.05],[sarcasm value=0.94]) Oh great, a forty minute wait.TTS gets:   <<apologetic>> Sorry about the wait. [[light breath]] Your order, <<slow: 48213>>, ships ^^today^^.

The best way to use RCS is an LLM fine-tuned on it: one that knows the standard, and when to use each part of it. Prompting is a weaker substitute. Large models tag accurately, but miss 30% or more of the moments that call for it. Small, fast models do worse: PhoneLLM Mini writes no tags at all.

So these models are a stop-gap, for when you’re running a model that can’t follow RCS, like PhoneLLM Mini, or can’t fine-tune one that does. The LLM writes words, and a small model handles each side of the standard.

Two small models

The tone reader reads each user turn, with the agent’s line before it, for emotion and sarcasm, with calibrated confidences the LLM can act on. The agent’s line matters: “oh, wonderful” after “your delivery is delayed again” is sarcastic. It also reads intent: whether a turn asks, answers, hedges or closes, and whether it’s only a “mm-hmm”. 24.2M parameters in a 48.2 MB fp16 ONNX file, 5 ms a turn.

The tone tagger labels each sentence of the LLM’s plain reply as it streams — a delivery, sounds, slow spans for details like order numbers, stressed words and breaths — and writes them in the TTS’s own names. 17M parameters in a 63.4 MB int8 ONNX file, 14 ms a sentence on one CPU thread.

Both drop into a Pipecat 1.15.0 pipeline:

from pipecat.processors.tone.reader import PipecatToneReaderfrom pipecat.processors.tone.tagger import PipecatToneTaggerpipeline = Pipeline([    transport.input(), stt, PipecatToneReader(), user_aggregator,    llm, PipecatToneTagger(), tts, transport.output(), assistant_aggregator,])worker = PipelineWorker(pipeline, params=PipelineParams(enable_expressive_conversation=True))

The tags stay out of the LLM’s context, so it doesn’t start copying them.

The tagger is also where the small sounds come from: a laugh when the caller jokes, a sigh, a breath before a sentence where a person would take one. We calibrated its breathing against the breath annotations in the ICSI Meeting Corpus. People breathe more in longer turns, so the tagger breathes in 6% of replies of 6–15 words and 44% of those over 60. It chooses a light breath, a deep breath or an exhale in the proportions observed in those conversations.

Metering, and the harness around them

Neither model keeps track of the call: the reader reads one turn, and the tagger labels one sentence. LLMs are prone to switching tone turn by turn — apologetic in every reply, laughing too often, warm one sentence and somber the next. It can sound erratic.

So we meter it, in plain Python around the models. In our RCS demo, a mood meter starts each call at 50 out of 100, and each turn’s reading moves it, weighted by the reader’s confidence. Losing ground is easier than winning it back, and sarcasm counts against it whatever the words say, so the mood moves a turn at a time, and one warm turn doesn’t undo a bad call.

Pipecat’s tagger processor does the same for expression: it tracks whether the agent has already apologized, whether the caller asked it to slow down and whether a detail has been said before, and caps breaths at one a reply.

LLMs writing their own tags drift as a call goes on, because their tagged replies pile up in context as examples. On tone-bench, gpt-5.6-luna scores 18 points worse late in a call than early on; the tagger, with its harness, scores 3 points better.

The models need these rules around them. Even an LLM fine-tuned on RCS needs a way to track what has happened in the call.

What the agent does with what it hears depends on the use case, too. A support agent that hears anger should stay calm, not get angry back; an NPC in a game might well snap back. The tagger’s twelve deliveries are a support agent’s — calm, sympathetic, apologetic, warm and the like, with no anger — and an expression policy narrows them further.

How they score

The reader, against three LLMs’ majority vote, on turns it never trained on:

Given 365 pairs of the same praise, once after a line it’s meant after and once after a line that flips it, it reads both right 94% of the time, up from 49% for the previous release.

The tagger, on tone-bench: eight scripted support calls through a real Pipecat pipeline, judged by Claude Opus 5.5. The score averages expression, perception and conversation. Recall is how often a model tags where the call needs it, holds is late-call checks against early ones, and first word is the median time to the first spoken word. Every other model writes its own tags.

Where they’re strong

• Works with models that don’t write tags. PhoneLLM Mini goes from no tags to the highest score in these tests.
• Consistent tagging in these tests. It slows every detail, laughs on cue every time, and picks a fitting mood in 86–97% of replies, against 60% for the best LLM.
• Fast. 14 ms a sentence lets you keep a fast LLM: first word in a third of gpt-4.1’s time.

Where they’re weaker

• The rules go a long way. The tagger’s rules alone score 90.0. The model earns its place on the harder moments — the rules never laugh — and scores twice as high against its teacher (0.751 against 0.38), but tone-bench needs harder calls.
• Rare moments are hardest. Delivery macro-F1 is 0.64.
• Words, not voices. A flat transcript reads neutral. Hearing the difference takes audio.
• No smarter LLM. On conversation, PhoneLLM Mini scores 85; the GPT models, 89–92.
• Model labels. Scores are against LLM labels, not listeners. English only, and the tagger learned from support calls.

Small models or the LLM?

An LLM has more context about its own reply. The tagger can miss a dry joke. That’s why we’d ultimately prefer a model fine-tuned on RCS. For now, you can use the reader for perception, and either a capable LLM or the tagger for expression.

What’s next

Text classification like this will only ever get you so far. We’re also experimenting with small models that work directly from audio, such as the Pipecat Backchannel Detector, which distinguishes a “mhmm” from an interruption using 734k parameters and about 0.3 ms of CPU per 40 ms of audio. We want to see how much of the gap between a text cascade and a speech-to-speech model we can close by combining models like this with the text classifiers.

A few things we’re working towards:

• Try a small audio model that reads pitch, energy and speech rate, with the text reader checking its reading against the words. RCS already has room for signals like speech_rate.
• Pass the reader’s intent predictions on as RCS grows an intent type, so the LLM knows a hedge from an answer.
• Let the LLM leave a hint for the tagger. One word of intent, like PhoneLLM Max’s microthinking, would tell the tagger a joke is a joke.
• Score against listeners, and write tone-bench calls the rules alone can’t pass.
• Cover more of RCS’s 118 deliveries, more languages and uses beyond support calls.

Both tone models, their data and their training code are BSD-2-Clause: models and data, training code, and the setup guide.