Voice AI evals stop at the network - just when it became more important

You open the eval run. Thirty sessions flagged. You start reading.

Session 12, the agent asked for the account number twice. Session 19, it answered a question nobody asked. Session 27, it talked over the customer and never recovered. You have transcripts, scores, and a latency breakdown per stage. By the third one you are already forming a theory about the prompt.

That is the right place to be looking. The STT-LLM-TTS pipeline is where voice AI is being invented right now, it is where the competition is, and it is where most of your failures genuinely live. Nobody should apologize for pointing eval tooling at the part of the stack that is moving fastest.

I want to talk about two things that can go wrong underneath it, though. Neither one shows up in anything you just read.

Both relate to the network transport. The first between the human and the model. The second... between the model and the human.

The model answers what it heard

Your model does not receive what the user said - it receives what arrived.

Between the two sits a network. Audio gets encoded, packetized, and sent across a connection that has opinions about timing. Some packets arrive late. Some arrive out of order. Some do not arrive at all, and something has to decide what to play in their place. By the time your speech-to-text stage sees audio, it may be a slightly different utterance than the one that left the user's microphone.

This goes through your VAD. It gets handled by turn detection. All that before hitting your STT. And then? The model reasons over that. And it responds sensibly to it, which is the part that makes this hard to catch. It is not confused, it is not hallucinating, it is not ignoring your prompt. It answered the question it was asked. The question it was asked was just not the question that was spoken.

Read the transcript afterwards and you see a model that went off script. Read what actually arrived and you see a model that behaved correctly on bad input.

The human hears something you never sent

Now run it backwards.

Your pipeline produced a good response. The text was right, the voice synthesis was clean, the timing was inside budget. All of that happened on your side of the connection. Then it goes out over the same network, and the user hears a clipped word, a half second of nothing, or a syllable that lands as something else.

They ask you to repeat it. Probably barging in. The model continues, missing the context of a poor network altogether. They answer the wrong question. They get frustrated and hang up.

Your logs show a correct response. Your eval shows a correct response. The user experienced a broken one. Both of those things are true at the same time, and only one of them is written down anywhere.

Neither one leaves a mark in the transcript

Here is what these two have in common, and it is the whole point.

In both cases the text is clean. The transcript of a session where audio arrived degraded looks like a transcript of a session where the model simply got it wrong. The transcript of a session where the user misheard a perfect answer looks like a transcript of a session that went fine.

That is not a gap in anyone's eval tooling. It is a property of the medium. A layer that scores conversations reads text, and there is nothing in the text to find. You could build the best conversational scoring system in the world and it would still be blind to both of these, because the evidence never enters the format it reads.

So how often does it happen? I don't know, and I have not found anyone who does.

While these things have been tested in labs, and through some controlled experiments, they are not part of the way evals are conducted today. Most of these pipelines don't work on real networks with real impairments. And when they do, they are tested separately from the model's eval.

The operational question is narrower and important to pause on: of the sessions your eval flagged this week, which ones were caused by transport issues? Answering that needs per-session transport data lined up against eval outcomes, and almost nobody is capturing the first half. That is the actual gap. The physics is not mysterious - the attribution is missing.

Full duplex removed the slack

You could argue this mattered less a year ago, and you would have a point.

Turn-taking architectures put silence between turns. The user speaks, stops, and something decides they are done. Then the agent speaks. That silence did quiet work: it was slack in the system, and slack absorbs timing variance. A packet arriving late during a gap where nothing was happening cost nothing. There was room for the network to misbehave without anyone noticing.

Full duplex takes the room away. When audio is streaming continuously in both directions, there is no gap to absorb anything. Timing problems stop landing in the quiet parts, because there are no quiet parts.

This is why the layer underneath matters more now than it did. Not because networks got worse. Because the architecture stopped hiding them.

Model, network, or device

Which brings you to a question your current tooling cannot answer.

When a session goes badly, there are three candidates. The model did something wrong (I treat the whole AI pipeline as "the model" here). The network degraded the audio in one direction or the other. Or the device did something ugly: a microphone that clipped, an echo canceller that ate half a word, a phone that switched networks mid-call.

Three candidates, and the honest position today is that most teams cannot separate them. You get a bad session, you have transcript evidence for exactly one of the three, and the other two are invisible. So the fix goes where the evidence is. Sometimes that is right. Sometimes it is a prompt change for a problem that was never in the prompt.

Getting to a single clean answer on that three-way split is what we are building toward at rtcStats. Our focus is on figuring out where failures and breakage live in WebRTC sessions.

What the transport layer actually knows

The layer knows a thing or two. Or three.

WebRTC reports on itself continuously. Every connection tracks what was sent, what arrived, how late it was, what got lost, what the receiver did to paper over the gaps, which network path got selected, and what the encoder decided along the way. Browsers expose this through an API called getStats(), and browsers also ship a page called webrtc-internals that can dump all of it to a file.

If you have never opened one of those files, be glad. It is thousands of metrics sampled over the life of a call, with no narrative and no verdict. The data that explains your session is in there. So is everything else.

rtcStats reads the dump for you and returns four things:

• Observations are the moments that mattered. The specific points in the session where something happened worth knowing about.
• Deductions are what those signals mean for the person on the call. Not "packet loss at 00:41" but what that did to their experience.
• Experience Score is how the call felt, in a single number.
• AI Summary is the whole session in plain English.

You get all four in under 30 seconds, which is the difference between checking a session and deciding not to bother.

Point it at a session that your eval flagged and you get the half of the story that your eval structurally cannot see. Not a replacement for conversational scoring. The other axis.

One thing worth saying plainly, because it is the first question I get asked. The collection layer is open source: rtcstats-js goes in your client, rtcstats-server runs inside your own infrastructure, and you decide which sessions get captured. Nothing sits in your call path, and what gets collected is metadata about the connection, never what was said on it.

The part I would like to hear about

If you are shipping voice AI on WebRTC today, you have almost certainly had a session that everyone agreed was broken and nobody could explain. The transcript looked fine. The traces looked fine. The customer was unambiguous that it was not fine.

What did that one turn out to be, in the end? Or is it still open?

If it is still open, I would genuinely like to see it.

https://rtcstats.com