Voice agent latency: the 800 ms budget

The budget

For a voice turn, I use a target budget, not a measured result: about 800 ms from end of speech to start of response. The budget is split across speech recognition, the model, any tool call, and speech output.

Stage	Target
Speech in and transcript	200 ms
Model first token	300 ms
Tool call	150 ms
Speech out start	150 ms

These are targets. The measured values are TODOadd p50 and p95 by stage.

Where it goes wrong

The model waits for a tool that waits for a network call that has no timeout. The user hears silence and talks again. The agent now has two overlapping turns.

The fix

• Give every tool a timeout.
• Start a short filler only when the wait is expected.
• Cancel the in flight turn when the user speaks again.
• Log the stage timings on every turn.
• Test no input, barge in, tool error, and timeout.

async def timed(stage, fn, *args):
    start = time.perf_counter()
    try:
        return await fn(*args)
    finally:
        log_latency(stage, (time.perf_counter() - start) * 1000)

Lesson

Voice quality is a latency product. The prompt cannot fix a two second tool call.