Voice agents and AI agent work

Result: the agents use typed tools, explicit fallback paths, and trace logs. Task success and latency are add task success and p95 latency.

Problem

An agent that sometimes works is hard to evaluate and harder to trust. A voice agent adds a second problem: latency. A good answer that arrives too late breaks the conversation. This work focuses on the boring parts that make agents reviewable: tool contracts, traces, timeouts, and a task set.

Data

The task set is add task count and success criteria. The transcript set is add source and size. No customer recordings are used. The demo recordings will be synthetic or consented.

Approach and why

Parse the user turn into an intent and a small set of candidate actions.
Call tools with typed inputs and validate the outputs against a schema.
Log every step: input, tool, arguments, result, latency, and fallback reason.
Return a clear fallback on timeout, low confidence, or tool error.
For voice, set a per stage budget for speech in, transcript, model, tool, and speech out.

I chose typed tools because free form loops are hard to inspect. A trace with a failed schema is a bug report. A trace without one is a story.

Results

Metric	Value	Notes
Task success	add result	add task set size
p50 latency	add measured value	By stage if possible
p95 latency	add measured value	By stage if possible
Tool error rate	add result	Validation failures and timeouts

Evaluation

Leakage: the task set is separate from the examples used to tune prompts.
Baseline: a single turn, no tool baseline so the agent loop has to earn its complexity.
Validation: a frozen task set with a written success condition and a failure taxonomy.
Metric: task success plus p50 and p95 latency. Cost per successful task when a paid model is used.
Reproducibility: versioned prompts, tool schemas, and traces.

A reviewer should challenge any "agent" claim without a transcript, a tool call trace, or a task set.

Limitations

The task set is small. The measurements are pending. Model behavior can drift when a provider updates a model. The fallback path is only as good as the fallback copy and the recovery flow.

Next steps

Publish one success and one failure transcript, measure latency by stage, add a cost per successful task, and test the fallback on no input, ambiguous input, tool error, and timeout.