Tear-down: how we build an AI voice agent that books jobs

A voice demo is easy. A voice agent you'd route a real business's phone number to is not. The difference is almost entirely in the parts you don't hear: what happens when the caller interrupts, when the calendar API times out, when someone asks a question the agent shouldn't answer. This is how we build one that survives contact with real callers.

The stack, in layers

An inbound voice system is four layers stacked together, and each one is a place it can break:

• →Telephony — the phone number and the media stream. The call has to connect, stay connected, and hand audio to the agent with low latency. This is where a cheap setup betrays itself: lag makes the agent talk over people.
• →The voice agent — speech-to-text, the language model that decides what to say, and text-to-speech. The model is the brain, but the turn-taking (knowing when the caller has finished) is what makes it feel human.
• →Actions — checking a real calendar, writing a booking, sending a confirmation text. This is where the agent stops being a chatbot and starts being useful.
• →Guardrails and handoff — knowing what it must not do, and escalating cleanly when it hits that edge.

The decisions that actually matter

Most of the build isn't prompt-writing. It's a handful of design decisions that determine whether the thing is trustworthy:

Latency budget. Every layer adds delay, and past roughly a second of silence a caller thinks the line dropped. We measure the round trip end to end and cut wherever it's slowest, because a fast, slightly plainer agent beats a smart, laggy one every time.

Scope. The agent should do a few things reliably, not everything badly. Answer, qualify, book, take a message, escalate. Every extra capability is a new way to fail on a live call, so we ship the narrow version and widen it only once the core is solid.

Grounding. The agent can only quote what it's been given — real services, real prices, real availability. It should never improvise a price or invent a policy. When it doesn't know, the correct behaviour is to say so and take a message, not to guess.

The rule we build to: the agent is allowed to be limited, but it is never allowed to be wrong. A “let me have someone call you back” is a fine outcome. A made-up quote is not.

The failure modes we design for

1. 01The interrupt. Real people talk over the agent. It has to stop, listen, and pick up the thread — not plough through its scripted line.
2. 02The API failure. If the calendar is unreachable, the agent takes a message and promises a callback rather than dropping the caller.
3. 03The out-of-scope ask. A legal question, a complaint, a request it can't handle — recognise the edge and hand to a human.
4. 04The silent caller. Background noise, a butt-dial, a nervous pause — the agent needs graceful timeouts, not an infinite “are you still there?” loop.

None of this is exotic. It's the difference between a system that works in a screen-recording and one that works on a Tuesday night when a real customer with a real problem calls a real business. That gap — demo to dependable — is most of the work, and it's the part worth paying for.