The pause after you stop talking is what gives an AI caller away

We have not put a phone assistant on a client's line yet. What we have done is build real-time voice for Arthur, the assistant we run MUVES on, and spend weeks making him feel less like a machine reading a script and more like a conversation. This is what we learned, and what we would listen for in any voice assistant before trusting it with your customers.

The gap

The first thing people notice is not the voice. It is the pause after they stop talking. Voice research is consistent on this: around half a second of silence after you finish a sentence feels like a delay, while a fifth of a second does not register.

Most assistants are built as a chain where every link waits for the one before. Wait until the caller stops. Turn the speech into text. Send the text to the model and wait for the whole answer. Turn the answer into speech, then play it. Each step is quick; together they add up to a pause long enough for callers to say "hello?" Better systems overlap the steps, and start speaking the first sentence while the rest is still being written.

Knowing when you have finished

How does the assistant decide you have stopped talking? The simple way is a silence timer. Our first version of Arthur waited a fixed time after each phrase, longer after a full stop. That works until someone pauses to think, or lists three things slowly, and gets cut off mid-thought. "I'd like to book for..." is not finished, however long the pause. Better turn detection listens to what was said, not only to the silence, and we have been researching small open models built for exactly that.

Being interrupted

People interrupt each other constantly, and a good conversation survives it. For an assistant, being interruptible is real engineering. When the caller starts talking it has to stop playing audio, cancel the speech still being generated, cancel the reply being written, and reset. Miss one and it either talks over the caller or stops and then finishes its old thought.

The opposite problem matters as much. Not every sound is an interruption. "Mhm", "yeah" and a cough mean "keep going." We rebuilt Arthur so that any sound over him pauses him first. If it turns out to be "mhm" or a cough, he carries on. If it is a real sentence, the caller gets the floor. A real interruption is answered as fast as before, and a false one costs nothing.

A phone assistant does not have to sound human. It has to keep up with one.

Echo

On a speakerphone, the assistant's own voice comes back through the microphone. If it is not careful, it hears itself, decides the caller is interrupting, and stops mid-sentence. We hit exactly this bug: after one interruption on speakers, Arthur's echo guard did not switch back on for the next reply. A code review found it along with two other faults, and each fix came with a test that fails if the fix is ever removed. Arthur's voice now runs 31 interruption checks and 21 voice screen checks with every change.

Knowing what it does not know

A natural voice that says something wrong is worse than a stiff one that says "let me take a message." A phone assistant should answer only from information you have approved, say when it does not know, and hand off to a person whenever the caller asks. We cover that side in what an AI receptionist should handle.

What to try before you buy

Call the assistant yourself, the way your customers will:

• Ask a question and count the silence. If you notice it, so will they.
• Pause in the middle of a sentence. Does it wait or jump in?
• Interrupt it. Does it stop cleanly, or finish its thought?
• Say "mhm" while it talks. Does it keep going?
• Call on speakerphone. Does it cut itself off?
• Ask something it should not know, then ask for a person. How fast do you get one?

If you want one built, our phone answering agents page explains how we would set it up and test it on real calls before it ever answers a customer. The voice assistant case study has the technical detail.