Voice agents · AI agents · Built in-house

A voice assistant that knows a cough from an interruption.

Built in-house at MUVES for Arthur, the assistant at the front of MUVES OS. The hard part of talking to a machine is not the words. It is knowing whose turn it is.

A desk speaker whose speech bubble loops back into the microphone beside it, so the assistant answers its own voice.

AI agents, at work.

See the ai agents services

Built: Real-time voice in the browser: our own mic path, turn-taking and barge-in
Stack: Whisper, run locallyWeb AudioBrowser echo cancellingStreaming speech
Timeline: August 30 to September 25, 2026
Status: Built and tested in code. A fresh real-microphone test is due after a dashboard rebuild
Scale: 31 interruption checks, 21 voice screen checks

Illustration: two speech bubbles overlapping, one politely stepping back

The problem

It kept answering itself.

The first version used the browser's own speech engine, which opened its own microphone. No echo cancelling reached it, so on speakers Arthur heard his own voice and replied to it.

The opposite problem was just as bad. A fixed pause length either cut people off while they were thinking, or made every answer feel slow. And any noise at all, a cough, an "mhm", stopped him mid-sentence.

What we built

Turn-taking as a set of states, not a timer.

The microphone decides quickly that a sound stopped. A separate step decides whether the thought finished. While Arthur talks, a third step decides whether a sound is a real interruption.

How it fits together. The top row is one turn. The bottom row runs only while the assistant is speaking. Red boxes are where it protects the conversation: from noise, from its own echo, and from remembering words you never heard.

Our own microphone path
We open the microphone ourselves with the browser's echo canceller switched on, so the speaker's sound is subtracted before we ever see it. That one change is what made talking over him possible.
A noise floor that moves
Speech starts when the sound rises well above the room's own noise, measured continuously. A quiet office and a busy one both work without retuning.
A fast "sound stopped"
A short silence ends a sound. That says nothing yet about whether the person is done.
A "did they finish?" check
Ending on words like "and", "because" or "the" means someone is reaching for the next word, so the assistant waits. A complete sentence gets an answer straight away.
Speech written down on our own machine
Whisper transcribes on the PC rather than in a cloud service, including the words a cloud engine might tidy away.
A raised bar while speaking
While Arthur talks, a sound has to be louder to count, so the trace of his own voice that leaks back is ignored. If a transcript matches what he just said, it is marked as an echo and dropped.
Barge-in that respects memory
A real sentence stops the audio at once. Then he remembers only what you actually heard. If the whole reply had arrived, the cut-off point is worked out from the audio. If not, he records only how many seconds you heard, rather than guess.

The tuning

The numbers behind "feels natural".

Starting points for a desk microphone in a quiet room. The noise floor adapts around them.

Setting	Value	Why
Speech starts at	3.2 times the noise floor	Above room noise, below a normal voice
While he is talking	2.6 times, on top of echo cancelling	Separates "you cut in" from "his voice leaked back"
A sound ends after	600 ms of quiet	Short on purpose. The finish check decides the rest
Shortest real speech	250 ms	Anything shorter is a cough or a click
Audio kept from before speech	320 ms	So the first syllable is not lost
Longest single turn	20 s	A stuck gate cannot record forever

What it sounds like

One exchange, four kinds of sound.

An illustration of the rules above, not a recording.

Who	What happens	What the assistant does
You	"Move my Thursday call to, and..."	Waits. The sentence ended on "and"
You	"...Friday at ten."	Answers
Arthur	"Done. Friday at ten. I also noticed that..."	Speaking, bar raised
You	A cough	Shorter than 250 ms. Keeps talking
Speaker	His own voice, leaking into the mic	Matches what he just said. Ignored
You	"Wait, make it eleven."	Stops at once. Remembers he only got as far as "I also noticed"

How it runs

Inside the dashboard, tested by machines first.

A pair of headphones resting on a desk microphone, a calm waveform on a screen behind with a small pause marker where a cough was ignored.

Tests before ears31 checks cover interruptions and 21 cover the voice screen. They run with the rest of MUVES OS's test suite.
Review found three faultsA code review turned up three bugs in the interruption logic. Each got a failing test first, then the fix.
The echo guard re-armsAfter every barge-in the echo protection is switched back on, so the next answer is protected too.
Headphones allow full two-way talkOn speakers the raised bar does the work. With headphones, the assistant can be interrupted by any real speech.
Where it standsThe dashboard was rebuilt after this work. The voice code passes its tests, and a fresh test with a real microphone is the next step before we lean on it again.

What we'd reuse

The same turn rules make a phone agent bearable.

For a business, we would not build the phone line from scratch. We would use a managed voice platform and a phone number, have the agent answer only from an approved fact sheet, book into your calendar and text a summary, and bring these turn-taking lessons with us. We do not do AI cold calling.

Phone answering agentsAnswers, books and hands off to a person

After-hours & overflowPicks up when you are closed or busy

Home servicesCalls that arrive while everyone is on a job

What makes an AI phone call feel naturalThe turn-taking lessons, written up

Related case studies