Six commits to make one AI stop talking

Nov 11, 2026AI Reliability Engineering6 min read

I went back through the commit history of a system my team at Aditi delivered this year, looking for the parts worth writing about.

An AI that always has something to say is not a colleague. It is an interruption with a voice.

I expected the interesting commits to be about the model. They are not. Six of them are the same bug, fixed six times, across ten days.

The system runs a structured joining call for a staffing client. Three participants: a new employee, a member of the HR team, and an AI that carries the scripted part of the conversation and answers questions. One of the three is software.

Here are the six, in the order they happened, over ten days in the spring.

A fix for the agent’s voice overlapping the slide narration. The same fix reverted whole, fourteen minutes later. Making the bot stay silent until HR starts it. The AI playing over the top of a demo video. Audio from that video bleeding into the slides on the way back. And suppressing the agent’s greeting when the connection opened mid-slide.

Every one of those is the same sentence written six ways: the AI spoke when it was not its turn.

Not one of them is about the AI being wrong. In all six cases the answer was correct, the voice was correct, the retrieval was correct. The failure was social.

Two kinds of silence, and only one of them is a model problem

There is a version of this conversation that gets all the attention, and it is the version about knowledge. Should the model answer this question, or does it not know enough. I wrote about that decision separately, because it deserves its own argument and because it is genuinely hard.

This is the other one, and it is the one I underestimated.

An AI in a live call has to be quiet for two entirely different reasons.

It has to be quiet because it does not know the answer. That is a property of the model, the knowledge base and a threshold, and you can reason about it with numbers.

It has to be quiet because somebody else is speaking, or a video is playing, or a person on the call has taken the floor. That has nothing to do with the model at all. The model has no idea a video is playing. It cannot have. It is not in the room.

I spent the design phase thinking hard about the first kind and treating the second kind as user interface work. That was the mistake, and the six commits are the invoice.

What I got wrong, and for how long

For most of my career, systems I built had one thing generating output at a time. A request came in, a response went out. Turn-taking was not a concept because there were no turns.

So when I designed this call, I put the AI’s silence where it felt natural: in the prompt and in the agent configuration. We set a conversation-style variable to make the agent reactive so it would stop volunteering things. We blanked its greeting text through a dynamic variable when the connection opened during a slide, so it would not re-introduce itself over the narration.

Both of those work, up to a point. Both of them are instructions to a model dressed as configuration, and the model follows them the way it follows anything, most of the time. The greeting was the clearest tell. Every time the connection opened during a slide, including a reconnect in the middle of the call, the agent introduced itself over the narration, and the fix was to blank its greeting at that exact moment. The fixes we put in the prompt and the configuration all had that shape: catch the model about to speak, and talk it out of it. The others were timing work in the browser, and some of the work was not about silence at all: the voice mispronounced the client’s name and two other terms, and we tried a pronunciation dictionary first, then spelled the name out letter by letter in the narration text. And the timing problem kept finding new moments: two weeks after those ten days we were still fixing repeated introductions, and agent audio arriving after the narration had taken over.

That is when it became obvious what I had built. I turned a scheduling problem into a persuasion problem.

The correction is one sentence, and I now say it out loud at design time on every build with a real-time model in it: whose turn it is must be a fact the server owns, not a behaviour the model exhibits.

Where the floor lives

In the finished system, the floor is a boolean on the call.

/// <summary>
/// True while HR has taken over the call and Priya's voice is paused.
/// </summary>
public bool HrTakeover { get; private set; }

public void BeginTakeover() => HrTakeover = true;
public void EndTakeover()   => HrTakeover = false;

It sits on the call entity, next to the segment index and the call status. It is persisted. It survives a browser refresh, which matters more than it sounds, because the person most likely to refresh mid-call is the nervous one on their first day.

Flipping it is a single command that does three things and nothing else: load the call, flip the flag, tell everyone.

if (active) call.BeginTakeover();
else        call.EndTakeover();

await _callRepository.UpdateAsync(call, ct);
await _notifier.TakeoverChangedAsync(callId.ToString(), active, ct);

The broadcast is the part that earns its place. Both browsers, HR and joiner, receive the same event from the same source of truth. Neither one decides locally that the AI should be quiet. Before this, more than one piece of browser code held its own opinion about whether the AI should be speaking, and those opinions disagreed in ways that only showed up on somebody’s real first morning.

One more detail, small and load-bearing. The team relabelled the HR controls partway through the build. The button that had said “Watching” became “Take Over”, and its opposite became “Back to AI”. Nobody could describe the state of the call while the label was a verb about observation. Naming the two states is what made the state machine visible. The very next commit to touch those controls deleted a duplicate mic handler that had been fighting the first one.

Stopping is harder than starting

This is the part I would not have predicted, and it is the most transferable thing in the build.

Telling a talking AI to stop is not a message you send. By the time you decide to interrupt, the audio it is going to say next has already arrived, already been decoded, and already been scheduled into the browser’s audio clock, several seconds ahead of what the person can hear. Closing the connection changes nothing. The queue plays out regardless, and the person hears the machine finish its sentence over the top of their colleague.

The actual interrupt is two lines.

function cutOffPlayback() {
  if (audioCtx) nextPlayTime = audioCtx.currentTime;
}

That is it. Every scheduled chunk is placed at a running cursor. Dragging the cursor back to the present abandons everything queued in the future. The machine stops mid-word, which sounds rude and is exactly right, because that is what a person does when someone else starts speaking.

If you are putting a voice model into anything live, this is the design question to ask on day one, not day forty: what is already in flight when I decide to stop, and what is the mechanism that discards it. Almost every real-time voice integration has an answer. Almost no integration guide leads with it.

The direction nobody designs

Handing over to the human is the demo. Handing back is the product.

The way back has to restore three things, and they are not the same three things that were suspended. Playback resumes. The listener that had been paused restarts, because the microphone pipeline was stopped and a stopped recogniser does not simply carry on. And the joiner needs to be told, in words on screen, that the AI is listening again, otherwise they sit in silence waiting for a person who has already stepped back.

There is also a gate that took a bug to discover. The Take Over button is disabled until the AI has actually been started, and it disables again the moment the AI is paused or stopped. You cannot take the floor from something that does not hold it. Before that gate existed, HR could take over a call the AI had never joined, which set the flag, muted a voice that was not speaking, and produced a call where nobody could work out who was supposed to talk next.

In those same ten days we went further and rebuilt the question and answer segment so that listening, thinking and speaking became three separate calls the server drives in sequence, instead of one always-open conversation the model runs. It costs a little latency. What it buys is that silence becomes the default state rather than a state we keep suppressing. The client later decided their new joiners did not need a separate question and answer segment, so it is switched off today; the pipeline is still in the code. That is its own lesson: some of the hardest engineering in the build went into a feature the client chose not to use. Every one of the six bugs above lived in the gap between those two designs.

What I ask now

On any build where a model produces output into a live human conversation, I ask three questions before the model is chosen.

Who owns the floor, and in which table is that written down. If the answer is a prompt instruction, there is no owner.

What is already in flight when the floor changes hands, and what discards it.

What restores the machine’s turn, and how does the human know it has been restored.

None of those questions are about AI. They are the questions you would ask about any two processes contending for one output device, and they have had good answers since long before any of this.

An AI that always has something to say is not a colleague. It is an interruption with a voice.