OpenAI实时API的语音打断，文档没说的三个独立机制

Experience: 3 months of hell building a voice agent with OpenAI's Realtime API - here's what nobody tells you about interruptions

I just trashed 2 months of work. Two. Whole. Months.

Why? Because I completely misunderstood how voice interruption actually works in OpenAI's Realtime API. My agent was either a brick wall that ignored people mid-sentence, or it cut them off like a caffeinated intern who's had four Red Bulls. If you're building voice agents right now, learn from my suffering.

I've been in the conversational AI trenches since the GPT-3 days (remember when we thought that was impressive?). When OpenAI dropped the Realtime API last October, I genuinely thought "finally, no more cobbling together these Frankenstein STT→LLM→TTS pipelines that break if you look at them wrong."

Oh sweet summer child. I was so naive.

The demo videos? They look like magic. Smooth, natural, like talking to a real person. What they conveniently leave out: what happens when a user sneezes mid-sentence, changes their mind halfway through, or—god forbid—actually tries to interrupt the AI like a normal human conversation. You know, the thing humans do constantly.

The architecture that broke my brain

Here's the thing about the Realtime API that the docs sort of... gesture at vaguely... but don't scream from the rooftops: interruption isn't one thing. It's actually three completely separate mechanisms that all need to dance together:

1. Server-side VAD (Voice Activity Detection) - the API's built-in turn detection

2. Client-side audio buffer management - what YOU do with the audio stream on your end

3. Response cancellation - the response.cancel event

I assumed #1 would handle everything. Just set it and forget it, right?

That's like assuming your car's cruise control will parallel park for you. Technically related to driving, sure. Not the same thing at all.

My biggest facepalm moment (and there were many)

Built this whole customer service agent for a client. Tested it internally with my team for two weeks, worked beautifully. We were high-fiving. Deployed to beta testers on a Thursday afternoon.

Friday morning: Slack is on fire. "It won't let me talk." "It keeps talking over me." "I said 'wait' three times and it just kept going." Someone sent a recording and I could literally hear their frustration.

The issue? I had turn_detection set to server_vad with the default thresholds. The default silence_duration_ms is 500ms. That's half a second. Sounds reasonable on paper, right?

Wrong. So wrong.

In real conversations, people pause mid-sentence ALL THE TIME. They say "I want to... um... book a flight to..." and the API goes "GREAT, LET ME HELP YOU BOOK THAT FLIGHT" because it detected 500ms of silence during their "um." The AI is enthusiastically responding to half a thought while the user is still formulating the rest.

But here's the counterintuitive part that drove me insane—if you make the silence threshold too long (say 2000ms), users feel like they're talking to a brick wall. They finish their sentence and just sit there in awkward silence while the AI... waits. And waits. And the user goes "hello??" which then interrupts the AI that was finally about to respond.

There's this uncanny valley of silence timing. Too short and you're interrupting. Too long and you seem broken.

What actually worked (after 47 iterations, no joke)

Here's my current setup. It's not perfect but it doesn't make me want to throw my laptop out the window anymore:

turn_detection: {
 type: "server_vad",
 threshold: 0.5, // lower than default, more sensitive to speech
 prefix_padding_ms: 300, // this is CRUCIAL - captures speech BEFORE the trigger point
 silence_duration_ms: 800, // longer than default, trust me
 create_response: true
}

But the real magic? It's in the client-side handling. The stuff the docs barely mention.

I maintain a circular buffer of the last 2 seconds of audio. Always recording, always buffering. When the user interrupts (detected via input_audio_buffer.speech_started event), here's what happens:

1. Immediately send response.cancel

2. Flush my buffer to capture what they said DURING the AI's response

3. Feed that audio back as the new input

This catches those "wait, no, I meant..." moments that happen while the AI is still yapping away. You know, like how actual humans interrupt each other.

Actually, wait—I should clarify something. The buffer doesn't "capture" audio in the sense of recording from the mic during playback. It's more like... you're always streaming audio to the API, right? So the buffer is just holding onto the last 2 seconds of that stream. When an interruption happens, those 2 seconds contain the beginning of the user's interruption. Does that make sense? I probably explained that badly.

The "barge-in" nightmare

Here's something I learned the hard way at 2am on a Tuesday: response.cancel doesn't actually stop audio playback immediately. There's a race condition. The API sends audio chunks, you're playing them through your WebSocket, user starts talking, you cancel, but there's still 200-300ms of audio in the pipeline.

So the user hears the AI keep talking for a split second after they've started speaking. It's jarring. It feels broken.

My solution? I mute the audio output the MOMENT I detect speech, before even sending the cancel event. Yes, it's a hack. Yes, it works. No, I'm not proud of it. My codebase judges me silently.

// Dirty but effective
// I literally wrote a comment above this that says "sorry future me"
audioContext.gainNode.gain.setValueAtTime(0, audioContext.currentTime);
// Then send cancel
ws.send(JSON.stringify({ type: "response.cancel" }));

The muting happens in like 2-3ms. The cancel takes 100-300ms. That gap matters.

The thing that still keeps me up at night

Okay so here's a fun edge case that I still don't have a perfect solution for. Actually, "fun" is the wrong word. "Hair-pulling" is more accurate.

What happens when: the user interrupts, the AI starts responding to the interruption, but the user was ACTUALLY interrupting to correct themselves, and now the AI is responding to the wrong thing entirely?

Like: User says "I need a flight to Denver—wait no, Chicago." The AI hears "Denver" and starts responding about Denver flights, but the user was correcting to Chicago. Now you've got the AI confidently talking about the wrong city while the user is getting increasingly frustrated.

I don't have a perfect solution. I've tried a few things. What I'm doing now is using a "cool-down" period after interruptions where I buffer everything and only commit after 1.5 seconds of actual silence. Not just VAD silence, but confirmed end-of-utterance. It's... fine. It reduced my "wrong context" errors by about 60% but that remaining 40% still stings.

I think the real solution involves some kind of semantic buffering where you're constantly re-evaluating whether the latest utterance supersedes the previous one. But that's a whole other rabbit hole I haven't gone down yet.

Real numbers from production (as of last week)

After 3 months and roughly 50,000 conversations:

• Default VAD settings: 34% of conversations had at least one interruption failure
• My tuned settings: dropped to 12%
• Adding client-side buffer management: down to 6%
• The remaining 6% are mostly edge cases (background noise, strong accents, people who talk like auctioneers, that one guy who was using it while driving with the windows down)

6% doesn't sound great but honestly? It's manageable. The key is having a graceful fallback. "Sorry, I missed that—could you repeat?" goes a long way.

What I wish the docs said

Instead of the happy-path demo that works in a quiet room with one person speaking perfect English at a measured pace, I wish the docs just came out and said:

"The Realtime API handles basic turn-taking. For anything resembling natural conversation, you need to implement interruption handling yourself. Here's a reference implementation, here are the edge cases, here's what will go wrong."

But no. We get the demo. The demo is a lie.

TL;DR because I wrote a novel:

• Server VAD is just the starting point, not the solution
• You NEED client-side audio buffer management for natural interruptions
• `response.cancel` has latency - mute audio output preemptively or it feels janky
• Tune `silence_duration_ms` higher than default (800ms worked for me, YMMV)
• Accept that ~6% of interruptions will still fail and have a fallback
• Test with real humans who say "um" and change their minds mid-sentence
• The demo is a lie

Anyone else wrestling with this? I've seen some threads about using WebRTC instead of WebSockets for lower latency—curious if anyone's actually gotten that working in production. I tried it for like a weekend and the tooling was a nightmare, but maybe I was doing it wrong.

Also, if anyone from OpenAI is reading this (lol, as if), please add a response.interrupt event that actually works synchronously. Please. I'm begging you.

Edit: Thanks for the gold! Since people are asking—yes, I'll share my buffer management code in a gist. Give me a day to clean it up. It's currently held together with console.logs and shame. There are comments like "// idk why this works but don't touch it."

Edit 2: Several people asked about WebRTC. I did try it briefly in January—latency is definitely better, like noticeably better, but the tooling is a nightmare and debugging WebRTC issues made me want to switch careers. Stuck with WebSockets for now. If someone has a good WebRTC + Realtime API setup, please DM me.

Edit 3: Someone asked what TTS voice I'm using. Shimmer, obviously. Is there even another choice?

#voiceai #openai #realtimeapi #conversationalai #webdev