voice-engine

A phone agent looks simple on a diagram: the caller speaks, you turn speech into text, a language model answers, you turn the answer into speech. The diagram hides the part that decides whether it feels human: what happens when the caller cuts in.

1. Caller speaks↓→
2. Speech-to-text↓→
3. Language model↓→
4. Text-to-speech↓→
5. Audio plays on the call

Problem 1: sent is not heard

Your server streams audio to the phone provider faster than the caller can listen to it. When the caller interrupts, a lot of audio you already "sent" has not been played yet. If you assume it was heard, your agent's memory is wrong. With Twilio Media Streams you can send a mark after the audio, and Twilio echoes it back when the audio has actually played. Only those echoes should count as heard.

Problem 2: from "milliseconds played" to "words heard"

Text-to-speech providers can return the start time of every character. Given the full sentence, that timing and how long the caller listened, you can split the sentence into the part that was heard and the part that was not. A word the caller only partly heard counts as unheard.

import { splitAtPlayedMs } from '@27colab/voice-engine';

const { spoken, unspoken } = splitAtPlayedMs(fullText, alignment, playedMs);
// spoken:   "Sure, I can help with that. We have"
// unspoken: "three offices in Brickell, and the first is..."

Problem 3: rewrite the agent's memory

If the model still believes it finished its sentence, its next answer will be confused: it may skip information the caller never received. The fix is to rewrite the last assistant message to the heard part, plus a marker that it was cut off.

import { rewriteOnBargeIn } from '@27colab/voice-engine';

rewriteOnBargeIn(history, spoken, { inHistory: true });
// last assistant message is now: "Sure, I can help with that. We have [interrupted]"

Try both on this page

The first demo drags the interruption point yourself. The second is a receptionist you write the instructions for; talk over it, and you will see its memory rewritten to what you actually heard.

[
 {
  "role": "assistant",
  "content": "Sure, I can help with that. We have three offices in Brickell, and the first is a thousand square feet on Brickell Avenue with parking included."
 },
 {
  "role": "user",
  "content": "Actually, I'd rather look in Coral Gables."
 }
]

[
 {
  "role": "assistant",
  "content": "Sure, I can help with that. We have [interrupted]"
 },
 {
  "role": "user",
  "content": "Actually, I'd rather look in Coral Gables."
 }
]

[]

Problem 4: "uh-huh" is not an interruption

People say "mm-hm" and "sí claro" while you talk. If every sound stops the agent, it sounds jumpy. The library classifies short backchannels separately from a real utterance, and the classifier can be swapped for your own.

Run it offline

The repository's quickstart simulates a whole call with fakes, no keys and no phone. The caller hears about two seconds, interrupts, and the model's next prompt shows the cut-off turn.

console.log(chat.seenMessages[1]);
// [{"role":"user","content":"Any openings this week?"},
//  {"role":"assistant","content":"We have openings Monday morning, [interrupted]"},
//  {"role":"user","content":"Actually, never mind, thanks"}]

What this does not do

• Played time is a lower bound, so the rewrite errs toward "heard less".
• It is single-process and in-memory, and pre-1.0. Nothing here has been benchmarked for latency or quality.
• It is a compact reference for the parts that are easy to get subtly wrong, not a voice-agent platform. Vapi, Retell, LiveKit Agents and Pipecat already ship barge-in.