System design

requirementsWhat I set out to build

A real-time voice conversation with a machine: sub-second perceived response, the ability to interrupt it mid-sentence and have it stop (barge-in), zero per-minute cost, and nothing leaving the machine. No API keys, no cloud bill, every stage a free local model. About 750 lines of Python, no framework, and every stage swappable between a local and a cloud provider behind one interface. The non-functional requirement that dominates everything else is latency: humans read anything past 1.5 seconds of silence as broken.

architectureThe cascade

I chose a cascaded pipeline over a speech-to-speech model deliberately. The cascade gives me text in the middle: every stage is inspectable, every provider is swappable, and the transcript is a free audit trail for logging, evals, and deterministic tool calls. Speech-to-speech models collapse the stages for latency and keep prosody, but tools, evals and audits all get harder. For an agent that does things, I want the text seam. For a companion where naturalness is the product, I would choose the other way, and I can argue both sides.

latencyThe budget, term by term

Under 300 ms feels magical, 500 to 800 ms feels fine, past 1.5 s the agent reads as slow, past 3 s people repeat themselves or hang up. The number that should bother you: the largest single line item is the hangover, the 600 ms of silence (20 frames at 30 ms) I wait before declaring the turn over, and it is a constant I chose. No model upgrade touches it. That is why the frontier is semantic endpointing and turn prediction, not better silence detection. The back-of-envelope habit is the same one as the estimation drill: know your budget before you defend your boxes.

controlThe state machine, and the one insight

LISTENING --speech end--> THINKING --first sentence--> SPEAKING
    ^                                                      |
    +--------- barge-in, or all sentences spoken ----------+

Three states, one loop, and the thing that is not in the diagram is the whole trick: the mic thread and the VAD never stop, even while the agent is speaking. Barge-in is not a feature bolted on top; it is just the VAD staying on during playback, pushing an event when it hears you, and the main loop stopping the player. If the mic muted during SPEAKING, the transition out of it could never fire. On interruption there are four decisions: stop playback immediately, discard the unspoken sentences, record in history only what was actually said aloud (or the model later references things you never heard), and treat any in-flight tool call as the dangerous case: finish, cancel, or make it idempotent.

speedSentence-level pipelining

# the trick that makes it feel alive
without:  TTFA = ASR + full LLM response + TTS(all of it)
with:     TTFA = ASR + LLM first sentence + TTS(first sentence)

I chunk the LLM's token stream into sentences and hand sentence one to TTS while the model is still writing sentence three. For a three-sentence answer that is roughly a 3x cut in perceived latency, with no better model, no faster hardware, no spend. The catch is boundary detection in a live stream: "Dr." looks like the end of a sentence, and my chunker once stalled on exactly that, silently reverting the whole system to full-response latency while every test stayed green. A latency bug that produces correct output is invisible to a correctness suite.

failureHow it dies in practice

• False endpointing on thinking pauses. Hangover too short and the agent cuts you off mid-thought, then cuts off your correction too. Trust dies in one turn. Too long and every turn pays dead air.
• ASR fed silence hallucinates. Endpointing mistakes hand Whisper near-silent audio and it invents text; faster-whisper's own VAD filter runs as a second line of defense on exactly this.
• TTS latency spikes hide between sentences. Playback blocks the loop, so a slow synthesis call shows up as a mid-reply gap, not as TTFA. If you only measure TTFA, you will not see it.
• Echo makes the agent interrupt itself. Without acoustic echo cancellation the mic hears the speaker, VAD fires, and the agent stops for its own voice. The repo assumes headphones and says so; production means AEC.

scaleFrom one laptop to a fleet

The single-process design maps cleanly onto a service. Websocket ingress with session affinity: a conversation is stateful (VAD state, dialogue history, playback position), so a session pins to one stateful session worker that owns its state machine. The real decision is model serving: a per-session monolith (all stages colocated with the session) is simple and keeps hops off the critical path but wastes accelerators, versus shared per-stage pools: an ASR pool, an LLM pool with continuous batching, a TTS pool. Pools win on utilization and independent scaling, and cost you network hops inside the latency budget. My answer: colocate at small scale, then pool the LLM first, because it is the most expensive stage and batches best. Observability is per-stage latency histograms, p50 and p99, with TTFA measured from the user's last word, not the endpoint decision: the second zero flatters you by the entire hangover. Then barge-in success rate and false barge-in rate, because those are the product.

pitchSixty seconds, cold