A voice agent is five systems glued together. The product lives in the seams.
This page is the whole build: the system design, the engineering that makes it feel alive, and the implementation, module by module. The stack costs nothing to run: Silero VAD, faster-whisper, any local model through Ollama, and Piper. No API keys, no per-minute bill, nothing has to leave your machine. The code is small enough to read in an afternoon: github.com/anishfyi/voicer-ai.
mic30 ms frames
VADvad.py
ASRasr.py
LLMllm.py
chunkerllm.py
TTStts.py
speakeraudio.py
the VAD keeps running during playback. that single loop is barge-in
01The free stack
Four stages, zero dollars
Every stage is a small protocol with a free implementation behind it. Providers swap with one flag, so you can trade quality against latency yourself instead of trusting a pricing page.
VAD · is someone talking?
Silero VADor EnergyVAD, zero dependencies
A 2 MB neural model, the de facto open standard. The fallback is an RMS threshold with hangover written against the standard library, because you should learn endpointing on code you can read.
MIT · local · CPU
ASR · speech to text
faster-whisperCTranslate2 port of Whisper
OpenAI's Whisper weights, rebuilt for speed. Free, private, GPU optional. base is the laptop default, large-v3 wants a GPU. Greedy decoding, because short turns do not need a beam.
MIT weights · local
LLM · the brain
Ollamaany OpenAI-compatible endpoint
The client speaks the OpenAI protocol, so the brain is whatever serves it: Ollama or vLLM on your own hardware for free, or a cloud endpoint when you want one. The agent cannot tell the difference.
local · your weights
TTS · text to speech
Piperor edge-tts for nicer voices
Fast neural synthesis on CPU, fully offline. edge-tts is the free upgrade path for voice quality, with the honest caveat that it is unofficial and routes through Microsoft.
MIT · local · CPU
API keys required 0Per-minute cost $0.00Audio leaving your machine noneInstall pip install -e ".[audio,asr,tts-piper]"
02System design
The cascade, stage by stage
This is a cascaded pipeline on purpose: every stage is inspectable, swappable, and debuggable, and there is text in the middle you can audit. Speech-to-speech models collapse the cascade for latency and prosody, but tools, evals and audit trails get harder. Pick each stage below to see its job, its knobs, and how it fails.
03The state machine
Three states and one interruption
The orchestrator in agent.py owns one loop. What makes it feel human is not the states, it is the dashed line: the microphone thread never stops, even while the agent is talking. Detecting the user's voice during playback and shutting up immediately is most of the difference between an agent and a walkie-talkie.
LISTENINGmic + VAD, buffering
speech end
THINKINGASR, then LLM streams
first sentence
SPEAKINGTTS per sentence
barge-in (VAD fires during SPEAKING), or all sentences spoken → back to LISTENING
Thread 1 · the ear
Reads 30 ms frames off the mic forever and feeds the VAD. Emits two events into a queue: a completed utterance, or a barge-in marker. It does not know or care what the rest of the agent is doing.
Thread 2 · the mouth
The main loop blocks on the queue. On an utterance: transcribe, stream the LLM, synthesize and play sentence by sentence, checking for barge-in between sentences. On interruption: stop the player, keep what the user said.
04Latency
The only number users feel
Under 300 ms feels magical. 500 to 800 ms feels fine. Past 1.5 s the agent feels slow, and past 3 s people repeat themselves or hang up. Time to first audio is a sum you can engineer term by term, and the first term is not a model at all: it is a constant you chose. Drag it.
1.90 sto first audiofeels fine
# why pipelining is the whole trick: speak sentence 1 while the model writes sentence 3
without: TTFA = ASR + full LLM response + TTS(all of it)
with: TTFA = ASR + LLM first sentence + TTS(first sentence)
05One call, second by second
Maya changes a delivery address
A realistic support call played against this exact architecture, timings and all. Watch the state machine, the 600 ms the hangover quietly eats, the barge-in at 8.6 s, and the filler sentence that covers a 1.2 s API hole. The long version, with every failure mode annotated, is in docs/worked-example.md.
idlet = 0.0 s
TTFA first reply 2.35 s from her last wordbarge-ins 1 successfultool calls 1, after read-backcontainment yes
06Engineering
Four bugs the tests could not catch
These all shipped, all passed the suite, and all degraded the agent in ways a unit test on the happy path never sees. Each fix is now a comment in the source, because the next reader deserves the war, not just the treaty.
bug 01 · llm.py
The chunker that silently stopped chunking
Symptom: TTFA quietly reverts to full-response latency. Nothing errors. The agent just feels slow.
"Dr." looks like the end of a sentence. When the boundary turned out to be false, the scanner restarted from zero, rediscovered the same match on every new token, and never emitted anything. Pipelining degraded into "speak after the whole response".
scan = 0
while True:
match = _SENTENCE_END.search(buffer, scan)
if not match: break
candidate = buffer[:match.end()].strip()
if candidate and _closes_a_sentence(candidate, min_chars):
yield candidate
buffer = buffer[match.end():]; scan = 0
else:
scan = match.end() # step PAST the false boundary
A latency bug that produces correct output is invisible to a correctness suite.
bug 02 · agent.py
The barge-in check that ate the interruption
Symptom: the user cuts the agent off, the agent stops, then ignores the very thing they said to cut it off.
The event queue carries two kinds of event. Returning early on the first barge-in marker left already-dequeued utterances consumed and gone. The fix drains everything, keeps the answer, and re-queues what it was not looking for.
interrupted = False; deferred = []
try:
while True:
event = self._events.get_nowait()
if event[0] == "barge_in": interrupted = True
else: deferred.append(event) # keep the utterance
except queue.Empty: pass
for event in deferred: self._events.put(event)
return interrupted
A queue with two message types is a protocol. Treat it like one.
bug 03 · llm.py
The agent that forgot every interrupted turn
Symptom: barge in, then ask a follow-up. The agent has no idea what either of you just said.
History was appended after the token loop finished. Every barge-in abandons that generator mid-stream, so interrupted turns never reached history. The fix is a try/finally around the stream: record what was asked and what was actually said out loud, even when the consumer walks away.
try:
for chunk in stream:
...; yield delta
finally:
self.history.append({"role": "user", ...})
if spoken:
self.history.append({"role": "assistant", ...})
Generators get abandoned. Anything that must happen needs a finally.
bug 04 · vad.py
The stdlib that left
Symptom: the whole package fails to import on Python 3.13. The VAD never even runs.
audioop.rms was removed from the standard library in 3.13. The definition is three lines of arithmetic over 480 samples, so the dependency was never worth it: compute the root mean square directly and byteswap on big-endian hosts.
samples = array.array("h"); samples.frombytes(frame)
if sys.byteorder == "big":
samples.byteswap() # frames are little-endian regardless of host
return math.sqrt(sum(s*s for s in samples) / len(samples))
A dependency you can replace with three lines was three lines all along.
07The implementation
Six modules you can hold in your head
The whole agent is about 700 lines of Python. Readability over cleverness: the point is to learn the stack, not to hide it behind abstractions. Each stage is a protocol with a local and a cloud implementation behind a factory, so make_asr("whisper-local") and make_asr("openai") are the same agent with a different bill.
Numbers to have cold
Sample rate
16 kHz mono, 16-bit PCM
Frame
30 ms = 480 samples = 960 bytes
Trigger
3 frames = 90 ms of speech before believing it
Hangover
20 frames = 600 ms of silence ends the turn
Pre-roll
10 frames = 300 ms ring buffer, so the first phoneme survives
Silero frame
512 samples = 32 ms at 16 kHz (not 480)
08Failure modes
Twelve ways a voice agent dies
For each one, ask whether the fix is machine learning, product policy, or audio engineering. It is usually not machine learning. The last four are bugs this repo actually shipped and fixed; they are section 06 in table form.
#
Failure
Cause
Fix
Layer
1
Cuts users off mid-thought
hangover too short
longer hangover, semantic endpointing
product knob
2
Dead air every turn
hangover too long, slow LLM
shorten hangover, pipeline sentences
product + eng
3
Agent interrupts itself
speaker leaks into mic, no AEC
AEC, mix-minus, echo gate, headphones
audio eng
4
Misses real interruptions
mic muted during playback
keep the mic thread alive always
eng
5
Stops on "mm-hm"
backchannel read as barge-in
duration gate, turn-taking model
ML + heuristic
6
Markdown mouth
LLM emits bold, lists, URLs
voice system prompt
prompt
7
Wrong order id, confident action
ASR error into a tool call
read back identifiers, ITN, validate
dialogue design
8
Silent 4 s while a tool runs
no filler
speak a filler, idempotency keys
product
9
TTFA reverts to full response
chunker stalled on an abbreviation
scan past false boundaries
bug, fixed
10
Interruption is ignored
barge-in check ate the utterance
drain fully, re-queue the rest
bug, fixed
11
Forgets the interrupted turn
history appended after the stream
try/finally around the token loop
bug, fixed
12
Won't import on Python 3.13
audioop removed from stdlib
compute RMS directly
bug, fixed
09Live demo
Talk to it, right here
The same loop, rebuilt on the free speech stack your browser already ships: Web Speech recognition stands in for whisper, speechSynthesis stands in for Piper, and a WebAudio RMS meter is the EnergyVAD, threshold line and all. The brain is deliberately not a model: it is a few dozen lines of deterministic intent matching, because the intelligence belongs in the logic, not the model call. Ask it about barge-in, VAD, latency, the free stack, or who built it. Then interrupt it mid-sentence and watch it stop.
voicer, browser edition
off
[system] press start, allow the microphone, and just talk.
[system] endpointing, transcription and synthesis all run on your browser's free speech stack. no keys, no server.
EnergyVAD · live RMS
in speechno
last TTFA-
barge-ins0
turns0
Chrome and Safari ship the Web Speech API; some browsers route recognition through their own cloud. Headphones make barge-in honest, exactly like the real thing: this page has no echo cancellation either.
This browser does not expose the Web Speech API. The call replay above shows the same loop, and the local quickstart below runs the real one.
10Run it yourself
Three ways in, cheapest first
The CLI reads the standard OpenAI environment variables, so pointing the brain at a local Ollama server is two exports, not a code change.
Fully local, fully free
whisper on your CPU, llama through Ollama, Piper for the voice. Nothing leaves the machine.
git clone https://github.com/anishfyi/voicer-ai && cd voicer-ai
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[audio,asr,tts-piper,cloud]" # cloud extra = the openai client, here used to talk to Ollama
ollama pull llama3.2
export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_API_KEY=ollama # Ollama ignores the value; the client insists on one
voicer run --asr whisper-local --llm openai --tts piper
No mic, no models, ten seconds
Prove the pipeline logic works before downloading a single weight.
pip install -e ".[dev]"
voicer demo # synthetic audio through VAD -> turn detection -> sentence chunking
pytest # the suite runs without audio hardware
The demo feeds a synthetic utterance through the real VAD and the real chunker and prints what each stage decided. It is the offline heartbeat of the whole design.
edge-tts uses Microsoft's neural voices through an unofficial client: zero config and very good, but cloud-routed and unsanctioned. Piper is the answer when "free" has to mean "offline".
11Honest gaps
What this is not, yet
A learning codebase earns trust by naming what it skips. Each gap below is also the next thing worth studying, and the docs map each one to the papers that solve it.
No echo cancellation. With speakers, the mic hears the agent and can trigger false barge-in. Production systems run AEC; this repo asks you to wear headphones and learn why.
Utterance-level ASR, not streaming. Transcription starts only after the endpoint. Streaming ASR with partial hypotheses overlaps it with the user's speech and buys back 200 to 500 ms.
Reactive turn-taking, not predictive. The agent waits for silence instead of predicting the end of a turn the way humans do. Turn-prediction models are the upgrade path.
One speaker assumed. No diarization, no speaker verification.
No filler sentences yet. A tool call longer than a second needs a spoken "one moment", because silence reads as failure.
Where the frontier went: speech-to-speech models collapse ASR, LLM and TTS into one model, cut response time to about 300 ms, and keep the tone, laughter and hesitation a cascade throws away at the ASR boundary. The industry answer is hybrid: a duplex core for feel, a text brain for actions.
the 77-paper syllabus behind that sentence: corpus/papers/canon.md
12Go deeper
The watchlist
The best video for each stage of the pipeline, each one tied to the module it explains. Watch the video, then reread the matching file in voicer/: theory from the talk, mechanism from the code. One honest note: VAD has no great video anywhere; the RMS episode below plus vad.py is more than most people have watched.