#DGXSpark #LocalLLM #VoiceAssistant #TTS #STT #VoiceCloning #OpenClaw

Three shims and a revert: putting a fully local voice assistant on a DGX Spark

A DGX Spark was already running three local LLMs. Spare memory was left over: not enough for another large language model, too much to leave idle. The obvious thing to put there was speech synthesis, and the obvious thing to build with it was a voice assistant that never sends audio anywhere.

The build took a day. Response latency started at 34 seconds and ended at about 5. Along the way the framework turned out to have no way to express three things the pipeline needed, and each time the answer was a small piece bolted on beside it rather than a patch inside it. One of those pieces failed in production, got reverted, and went back in with the failure understood.

This is what got chosen, what broke, and what the numbers were.

0. Why local, and what counts as fast

Two things worth fixing before any number below means anything.

The case for running it yourself

Four reasons come up consistently in 2026 write-ups of self-hosted voice stacks. (Asterisk community, TechFuel)

Reason	What it buys
Data sovereignty	Audio and conversation never leave the building. Usually the deciding reason
Predictable cost	Hardware and electricity instead of per-minute billing. Wins when usage is steady
Consistent latency	No network jitter, no rate limits. Slow, but slow the same way every time
Control	Swap models freely. Nothing gets deprecated out from under you

What you take on in exchange is operational responsibility, and most of this article is about that.

Two different latency bars

Getting the bar wrong distorts everything downstream. The widely quoted target is 500 to 800 milliseconds from the end of user speech to first audio, with interaction breaking down past a second because people assume they were not heard and start again. Per-layer that is roughly STT 60–120 ms, LLM first token 100–250 ms, TTS first chunk 40–100 ms. (Prodinit, The Prompt Bench)

That is the bar for commercial real-time services. Self-hosted cascaded stacks get quoted a looser one: under 1.5 to 2 seconds to first response reads as natural, and a speech-to-speech model runs at roughly half the end-to-end latency of a cascade. (Dograh)

This build lands at about 5 seconds. It clears neither bar, and the last section takes the remaining budget apart segment by segment to show why. The short version is that the reason is not the hardware.

The machine

A DGX Spark is a small desktop box built around the GB10 superchip: 20 Arm cores and a Blackwell-generation GPU that share one pool of unified memory. The spec is 128 GB; the OS reports 121 GiB usable, and every “121” in this article is a value actually read off the machine. It lists at $4,699, with a 140 W TDP and a 240 W supply. (NVIDIA, Kubesimplify)

The number that matters here is not throughput, it is the shared pool. Keeping three models resident and pushing audio into one of them is possible because none of them owns its own VRAM.

The voice-cloning landscape

Anyone building a custom voice locally in 2026 has a wide field to pick from; the four candidates below are a slice of it. (BentoML, Inworld)

Model	Character
XTTS-v2	Zero-shot cloning from a ~6-second sample. Strong multilingual and cross-lingual transfer
Fish Speech / S2	80+ languages, good zero-shot cloning
CosyVoice 2 family	Strong at real-time and streaming, with emotion and prosody control
IndexTTS-2, NeuTTS Air	Zero-shot with fine control such as duration. NeuTTS Air targets on-device at 0.5B
Chatterbox family	Real-time generation, emotion control, cloning
Piper	No cloning, but light and good. Common in smart-home setups

The trend in one line: zero-shot cloning from a short reference is now the default, and training a model per voice is a choice you make for a specific reason. Both approaches got tried here.

1. Choosing the TTS model

A custom voice was the goal, so every candidate had to be able to speak as a particular person. Four were evaluated. Three clone from a reference; the fourth trains on collected data.

Candidate	Approach	Outcome
Higgs Audio v3 (Boson AI)	Zero-shot cloning	Adopted
CosyVoice 3	Zero-shot cloning	Will not run on this GPU
Fish Audio S2 Pro	Reference cloning	Lost on quality. Was the incumbent
GPT-SoVITS fine-tune	Collect data, then train	Worked, but the wrong operating shape

Candidate 1: Higgs Audio v3, better output on a stack that did not fit

A 4B model Boson AI released in June 2026 (bosonai/higgs-audio-v3-tts-4b). It uses a Qwen3-4B backbone, and its audio tokenizer emits 8 codebooks at 25 fps reconstructed to 24 kHz. The figures below come from the model card and our own evaluation record.

Item	Detail
Parameters	~4B, Qwen3-4B backbone (36 layers, hidden 2560, GQA 32/8)
Audio	8 codebooks @ 25 fps → 24 kHz
Features	Zero-shot cloning, 21 emotions, inline prosody and SFX tags, streaming
Languages	102 supported, 85 of them under 5% error, Korean among them
Licence	Research / non-commercial, with a separate creator-use grant allowing monetised creator use with attribution

This model had already been evaluated in June and rejected. Quality was clearly ahead even then, but it does not run inside a llama.cpp-style stack, and with a cloud TTS covering the need there was no reason to operate a second Python serving stack.

The condition changed. Once “everything local” became the requirement, cloud was not an option, and the June rejection reason turned into an acceptable cost.

Serving it is genuinely awkward. llama.cpp does not support the architecture. The SGLang-Omni docker image has no arm64 manifest, so it is unusable on GB10. transformers does not recognise the architecture either, as of 5.12.1. The path that worked was running a vLLM-family server inside NVIDIA’s PyTorch container.

There is also an artifact worth knowing about. At default temperature, with no reference clip — both conditions together — emotion and SFX tokens free-run. A Korean sample grew a scream at the head and a runaway tail: 9.3 seconds of audio came out as 18.9. Dropping temperature to 0.4, supplying a speaker system prompt, or pinning the voice with a reference clip all remove it. The last is the intended mode and the one in use here.

The cause is simple: with no reference, the model samples a speaker from its learned distribution on every run, and that randomness is the artifact. In production, pin the voice.

The Fish S2 comparison below was produced without a reference clip, because putting the two models on equal footing meant giving neither one, with temperature 0.4 applied to Higgs as the stated mitigation. That is not the production configuration, so read that table as a relative comparison only.

Candidate 2: CosyVoice 3 does not run on this box

It failed. The code targets torch 2.3.1, and GB10 is sm_121 which needs CUDA 12.8 or later, so the pinned dependency set cannot drive this GPU at all.

Unpinning and moving to torch 2.13 reaches the model, then stops here:

mat1 and mat2 must have the same dtype, but got Float and BFloat16

This is where a wrong conclusion was available: was it the Korean input, or our configuration? Running the model card’s own example produced the identical failure, which isolates it from anything we were doing, and that ended the evaluation. Two smaller walls came before it: a missing pkg_resources needing setuptools<81, and torchaudio 2.13 routing load() through torchcodec. A retry would be cheaper on torch 2.8 cu128.

Candidate 3: Fish Audio S2 Pro, right stack, second-best output

The incumbent. It outputs 44.1 kHz and clones from reference audio. Its biggest advantage was fitting the existing stack exactly: no Python serving layer, one system service, almost no operational load. The description here stays inside what our own evaluation record supports.

Both models generated the same sentence per language, default voice, no reference, Higgs at temperature 0.4.

Sample	Higgs v3 (24 kHz)	Fish S2 Pro (44.1 kHz)
Korean	9.3 s	8.2 s
English	6.5 s	7.8 s
Japanese	8.7 s	10.4 s

Higgs was faster in two of the three and slower only in Korean. Fish S2 has the higher sample rate on paper. Listening decided it: prosody and rhythm were clearly more natural on Higgs, Korean pronunciation noticeably smoother. No number available reverses that, so the judgement had to be made by ear, and it was not close.

Candidate 4: fine-tuning works, and costs something different

Structurally unlike the other three. They imitate a voice from a few seconds of reference; this one collects voice data and trains a model. It was actually done, with GPT-SoVITS, and it produced a working model.

extract audio from source video
   → Demucs to separate vocals (strip music and effects)
   → Whisper large-v3 for transcription with word timestamps
   → speaker embeddings + clustering to isolate the target speaker
   → manual curation
   → GPT-SoVITS training

Three training runs, and the progression is the point.

Run	Data	Result
1	341 clips	Poor audio. Background-music artifacts had come along in the training data
2	229 clips, hand-curated	Better audio, better similarity, but flat affect
3	Same 229, LoRA	Emotion and intonation much improved, audio better again

What changed between run 1 and run 2 was not the training configuration. It was deleting 112 clips with noise and music bleed by hand. Clip count fell from 341 to 229 and the result improved. In this approach the time goes into the data, not the training.

Inference was fine: 3.5 seconds of audio in 1–2 seconds on an RTX 3080.

It was still not used here, and the reason is operating shape rather than quality. Changing the voice means collecting data and training again; with zero-shot you swap the reference clip. And the training pipeline wants its own workstation and its own Python environment, which cuts against consolidating onto one box.

Summary

Candidate	For	Against
Higgs Audio v3	Best output. Korean under 5% error. Streaming. Voice changes by swapping a clip	Needs a separate Python serving stack. Artifacts at default settings. Non-commercial licence with a creator carve-out
CosyVoice 3	Would have been a contender	Does not run on this GPU
Fish Audio S2 Pro	Native to the stack, simplest to operate. 44.1 kHz	Loses on prosody and Korean pronunciation
GPT-SoVITS fine-tune	High similarity when training goes well. Light inference	Data collection and curation cost. Retraining to change voice. Separate environment

Three conditions decided it, and only one candidate met all three: Korean under 5% error, voice changeable by swapping a reference clip, and streaming output. The third looked like a bonus at selection time. It turns out to be the single most valuable property in the whole build.

Picking a reference clip is not an ear-only job

Cloning needs a few seconds of reference. Candidate segments were extracted and ranked automatically on length in a 4–8 second band, dynamic range, and level stability.

The top-ranked clip was rejected on listening. Other people were laughing underneath the speaker. None of those three metrics sees that.

So a fourth axis went in: speaker purity. Cut the segment into 1.5-second windows, take speaker embeddings, and measure consistency across windows. The value that matters is the minimum, not the mean — a second voice intruding briefly washes out of an average and survives in the minimum.

With that axis the offending clip fell from rank 1 to rank 22, minimum purity 0.701 against 0.85–0.95 for the rest. Ear and metric reached the same conclusion independently, which is the only reason to trust either.

Two cheap tests came out of it and are worth reusing.

• The noise floor of the pauses tells you whether a music bed is present. A genuinely clean take reads −60 to −67 dB; a residual bed stops around −35 to −45.
• If a measurement comes back suspiciously clean, check the tool was allowed to speak. ffmpeg -v error suppresses silencedetect and volumedetect output entirely. That produced “zero silences at every threshold” and very nearly a confident wrong conclusion.

Serving defaults assume concurrency you do not have

The server came up holding 16,893 MiB. One person uses this assistant, so there is no need to plan for simultaneous requests — but the defaults plan for them and reserve memory accordingly.

Flag	Value	Reason
--max-running-requests	1	One conversation at a time
--max-total-tokens	8192	The model’s own training length. The KV pool never needs more than one utterance
--talker-cuda-graph	off	CUDA graphs are allocated outside the static memory fraction and buy nothing at batch size 1
--mem-fraction-static	0.30	Down from 0.40

16,893 MiB to 11,164 MiB, a 34% cut. Streaming first byte stayed at 1.147 s. Nothing was traded for it; reserved-but-unused capacity was simply handed back.

The general form: before touching a knob whose name sounds like memory, look at what the service is actually doing. This saving did not come from lowering a fraction. It came from noticing that a single-user service was provisioned for concurrency.

Also, --thinker-cuda-graph and --thinker-max-running-requests do not exist on this pipeline. They fail argument parsing with Stage ‘thinker’ not found in pipeline. Only the talker-prefixed and unprefixed forms are real.

2. The LLM was thinking, not answering

An already-resident model made this choice easy: Gemma-4 26B-A4B QAT — quantisation-aware training, so the model was trained to tolerate being compressed — at Q4_K_M, 16.8 GB. Q4_K_M and the Q4_K_XL below are llama.cpp weight formats of roughly four bits per weight, XL being the larger and slightly more faithful of the two. It is an MoE model, so few parameters are active per token and responses come back quickly. A voice assistant needs speed rather than depth, which fits.

Measured directly, it is quick:

                      run 1     run 2
time to first token   0.035 s   0.004 s
sentence complete     1.42 s    1.74 s

In the actual voice loop, a turn took 32 seconds, of which 27.95 s was response generation. Nothing about a 23-character reply justifies that.

Calling the model directly with and without reasoning found it:

reasoning on (default)   2.36 s   221 tokens   565 chars of reasoning
reasoning off            0.27 s     8 tokens   "네, 잘 들려요!"  ("Yes, I can hear you!")

Nine times the latency on a short prompt, and in a real conversation the persona and dialogue history are attached too, which stretches it to 28 seconds. That the LLM accounts for around 70% of total latency in real-time voice pipelines is a known observation; what is different with a local model that reasons by default is that the multiple runs into double digits.

The framework had no way to turn it off

The model side is easy: send reasoning_effort: "none" and it answers in 0.27 s. The problem was that OpenClaw does not put that value in the request.

A proxy that prints the request body settled it. Six configurations, each judged on what was actually sent rather than what should have been:

Model-entry setting	What the agent path actually sent
thinkingLevelMap all → none	nothing
thinkingLevelMap all → low	nothing — this path never consults the map
reasoning: false	nothing
compat.thinkingFormat	chat_template_kwargs {enable_thinking: true}
…the same, with level off	still true — the requested level is ignored
model entry params	never reaches the body

“Nothing sent” means reasoning on, because an unspecified request falls back to the server default and that default reasons. So the table has no winning row.

One trap nearly produced a false report. The framework’s CLI does send reasoning_effort, and with a thinkingLevelMap it correctly sends "none". The agent path is different code and does not. Verifying through the CLI alone would have signed off a change that does nothing. Check on the path the feature actually uses.

Shim one

If it cannot be expressed in configuration, inject it where the request passes. A small local server sits in front of the model and only the voice route goes through it.

if self.path.endswith("/chat/completions"):
    payload = json.loads(raw)
    payload["chat_template_kwargs"] = {"enable_thinking": False}
    # A surviving reasoning_effort re-enables reasoning in the chat template,
    # which is the exact thing this hop exists to prevent.
    payload.pop("reasoning_effort", None)
    payload["model"] = UPSTREAM_MODEL
    raw = json.dumps(payload).encode("utf-8")

A second model entry points at the proxy, and only the voice config uses it. Text chat calls the same model directly and still reasons.

models.providers.<provider>.models += { "id": "model1-fast",
                                        "baseUrl": "http://127.0.0.1:8097/v1" }
agents.defaults.models["<provider>/model1-fast"] = { "streaming": true }
channels.discord.accounts.<account>.voice.model = "<provider>/model1-fast"

The middle line is not optional. Agents carry their own allowlist of usable models, and without registration the override is refused with Model override "..." is not allowed for agent "main". The message names the agent rather than the allowlist, which cost time.

27.95 s to 2.22 s, a factor of 12.6. Total turn went from 34 s to 12 s. Every total in this article includes the silence-detection wait.

3. STT: the model was ready, the framework could not call it

Keeping this stage local is most of the point of a local build. Voice is more personal than text, and sending only this part to a cloud halves the value of the rest.

A dedicated ASR model was an option, but that means a fourth server and another division of memory. Instead, a model already resident accepts audio input: Gemma-4 E4B-it QAT, Q4_K_XL at 4.2 GB, previously used for RAG fact extraction. Not a dedicated ASR, but it can be handed audio and told to transcribe.

It was accurate — Korean speech at 24 kHz transcribed correctly in under 1.5 seconds. One pipeline stage solved with capacity that was already paid for.

A swap into that slot that had to be reverted

An uncensored fine-tune went into the same slot once and came out the same hour. The server started, /props still reported audio: true, and transcription collapsed at the syllable level.

Source	Same 6.2 s clip
whisper	…거짓말 아니고 샤넬 립밤 꼭 챙기고요
stock QAT model	…거짓말 아니고 선의 립밤 꼭 챙기고
fine-tuned model	나 파차나 커피스러드라가나 가르마나니고 차나리빠 고팅기고

The clip is Korean. The first two rows disagree on a single word, a brand name heard as a similar-sounding noun, which is the ordinary kind of error. The third row is not Korean at all: the syllables are real but the words are not, and no reader could recover the sentence from them.

Other causes were eliminated first. Our slot’s flags and KV-cache quantisation were suspected, so the model card’s own recommended settings were tried and failed identically. Sampling was tried at both the card’s values and greedy, and failed both ways. File corruption was excluded because both downloads matched size and hash exactly.

Vision was the deciding test. The same model through the same projector read text out of an image correctly in 0.6 seconds. Vision tower intact, audio tower misaligned by the fine-tune.

Two general forms worth keeping. When one modality misbehaves, test another modality through the same projector before blaming the projector or the config. And a healthy /props flag and a 200 on /health are not evidence that a modality still works — only real output is.

A valid configuration that did nothing at all

With the model ready, the config went in:

tools.media.audio.models = [
  { "provider": "<local>", "model": "<audio-model>", "capabilities": ["audio"], "type": "provider" }
]

This passes config validate. It also never transcribed anything. The assistant sat in the voice channel and did not respond.

The first reason it took time to find is that there were no logs. The log file size cap was set to 1024 bytes, so the file rotated every 736 bytes and lines disappeared between reads even when following live. Raising it to 50 MB was what made the system observable at all.

The cause, once visible:

if (!provider.transcribeAudio)
  throw new Error(`Audio transcription provider "${providerId}" not available.`);

An object that provides a chat model and an object that provides transcription are different kinds. Transcription needs a provider implementing transcribeAudio, and only seven commercial ones do. A locally registered provider has no such implementation. Writing capabilities: ["audio"] on the entry does not create one — that field records an intent the transcription path never reads.

One check was done this time that had been skipped in section 2: the CLI reproduction was only trusted after confirming in code that the CLI and the voice path call the same function, transcribeAudioFile.

Shim two

The audio entry schema accepts type: "cli": run a command, take its stdout as the transcript. That removed the need for an HTTP adapter entirely — no resident process, no port, no service to supervise.

tools.media.audio.models = [
  { "type": "cli",
    "command": "/path/to/stt.py",
    "args": ["{{MediaPath}}", "{{Language}}"],
    "capabilities": ["audio"] }
]

The script base64-encodes the wav, posts it to the model, and prints only the transcript. Reasoning has to be disabled here too: with it on the transcript goes to reasoning_content instead of content, and if the budget runs out content comes back empty.

And there is no fallback.

if finish == "length" and not text:
    print("STT hit the token limit before producing any transcript",
          file=sys.stderr)
    return 1

# Silence is a real outcome, not an error. Emit nothing and exit clean so the
# runtime records "no transcript" rather than a failed segment.
if text:
    sys.stdout.write(text + "\n")
return 0

A wrong transcript in a voice assistant passes silently, because the model answers something the user never said. Failures should be loud; only genuine silence exits clean.

4. The whole thing

With three servers ready, what remains is wiring them into one conversational flow.

The three voice models come to about 33 GB — 16.8 + 4.2 + 11.2 plus the 26B’s 1.2 GB mmproj, the multimodal projector that lets a text model accept audio and images. A separate 27B model at 17.6 GB is also resident, and KV caches and headroom sit on top; with all four up the machine reports 108 GiB of 121 GiB in use.

The important structural point: not one line of the framework was modified. All three times, a small piece went beside it. Not forking was a deliberate choice — it avoids carrying merge burden on every upstream update, and each of the three reverts in about a minute. One of them actually did.

5. What went wrong on the way

Exactly one provider streams

With reasoning off, synthesis became the largest remaining block, and the log said the same thing every turn:

TTS stream: provider openai skipped (openai does not support streaming TTS)

The synthesis server can stream, first byte at 1.15 s. The framework could not use that path, so it waited for the entire reply before starting playback — 7.35 s measured.

Searching the codebase, exactly one provider implements streamSynthesize. No configuration makes another one stream. So the shim imitates that one provider’s request shape.

Transcoding nothing is the design

The framework takes the synthesis stream and pipes it straight into ffmpeg:

const ffmpeg = spawn(resolveFfmpegBin(), ["-i", "pipe:0", ...FFMPEG_PCM_ARGUMENTS, "pipe:1"]);

ffmpeg sniffs container formats on its own, so no conversion is needed. But it cannot sniff headerless PCM, and the framework passes no -f or -ar to tell it.

So instead of inserting an encoder, the shim writes a 44-byte WAV header and passes the PCM through untouched. The length is unknown when the header goes out, so the size fields carry the unknown-length value.

def wav_header():
    """RIFF header for a stream whose total length is not known yet.
    ffmpeg accepts this form."""
    byte_rate = SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH
    block_align = CHANNELS * SAMPLE_WIDTH
    return b"".join([
        b"RIFF", struct.pack("<I", UNKNOWN_SIZE), b"WAVE",
        b"fmt ", struct.pack("<IHHIIHH", 16, 1, CHANNELS, SAMPLE_RATE,
                             byte_rate, block_align, SAMPLE_WIDTH * 8),
        b"data", struct.pack("<I", UNKNOWN_SIZE),
    ])

An encoder in this hop would reintroduce the latency the hop exists to remove.

Two more constraints surfaced. The synthesis server only streams when response_format: "pcm" is set, and rejects a bare stream: true with a 400 that names the condition in the body. And the framework validates the voice name against /^[a-zA-Z0-9]{10,40}$/, so a six-character name cannot be used directly — it is registered under a longer name and mapped back inside the shim.

Length	Audio	Blocking	Streaming first byte
Short	1.48 s	1.86 s (1.25×)	1.16 s
Medium	4.60 s	5.08 s (1.10×)	1.15 s
Long	9.48 s	10.21 s (1.08×)	1.15 s

Generated whole, it costs about 1.1× the spoken duration, so waiting grows with reply length. Streaming’s first byte stays at about 1.15 s whatever the length. Short sentences cost 1.25× because roughly 0.4 s of per-call overhead lands on a small amount of audio; that figure returns in the last section.

Making it talk faster does not work

Shorter audio should mean faster generation. Measured, it does not.

speed=1.0   audio 5.32 s   generation 5.84 s
speed=1.2   audio 4.97 s   generation 6.52 s

Audio shortened by 6% and generation got longer. Dead end.

Then it broke

The turn measured right after streaming went in was good: 3.67 s from the end of the capture window to first audio, about 4.7 s counting the silence wait. A while later, the conversation stopped.

10:50:02  TTS stream: starting
10:50:32  30-second timeout, failed
10:50:54  re-synthesised through the old path and played
          user waited 52 seconds

The shim’s own log had this line:

client stopped draining after 5.13s, 49152 pcm bytes (1.02s audio)

The consumer read one second of audio and stopped. 45,056 bytes on the real failure, 49,152 on a reproduction, 53,248 the next time. All three about one second, differing only in 4 KiB steps — and 4 KiB is the read size, so this is the approximate ceiling of what the receiving side can hold, not a coincidence.

The code explains it. The framework opens the stream when the reply text is ready and only drains it once playback starts. Between those moments the only storage is ffmpeg’s stdin pipe plus socket buffers, together about one second. Push more and the write blocks, while the 30-second timeout runs.

Reverting first

Two options: keep digging, or return to a known state.

Streaming buys the first sound moving from 7.35 s to 1.15 s — about 6 seconds a turn. That is not small. Even so, six seconds is not worth a conversation that occasionally stops for fifty. Overlapping speech is not an edge case in conversation, it is what conversation is. Without streaming every turn costs 12 s and none of them stall, and consistently slow beats occasionally frozen.

The shim and its config were kept, renamed rather than deleted. Turning it back on took a minute.

Four changes on the way back in

The important one was decoupling production from consumption. Previously the shim read from the synthesis socket and wrote straight to the client, so a stalled client propagated back into the synthesis server — a GPU slot pinned for a client that is not listening yet.

# Drain upstream at full speed; let the client take it at its own pace.
chunks = queue.Queue()

def pump():
    try:
        with upstream:
            while True:
                chunk = upstream.read(4096)
                if not chunk:
                    break
                chunks.put(chunk)
    finally:
        chunks.put(None)

threading.Thread(target=pump, daemon=True).start()

One utterance is at most about 80 seconds at 24 kHz mono, under 4 MB, so the buffer needs no bound. Connection reuse was also switched off — a chunked response cut mid-stream leaves a keep-alive connection desynchronised and the next request on it never gets answered — and a write timeout was added so a blocked write shows up in the log instead of hanging quietly.

Still not calling it fixed

Six consecutive turns after re-enabling, no stalls. A 10.24-second reply streamed through cleanly. The condition that used to break it — the user speaking while a reply plays — occurred twice with nothing following.

It is still not called fixed, because four things changed at once after the last failure: the reader thread, connection reuse off, the write timeout, and the synthesis timeout going from 30 s to 120 s.

The last one is the suspicious one. The 52-second stall was a timeout firing at 30 seconds. At 120 the same wait would simply have completed, which would mean the cause is untouched and only the symptom is hidden. This run cannot separate the two, because no wait occurred at all.

So the instrumentation stays. If it stalls again, one line decides it: the gap between the framework logging TTS stream: starting and the shim receiving the request.

One more note. During the investigation the timeout was lowered from 30 s to 10 s, which was backwards. A stream waiting its turn to play gets killed by a short deadline, so the containment creates the failure it was meant to contain. It was raised again once the mechanism was understood.

6. If you build one of these

The last column is the range across the six turns run after streaming was re-enabled.

Segment	Start	Reasoning off	With streaming
STT	1.15 s	0.42 s	0.37–0.46 s
Generation	27.95 s	2.22 s	2.07–2.70 s
Synthesis	2.89 s	7.35 s	1.15–1.27 s
Silence wait	2.0 s	2.0 s	1.0 s
End of speech to first audio	34 s	12 s	~4.9 s

The silence wait was one config value. Deciding an utterance has ended — the VAD window — defaulted to 2 seconds; at 1 second, utterances of 2.1 s and 3.2 s were still captured whole. Real-time voice agents budget about 200 ms here, so even at 1 second this build spends five times that.

Check it runs on your box before anything else

Executability outranks quality metrics in model selection. CosyVoice 3 was eliminated because its pinned dependency set does not support this GPU, not because of any benchmark. And when something fails, run the model card’s own example first — that single step separates “our input” from “this build”.

Look at what you already have

A dedicated STT model would have meant a fourth server and another split of memory. The model already doing RAG work accepted audio and was good enough. The judgement that a dedicated model is needed can wait until after you have tried the one already running.

Single-user services should not keep concurrency defaults

Defaults assume many users. A voice assistant usually handles one conversation at a time, so matching concurrent-request count and KV pool size to real usage returns memory without costing latency. Here that was 34%.

Most of the latency is arrangement, not the model

Remaining ~5 s	What it is	Does faster hardware help
Silence wait 1.0 s	A config value	No. The usual budget here is around 200 ms
STT 0.45 s	Not a dedicated model	A little
Generation 2.1–2.7 s	Waiting for the whole reply	No. This is structural
Synthesis 1.15 s	Model and serving	A little

The third row is the one. The model produces its first token in 0.035 s and 0.004 s across two runs — effectively immediately. The two-plus seconds of waiting is not the model being slow, it is the reply being generated in full before it is handed to synthesis. The way real-time agents get under a second is to chop tokens as they arrive and push them into synthesis, overlapping the two stages.

But overlapping has a floor

The obvious next move is to chop finer and overlap. In this configuration that does not work.

A cloning synthesis model conditions on a reference clip, so every call re-establishes that condition. Cut below a sentence and each fragment restarts its own prosody, and the seams are audible. The arithmetic does not work either. The next fragment’s first byte has to arrive before the current one finishes playing, and the first byte is 1.15 s. Measured speaking rate on this model converges near 5 syllables per second as sentences lengthen (5 syllables in 1.48 s, 11 in 1.96 s, 22 in 4.68 s, 44 in 8.84 s), which makes 1.15 s about five or six syllables. Long clauses clear that; short ones like “네” (“yes”) or “알겠어요” (“got it”) do not, and one of those is enough to break the audio at that seam. Per-call overhead of roughly 0.4 s also multiplies by fragment count.

So a sentence is the floor, and even then the estimate is about 1.8 s. The inputs to that calculation are measured; the 1.8 s itself is not yet tested. And sub-second is not reachable with this synthesis model, because a 1.15 s first byte exceeds a second by itself. Where the cited per-layer budget allows synthesis 40–100 ms, this build spends 1.15 s. That is the real floor.

Turn reasoning off per route, not globally

Disabling reasoning does make the model less capable, which is why it is disabled only on the voice route. The same model reasons normally in text chat. Voice exchanges are short round trips that want speed rather than depth; anything needing real thought can be typed. Splitting the model entry was how that distinction got expressed, not an abandonment of it.

The two alternatives that beat this on latency

The strongest objection deserves the straightest answer. On latency alone, both alternatives win.

Option	To first audio	What you give up
Commercial real-time voice API	500–800 ms target	Audio leaves the building. Usage billing. Subject to policy and model changes
Speech-to-speech model	About half a cascade	Stages collapse into one, so there is nowhere to reach in. Model choice and fine control go with it
This build (cascaded, fully local)	About 5 s	Latency

That speech-to-speech runs at roughly half a cascade is cited consistently, because the conversions and waits between stages disappear. (Dograh, Softcery)

So the reason to choose this shape is not speed. It is that audio stays put, models can be swapped, and you can reach in between the stages. All three pieces in this article exist because of that last property. In a speech-to-speech model there is nowhere to put a reasoning-off proxy or a streaming shim.

The inverse holds too. If latency is what matters most, this is the wrong build. For anything where the other party will not wait — phone support, for instance — a commercial API or a speech-to-speech model is the right answer. This shape fits when the user is one person who can wait five seconds and would rather the conversation stayed at home.

Two things still unverified

Worth stating plainly: the ~5 s figure sits on the same re-enabled streaming that section 5 declines to call fixed. It comes from those six turns, and if six turns are not proof of a fix then this number is exactly as certain as that. Nothing has stalled so far is the whole of what can be claimed.

The other open item is whether sentence-level overlap actually lands near 1.8 s. Both get written up when they are known.

The framework was not modified anywhere in this. Three times a piece went beside it, each reverting in about a minute, and one of them was reverted for real. When a framework has no field for what you need, adding the field is not the only option.