thinking-token reasoning-budget llama.cpp Gemma-4 Qwen3
🔍 Background — Why the Translation Output Was Broken
The Atelier blog translation output had been consistently poor. Sentences were awkward and kept cutting off mid-way. Initially diagnosed as a model quality issue, but tracing the root cause pointed elsewhere.
llama.cpp manages thinking tokens and answer (content) tokens under a single max_tokens budget. When thinking is active and the model burns excessive thinking tokens on trivial tasks, there’s no budget left for actual content generation. The truncated translations were not a model quality problem — they were a budget exhaustion problem.
The max_tokens budget is shared between thinking tokens and content tokens. When the model over-thinks on trivial tasks, the budget runs out before content generation, returning finish_reason=length.
📊 Measurement — 3-Slot Matrix
Ran measurements across all 3 DGX slots. Fed each slot a trivial task (1-line translation) and a reasoning task (math problem) to measure thinking token consumption. Conditions: max_tokens=2048, finish_reason=stop.
| Slot | Model | Trivial Task (1-line translation) | Reasoning Task (math) |
|---|---|---|---|
| model1 | Gemma-4-26B-A4B | think 1,601c / 5.4s | think 742c / 4.0s |
| model2 | Gemma-4-E4B | think 943c / 2.3s | think 253c / 1.4s |
| model3 | Qwen3.6-27B | think 4,498c / 63.5s | think 906c / 23.4s |
Qwen3.6 (model3) consumed 63.5 seconds for a single-line translation. Over a minute for a simple translation — not a math problem. This is the classic ‘over-thinking’ failure mode of the Qwen3 family.
🔬 Community Cross-Check
Verified that measured data aligns with model specs. This step confirms whether the directly measured numbers match the model’s design intent.
Gemma 4 — Native Thinking, enable_thinking Flag
Gemma 4 supports native thinking mode, controlled via the enable_thinking flag. The official documentation doesn’t specify a separate budget guide, but community convention governs operation. Removing previous thinking blocks during multi-turn conversations is an official requirement.
enable_thinking=false. Only the E4B model fully omits this block.
Qwen3 — Hybrid Thinking, Severe Over-Thinking
The Qwen3 family has a hybrid thinking architecture. It can be disabled via enable_thinking=false or the /no_think system prompt, but completely disabling it degrades tool-calling performance. The conclusion from the official thinking_budget documentation is that setting a budget cap — rather than fully disabling thinking — is recommended on agentic paths.
llama.cpp — –reasoning-budget Spec
Per the official README, valid values for --reasoning-budget are: -1 (unlimited) / 0 (terminate immediately) / N > 0 (token cap). Default is -1.
There’s an important priority rule. Per GH disc 21445, per-request thinking_budget_tokens only takes effect when the server is set to --reasoning-budget=-1. If the server flag is 0 or positive, per-request settings are completely ignored.
</think> and continuing the response — recovers it to 89%.
🛠️ Verification — Per-Request Control (build b9743)
After confirming theory, verified that per-request control actually works. Targets were model3 (Qwen3.6-27B) and model2 (Gemma-4-E4B), applied without restart.
| Condition | content | reasoning | Elapsed | Speedup |
|---|---|---|---|---|
| model3 baseline | 106c | 7,262c | 102.7s | — |
model3 thinking_budget_tokens=256 |
106c | 827c | 16.1s | 6.4× |
model3 enable_thinking=false |
109c | 0c | 2.4s | 43× |
model2 enable_thinking=false |
110c | 0c | 0.7s | — |
Per-request control works as expected. model3 baseline 102.7s dropped to 16.1s with budget cap applied. Fully disabling thinking brings it down to 2.4s. Content token count was identical across all three conditions at 106–109c. This proves the issue was thinking token overconsumption — not a model defect.
⚙️ Implementation — terry-llm v0.4.5 (commit 220b85a1)
Rolled the verification results into the terry-llm gateway. Key changes:
1. providers/router.py — Per-Request Budget Passing
Passes the client-supplied thinking_budget_tokens to the llama.cpp server via extra_body.
# providers/router.py
if request.thinking_budget_tokens is not None:
extra_body["thinking_budget_tokens"] = request.thinking_budget_tokens2. server.py — Agentic Path Budget Cap
Reads a cap from config to prevent unbounded thinking token consumption in the agentic loop. Default is 4096 tokens.
# server.py _do_agent()
budget = cfg.defaults.agent_thinking_budget_tokens
# config.yaml:
# agent_thinking_budget_tokens: 40963. agent/loop.py — Honest Empty-Content Handling
Distinguishes finish_reason=length (truncated) from finish_reason=stop (normal). Returns empty content with a truncated flag. Previously, empty strings were returned as if they were valid responses.
4. run_http.py — Session-Independent HTTP Service (:8090)
A standalone HTTP service on port :8090, independent of any MCP session. The HTTP endpoint stays up even when a session terminates.
✅ Final 3-Slot State
| Slot | Model | Status | Handling |
|---|---|---|---|
| model1 | Gemma-4-26B-A4B | ✅ Optimized | Gateway budget cap (4096) + one-shot path enable_thinking=false |
| model2 | Gemma-4-E4B | ✅ Already optimized | terryragsys applies enable_thinking=false and GBNF constrained decoding |
| model3 | Qwen3.6-27B | ✅ Intentionally preserved | Dedicated coding model — thinking capability retained |
Actual changes applied to model1 only. model2 was already optimized; model3’s thinking capability is essential for coding tasks, so it was preserved.
⚠️ Pitfalls
- DGX server must keep
--reasoning-budget=-1: If this isn’t -1, per-requestthinking_budget_tokensis ignored. Changing the server setting invalidates all per-request control at once. - Transition message recommended when using positive budget cap: Forcefully terminating thinking without injecting
</think>causes a sharp performance drop. HumanEval drops ~16 percentage points for Qwen3 — manual implementation required if the library doesn’t support it. - Gemma multi-turn — must remove previous thinking blocks: This is an official requirement. Multi-turn pipelines must strip thinking blocks from previous turns.
- Gemma (non-E4B) — outputs empty thought blocks: Even with
enable_thinking=false, empty blocks like<|channel>thoughtmay appear. Parsing logic must handle this.
📚 References
- Gemma 4 Thinking Official Docs — Google AI for Developers
- Gemma 4 Model Card — Official spec for E4B thinking disable behavior
- llama.cpp server README —
--reasoning-budgetofficial spec - GH disc 21445 — Per-request
thinking_budget_tokenspriority rule - GH disc 21338 — Gemma 4 E4B thinking disable bug report
- r/LocalLLaMA — HumanEval benchmark and transition message effect
- Qwen3 thinking_budget official docs — QwenLM
Related DGX infrastructure operation logs are available in the infra archive, including previous model mixing cases and latency improvement results.
✅ Summary
The problem was straightforward. With thinking enabled, the max_tokens budget is shared between thinking and content — when the model over-thinks on trivial tasks, content gets truncated. Qwen3 consumed 63.5s for a 1-line translation; baseline reasoning tokens hit 7,262c.
The fix was equally clear. Keep the DGX server at --reasoning-budget=-1, pass per-request thinking_budget_tokens from the gateway, and apply a 4096-token cap on the agentic path. Applied to model1 only, without restart.