#DGXSpark #llamacpp #SpeculativeDecoding #MTP #Qwen3 #LocalLLM #InferenceTuning
One flag, three answers: sweeping speculative decoding draft length on a local Qwen3.8-27B
We raised the speculative decoding draft length — --spec-draft-n-max — from 2 to 6 on a local llama.cpp server and got 23% more tokens per second on short prompts, 28% on long generations, and 65% on requests carrying a 45,000-token document. Same model, same quantisation, same hardware. One flag.
That is the headline and it is the least useful part of this post. Three things we found along the way are worth more: the knob is not monotonic, the variable that drives it is not the one we assumed, and we picked the wrong value first — from a table that looked finished and was not.
Where the idea came from, and what did not come with it
Reading someone else’s setup notes. syv-ai/qwen38-27b-rtx3090 documents getting Qwen3.8-27B onto a single RTX 3090, and its whole single-user chapter is about one axis: how many tokens to draft ahead.
That repo targets vLLM and contains no references to llama.cpp at all. Different engine, different GPU, different quantisation. Its numbers — 121 tok/s single-user, ~1,035 batch — are not comparable to ours and we are not going to put them side by side with our 32. What transferred was the question, not a figure: is draft length worth sweeping?
So we opened our own launcher. The value was 2, taken from the model publisher’s documentation when we first brought the model up, and never touched since. llama.cpp’s own default is 3, so we had been running below upstream without noticing.
What speculative decoding is buying
A language model emits one token per forward pass, and at 27B that pass is expensive. Most of the expense is memory bandwidth rather than arithmetic: reading the weights costs more than using them. That asymmetry means verifying several tokens in one pass costs not much more than verifying one.
Speculative decoding trades on it. Something cheap guesses the next few tokens, the full model checks them all in a single pass, and every guess that survives is free. Verification stops at the first rejection, so if the fifth of six drafts is wrong, everything after it is thrown away — and the model’s own token for that position, computed in the same pass, takes its place. Four accepted drafts therefore confirm five tokens, not four.
The guessing used to require a second, smaller model held in memory alongside the first. Current models ship a multi-token prediction (MTP) head inside the same file — a light layer riding the target model’s own hidden states, so there is nothing extra to load and its acceptance beats a separate draft model of comparable size. Qwen3.8-27B works this way; llama.cpp turns it on with --spec-type draft-mtp. --spec-draft-n-max is how far ahead that head is allowed to guess.
Check the flag is live before you sweep it
Our GGUF — the single-file weight format llama.cpp loads — reports nextn_predict_layers = 1. One MTP head. Depending on the implementation, llama.cpp could cap drafts at one or two and ignore the flag silently — in which case a sweep measures noise and reports it as tuning.
The discriminator is how many drafts are actually produced per decode step, which falls out of the server’s own timing fields:
steps = predicted_n - draft_n_accepted
drafts/step = draft_n / stepsRaise the setting; if that number does not follow, the flag is inert and the question is a llama.cpp capability question rather than a tuning one. It followed:
| n_max requested | drafts/step measured |
|---|---|
| 2 | 1.97 |
| 4 | 3.93 |
| 6 | 5.82 |
| 8 | 7.81 |
| 12 | 11.57 |
| 16 | 15.29 |
One head, iterated. Two minutes of work, and it is the difference between a result and a coincidence.
How it was measured
A small harness that waits until no other consumer holds a slot, sends a request, reports tokens per second, and prints drafts/step alongside. Three prompt shapes, 2–3 runs each, spread under 1% unless noted:
- Short prompt: an ordinary request generating about 400 tokens.
- Long generation: a short prompt producing 2,000 tokens.
- Long document: a 45,000-token document attached, with a question about it.
| Component | Value |
|---|---|
| Machine | NVIDIA GB10, Ubuntu 24.04.4 LTS, 20 threads |
| Memory | 121 GiB unified |
| Engine | llama.cpp build 10495 (commit 3dc7285b4, aarch64) |
| Model | Qwen3.8-27B, UD-Q4_K_XL |
| Serving | --ctx-size 524288 --parallel 2, 262,144 per slot |
| Speculation | --spec-type draft-mtp, MTP head inside the GGUF |
The engine version there came from resolving the running process’s executable, not from our own notes — which still recorded a build from a month earlier. In a performance write-up the version can change the conclusion, so it is worth reading the process rather than the record.
The sweep
| n_max | Short prompt | Long generation | 45k document |
|---|---|---|---|
| 2 (old setting) | 26.08 | 24.33 | 19.73 |
| 4 | 31.11 (+19%) | — | — |
| 6 (landed) | 32.20 (+23%) | 31.04 (+28%) | 32.51 (+65%) |
| 8 | 31.31 (+20%) | 28.82 (+18%) | 35.46 (+80%) |
| 12 | 27.07 (+4%) | 24.77 (+2%) | 39.12 (+98%) |
| 16 | 23.45 (−10%) | — | 39.56 (+100%) |
Read the columns rather than the rows and they disagree. Short prompts and long generations peak around 6 and decline. The document column climbs all the way to 16. At 16 the short-prompt figure is below the setting it was meant to replace. Whatever else is true, “turn it up until it stops helping” is not a strategy this table supports.
The baselines differ by column too. The document column starts at 19.73 against the short prompt’s 26.08 — a bigger KV cache means more attention per step. Our slowest real workload is the one this change helps most, which is a nicer result than the headline percentage.
The variable is grounding text, not context length
We went in expecting acceptance to rise with context length. It does not.
| Prompt shape | Draft acceptance at n_max = 6 |
|---|---|
| Short prompt | 63.4% |
| Long generation (2,000 tok) | 61.1% |
| 45k document | 89.7% |
A long generation accumulates context continuously and its acceptance stays with the short prompt’s. Only the document is different. The variable is whether the model has source text in front of it to copy from, not how much text there is.
Which makes sense once stated. Summarising, quoting or answering about a document means the next token is frequently already written somewhere above; the draft head is doing recall, not invention. Writing 2,000 tokens from a one-line prompt gives the head a long context that says nothing about the sentence coming next.
The same axis shows up from outside. In llama.cpp discussion #23053, the same configuration (n_max 5) gives 1.35× on code generation and 0.91× on creative writing — slower than not speculating at all. Syntax constrains what comes next; prose does not. Predictable text is accepted and open-ended text is not, measured twice from different directions.
Why 6, and the empty cell that chose 8
On the document column 12 or 16 would be better. We took 6 because this server has more than one consumer: a voice assistant sending short exchanges, a local model router sending medium work, and a retrieval system attaching long documents. All three shapes are real traffic.
So the rule was: take the value that wins the most prompt shapes. 12 wins the document column and loses the other two. 8 does the same thing — ahead of 6 by 9.1% on documents, behind it by 2.8% on short prompts and 7.2% on long generations. 6 wins two of three and is never less than 23% faster than the setting it replaced. (All three figures against the landed n=6.)
We landed 8 first
Worth saying plainly, because tuning posts usually end at the winning value and this is the part a reader can actually use.
The first value we adopted was 8. At that moment the table had a hole in it: n=8’s long-generation cell was measured and n=6’s was not. Compare the two on what was on the page and 8 wins.
Filling that one cell reversed the decision. n=6’s long generation is 31.04 against n=8’s 28.82 — 7.2% against the landed value.
Obvious once the table is complete; it was not obvious at the time. An incomplete table looks exactly like a complete one, and a blank cell distorts the conclusion without announcing itself. If you take one thing from this post, take that rather than the number 6.
Where this loses
Our numbers are a fact about this machine, not about the technique. Raising the same flag in the same direction has been measured making things slower, and more than once.
Issue #23752 reports Qwen3.5-9B on an M1 Max over the Metal backend: 25.3 tok/s with speculation off, then 22.4 at n_max 0, 21.9 at 2, 19.3 at 6. The reporter’s summary is blunt — “Ceiling (n-max) drives the regression. Floor (n-min) has no effect”, and “No configuration beats baseline.”
Discussion #23053, quoted above, is Metal as well. Every regression report we found is Apple Silicon, with draft-verification overhead exceeding the speculative gain on that backend. If you are running a local model on a Mac and you follow this post, you may well get slower. That is the single most important sentence here for a reader who is not us.
A lower acceptance rate can be faster
It is tempting to optimise acceptance. That is also wrong. Discussion #25198, on Gemma-4-31B:
| Configuration | Global TPS | Acceptance | Avg draft length |
|---|---|---|---|
n-max 16, p-min 0.8 |
119.53 | 0.557 | 5.20 |
n-max 4, p-min 0.0 |
99.29 | 0.678 | 3.71 |
The configuration with acceptance 17.8% lower is 20.4% faster. Acceptance is a diagnostic, not the objective; what you are actually buying is confirmed tokens per step, which is closer to acceptance multiplied by draft length.
What we did not measure
The table above contains this post’s largest limitation. The configuration that won it sets p-min to 0.8. --spec-draft-p-min stops the draft head early when it is not confident, and llama.cpp’s default is 0.00.
Our entire sweep ran at 0.00, because we never set the flag. That changes what the n=16 row means. It is not evidence that long drafts are slow; it is evidence that long drafts with no brake are slow. We have not measured the combination, and that is the next experiment.
A practitioner running the same model on a 24 GB M4 Mac mini reported the same axis: holding n_max at 4 and moving p-min from 0.0 to 0.8 took 3.05 tok/s to 5.71 — the default was costing 46.6% of throughput at the same draft length. On that machine, though, 5.71 was still under the 5.91 they measured with MTP off entirely. p-min stopped the collapse; it did not recover the baseline. We could not open that post ourselves — it is behind an access wall and a colleague retrieved it — so read it as one practitioner’s report rather than a settled figure.
Three more gaps, stated rather than glossed:
- Concurrency was re-measured, and the first attempt was wrong. The original note recorded two per-stream figures and an aggregate that did not equal their sum, and the raw output was gone, so the numbers could not be repaired — only replaced. Re-run at the landed n=6: two simultaneous 400-token streams gave aggregates of 59.54, 62.70 and 61.76 tok/s across three runs, the median run being 31.55 + 30.21 = 61.76 against a 36.26 single-stream median from the same session. 1.70×, so speculation still scales across both slots rather than thrashing on wasted draft compute.
- That 36.26 is not comparable to the 32.20 in the sweep table — different prompt, different day. The single-stream reference was measured inside the same run precisely so the ratio means something; quoting it against the table would not.
- Output quality was never checked. Speculative decoding should be output-identical in principle. We did not verify that, and this post does not claim it.
- One machine, one model, one quantisation. See the section above.
If you want to try this
- Confirm the flag is live first. Raise n_max and check that drafts/step follows. If it does not, you have a capability question, not a tuning one.
- Measure the shapes your traffic actually has. Short requests and long documents have different optima. Tuning on one shape costs you on the others.
- Fill the table before choosing. A value picked from a table with a hole in it can reverse when the hole is filled. Ours did.
- Measure your no-speculation baseline too. Without it you cannot tell that you have made things worse.
- Write down the rollback. Ours is a one-line sed in a comment beside the flag, with the backup filename next to it.
And keep in mind where the number you are replacing came from. Ours was 2, from the model publisher’s documentation, below llama.cpp’s own default of 3. Neither is wrong — a default is a value that worked on somebody’s machine, and that machine is not yours.