Gemma 4 E4B MTP speculative decoding llama-swap llama.cpp Open WebUI

🔍 Why I Dropped Ollama

When running local LLMs on MacBook M2 Max, Ollama is the first thing most people reach for. The install is trivial, and a web UI plus model pull come built in. But to get the most out of Gemma 4 E4B, Ollama falls short.

The core issue is MTP (Multi-Token Prediction). Gemma 4 supports speculative decoding via a drafter model to accelerate inference. Ollama does not currently support Gemma 4’s MTP — which means llama.cpp is required.

Measured result: ~81 tok/s with Gemma 4 E4B + MTP on M2 Max 32GB. That beats Ollama’s MLX engine running an 8B model (65–75 tok/s). Despite being a 4B model, that’s 35–60% higher throughput than running E4B in llama.cpp without MTP (50–60 tok/s). This post documents the full setup and what I ran into along the way.

📋 Setup — Three Components

Install base is ~/Git/local-llm/. MacBook-specific config, gitignored.

llama-swap exposes an OpenAI-compatible API on localhost:11434, and Open WebUI connects to that endpoint. When a user picks a model from the dropdown, llama-swap starts the corresponding llama-server and unloads it automatically after 600 seconds of inactivity.

🧠 MTP — The Core Technique

On this MacBook, the Gemma 4 E4B acceptance rate runs 42–57%. On the DGX Spark server it climbs to 84–93%. Consistent context patterns give the drafter better predictive accuracy. Per the Google announcement (2026-05-05), MTP is Gemma 4’s primary inference acceleration technique.

Two configuration points matter.

• --flash-attn conflicts with MTP. --flash-attn off is mandatory.
• The sweet spot for draft depth is --spec-draft-n-max 4. Beyond 4, returns diminish. On DGX, dropping n-max to 2 yielded +35% (80→109 tok/s).

🛠️ Model File Setup

Two models are currently running. The default is Unsloth’s UD-Q4_K_XL, using QAT (Quantization-Aware Training) for a quality edge over standard Q4_K_M. The uncensored model is the HauhauCS Aggressive community fine-tune.

The MTP drafter (Q8_0) is ~156MB and occupies roughly 1.5–2GB additional RAM. On a 32GB machine, that’s comfortable headroom. Both models support vision via a separate mmproj (F16.gguf).

One quirk with the uncensored model: open-ended prompts trigger lengthy English reasoning that floods the output, pushing actual response content out of view. Fixed by suppressing thinking output via the Open WebUI system prompt. Recommended sampler settings: temp=1.0, top_p=0.95, top_k=64, --jinja.

⚙️ Runtime Configuration

A unified startup script launches llama-swap and Open WebUI together.

# Start everything
./start-webui.sh

# OpenAI-compatible API endpoint
http://localhost:11434/v1

# Stop all services
pkill -f "open-webui serve" \
  && pkill -f "llama-swap" \
  && pkill -f "llama-server"

Key settings from llama-swap.yaml. ${PORT} is assigned automatically by llama-swap.

healthCheckTimeout: 120
ttl: 600  # auto-unload after 10 min idle

models:
  gemma-4-E4B:
    cmd: |
      llama-server
        --model /path/to/main-Q4_K_XL.gguf
        --model-draft /path/to/draft-Q8_0.gguf
        --mmproj /path/to/mmproj-F16.gguf
        --spec-type draft-mtp
        --spec-draft-n-max 4
        --flash-attn off
        --port ${PORT}

  gemma-4-E4B-uncensored:
    cmd: |
      llama-server
        --model /path/to/uncensored-Q4_K_M.gguf
        --mmproj /path/to/mmproj-uncensored-F16.gguf
        --port ${PORT}

📊 Speed Comparison — Measured and Referenced

M2 Max (memory bandwidth 400 GB/s) benchmarks — direct measurements and external data.

External hardware references: GitHub karany97/llamacpp-gemma4-mtp, Reddit — RTX PRO 6000 3.34x verified, Medium — local inference benchmarks (2026-05-22).

GPU environments push acceptance rates to 70–100% due to repetitive coding patterns. The MacBook Metal environment sits at 42–57%, but that still translates to a measurable throughput gain. MTP support in llama.cpp landed via PR #23398 (b9549, 2026-06-07). Reddit LocalLLaMA continues to report +40% speedup consistently.

🔧 Troubleshooting — Key Issues

The thinking leak on the uncensored model initially looked like a model defect. Turned out the entire English reasoning chain was printing before the actual content, pushing it offscreen. Suppressed it via the Open WebUI system prompt. Not a model bug.

📚 References

• Google — Gemma 4 MTP official announcement (2026-05-05)
• llama.cpp PR #23398 — MTP merged (2026-06-07)
• Reddit LocalLLaMA — MTP Gemma 4 +40% speedup
• Reddit — RTX PRO 6000 3.34x verified
• GitHub — llamacpp-gemma4-mtp benchmark harness
• Medium — Gemma 4 MTP local inference benchmarks (2026-05-22)
• Hacker News — MTP discussion (2026-05-12)
• AI Weekly — llama.cpp Gemma 4 E4B MTP support
• Unsloth Gemma 4 E4B GGUF (HuggingFace)
• HauhauCS Gemma-4-E4B-Uncensored (HuggingFace)

✅ Summary

For efficient local LLM usage on M2 Max 32GB, the llama.cpp + llama-swap combination is the right choice. Setup is more involved than Ollama, but MTP support and fine-grained parameter control justify the effort. Gemma 4 E4B + MTP clocks ~81 tok/s — a 4B model that outpaces Ollama’s MLX engine on an 8B model (65–75 tok/s). All components are open source under MIT or the Gemma License.

llama-swap’s TTL-based idle unload is genuinely useful for multi-model setups. Model switching is seamless, and unused models release from memory after 10 minutes, preventing unnecessary SSD wear.

More on local LLM configuration in the Local AI category.