Bottom line first. Gemma-4-E4B is genuinely solid as a dedicated RAG fact-extraction model. The catch is audio — multimodal audio input does work through our llama.cpp server, but leave it on default settings and any clip past 4 seconds comes back as an empty response with no error at all. Tracing the cause, the model was burning its entire token budget “thinking,” and the real answer never made it out. This piece covers reproducing and fixing that trap on our own DGX Spark, the RAG extraction numbers, and where this model actually fits.

🔍 What Is Gemma-4-E4B, Exactly?

Gemma-4-E4B is the edge variant of Google’s Gemma 4 family (technical report published July 2, 2026). Per the official model card, it runs roughly 4.5B effective parameters (8B counting embeddings), a 128K context window, and a PLE (Per-Layer Embeddings) plus hybrid local/global attention architecture. Google’s own description calls it natively multimodal across text, image, video, and audio.

📊 RAG Fact Extraction: How Much Faster, Really?

Our memory system (RAG) constantly runs a fact-extraction pass that pulls facts out of conversations and documents into structured JSON. That job used to be shared with the main model, Gemma-4 26B — but extraction itself doesn’t need heavy reasoning. So we ran 5 Korean samples head-to-head against E4B and 26B (both with thinking off, same prompt).

E4B ran about 1.45x faster (180-190 tok/s vs. 125 tok/s), with valid JSON and zero hallucination across all 10 runs. That said, this is our own in-house comparison. One benchmark reports Qwen3-4B hitting 95.64% accuracy on RAG fact extraction — but that’s Gemma-3-generation turf, not a direct comparison against E4B. The same benchmark also found reasoning-tuned small models actually getting worse with RAG attached — “small model + RAG” isn’t an automatic win.

🎙️ Audio/STT: I Ran It Myself

The surprise finding was audio. E4B takes audio as well as text, so we asked it directly on our own DGX Spark (llama.cpp b9917). A short Korean TTS clip (“Hello. The weather’s really nice today,” 2.9 seconds) came back transcribed letter-perfect.

Then it broke. A slightly longer clip (5.2 seconds: “Raspberry Pi 5, Claudie, Mini, and Siwol tested the multimodal model”) came back as an empty string — no error at all.

The fix was simple — add these two fields to the request to turn thinking off:

{
  "chat_template_kwargs": {"enable_thinking": false},
  "reasoning_format": "none"
}

Sending the same clip again came back in 0.5 seconds, using just 30 tokens, at the same accuracy. Leaving thinking on and just raising the token budget to 1000 also works (2.4 seconds, 312 tokens), but disabling it outright was 5x faster at a tenth of the cost.

The transcription behavior itself was interesting too. “Siwol” (시월, a Korean name) got misheard as “October” (10월) — a genuine homophone mix-up, and it reproduced identically across two separate tests. Numbers weren’t transcribed literally either — spoken “twelve percent” and “two hundred milliseconds” came back written as “12%” and “200ms,” closer to cleaned-up notation than a literal phonetic transcript.

This result comes with real limits. Five clips, all clean macOS-TTS audio, not human speech. It’s an operational spot-check, not a formal CER/WER benchmark like KAIST’s Korean Whisper evaluation (14.66% CER for large-v2). And this was tested against the llama.cpp server directly — Ollama-layer multimodal bugs reported in the community are a separate layer of the stack, and this test can’t rule those out.

🌱 Where E4B Actually Stands in the Community

Community sentiment makes E4B’s role pretty clear. On r/LocalLLaMA, the model most often named as the local all-rounder isn’t E4B — it’s Gemma-4 26B A4B (or Qwen 3.5/3.6 MoE). One commenter put it directly: “Gemma 26B A4B is amazing… E4B was quite underwhelming in comparison.” E4B gets treated as an always-on edge/sidecar model, not the daily driver.

There’s also a disk-size debate. The official gemma4:e4b tag runs around 9.6GB at 4-bit quantization. “Why is a ‘4B’ model nearly 10GB” is a complaint that keeps coming up in the community. Effective parameter count and disk footprint are just different numbers.

🔧 Setup Notes (a.k.a. the War Stories)

Getting one small model running properly was not smooth sailing.

• Flash-attention crashed. Turning FA on for E4B on a Blackwell GPU killed the process mid-warmup at fattn.cu:110. The FA kernel’s shared-memory requirement exceeds this architecture’s limit — an already-known pattern. We run with FA off.
• The 128K ceiling. Trying to push the context window higher confirmed this model’s training limit is 128K. Anything past that gets silently truncated.
• The “thinking mode” trap, text edition. Extraction needs to be deterministic, so thinking was disabled entirely at the server level — which also closed off any path to turn it back on when actually needed. So the policy got flipped: the server now defaults to thinking ON, and only callers that need deterministic output (like the extraction step) turn it off per request.
• The “thinking mode” trap, audio edition. That policy flip came back around. Any newly wired-up path, like audio, inherits the server’s default (thinking ON) — the audio section above is exactly that trap reproducing itself.
• Routing is by port. This server routes by port, not by the “model name” in the request. The alias is display-only.

🤔 So Is E4B Always the Right Call?

There are honest counterarguments worth naming.

First, early-release instability was real. A goose issue reported that an older llama.cpp build couldn’t handle Gemma-4’s Jinja chat template and produced garbled output — upgrading llama.cpp fixed it. An open Zed issue reports a locally-served Gemma 4 writing files incorrectly or printing to console instead during agentic tasks. Running a freshly-released model locally means the whole stack needs to be current.

Second, there’s still no direct comparison against Qwen3-4B. The strongest small-model RAG numbers out there (Qwen3-4B at 95.64%) were measured on the Gemma-3 generation — E4B has never been put on that same stage. “E4B is the best small extractor” is unverified; “a strong candidate” is the honest framing.

Third, if real Korean STT is the actual goal, Whisper is still the safer bet. KAIST’s evaluation got fine-tuned Whisper-large-v2 down to 10.48% CER. Gemma’s audio path is confirmed working now, but that’s not the same as formally benchmarked accuracy. Short, low-stakes voice notes can reasonably go through the same model — but for serious transcription work, pairing with Whisper is still the practical answer.

🎯 So When Should You Actually Use E4B?

For deterministic, lightweight work like RAG fact extraction, and for short voice-note processing, E4B is the right weight class. If you’re after a daily-driver coding/agent model, 26B A4B or Qwen 3.5/3.6 MoE is what the community actually reaches for. If serious Korean transcription is the goal, Whisper remains the proven choice. And whichever audio path you use — check the thinking-mode setting first. Leave it on, test a couple of short clips, and it’s easy to conclude “it works” — then watch it go quietly silent the moment a clip gets even a little longer.

📚 References

• Gemma 4 Technical Report (arXiv:2607.02770, 2026-07-02)
• HuggingFace — google/gemma-4-E4B
• Google AI for Developers — Gemma 3n overview
• Benchmarking SLMs/SRLMs on System Log Severity Classification (arXiv:2601.07790)
• Evaluating ASR Systems for Korean Meteorological Experts, KAIST (arXiv:2410.18444)
• goose — Issue #9110, Gemma-4 Chat Template Support
• Zed — Issue #57416, Local Gemma4 + llama.cpp tools problems
• Ollama Library — gemma4
• r/LocalLLaMA — Gemma 4 26B is the perfect all around local model
• ollama/ollama — Issue #16532

See also — Building a Mac Mini remote dev server