What's in this revision (vs the W4A4 revision)
This revision replaces the text-transformer embeddings (embed_tokens, the tied lm_head, and the per-layer embedding table embed_tokens_per_layer) with FP8 e4m3 + per-row F32 scales. Everything else — the W4A4-quantized 42 text layers, the BF16 vision tower and audio encoder, the tokenizer, the configs — is byte-identical to the W4A4 revision of this repo.
Measured on the same hardware, same time window, same harness (RTX 5090 shared GPU, vLLM 0.28, MTP K=5, greedy, 2 runs × 6 fixed prompts, streaming with include_usage):
Table with columns: W4A4 revision, FP8-embeds revision (this) | W4A4 revision | FP8-embeds revision (this) |
|---|
| Checkpoint size | 9.7 GB | 6.2 GB |
| Decode (long form, MTP K=5) | 512.7 tok/s | 516.0 tok/s |
| Decode (mixed workload, MTP K=5) | 500.0 tok/s | 533.8 tok/s |
| Decode without MTP (K=0) | 241.9 tok/s | 242.7 tok/s |
| Drafter acceptance (mean, per-position) | 0.2768 | 0.3049 |
KV cache at --gpu-memory-utilization 0.40 | 126 087 tokens (3.00x) | 260 888 tokens (6.21x, +107%) |
| Embedding tables in VRAM | 5.25 GiB BF16 (PLE alone) | 2.62 GiB FP8 (−2.62 GiB) |
Decode speed is flat (within run-to-run noise) — the FP8 conversion buys memory, not speed: a smaller resident PLE table leaves ~2.6 GiB for KV cache, which at gpu-memory-utilization 0.40 (the shared-GPU setting) is +134 801 tokens of KV cache (260 888 vs 126 087, a 2.07× increase). On a dedicated GPU you can raise --max-model-len to the full 131 072 window and still keep the KV cache the W4A4 revision had at 42 000.
Behavior on the 8-prompt greedy battery (fact traps + controlled questions): 1/8 byte-identical to the W4A4 revision, 1/8 byte-identical to the W4A16 revision, while the W4A4 vs W4A16 reference pair is 0/8 — i.e. FP8-embeds sits between the two revisions on the same confidence axis. Fact traps are answered the same; post-cutoff traps hedge slightly more than W4A4, less than W4A16. No new error categories appeared.
Why the checkpoint needs a patch to serve
vLLM 0.28's VocabParallelEmbedding has no native path for quantized embeddings — it assumes the loaded weight is the dtype the forward pass computes in. This checkpoint stores the two embedding tables as FP8 e4m3 with per-row F32 scales (2 bytes per weight → 1 byte per weight, plus a 4-byte scale per row). Serving the checkpoint unpatched loads the FP8 tables and casts them to BF16 at load time with no rescale, so every embedding is ~448× too small and the model degenerates (repeated <unused53> + EOS).
sitecustomize.py in this repo fixes that on PYTHONPATH:
embed_tokens_per_layer (262 144 × 10 752, the PLE table) stays resident in VRAM as FP8 e4m3. The patched VocabParallelEmbedding.forward gathers the selected rows as FP8 (110 KB/token), casts to BF16, multiplies by the per-row scale, and views as (n, dim). Bit-identical to loading the BF16 table.embed_tokens (262 144 × 2 560, tied to lm_head) is dequantized to BF16 once, in-place, at process_weights_after_loading. Doing the per-row dequant on the forward path for the tied lm_head would add a per-token F8→BF16 gather to the logits path for no gain — this table is only ever read once per token as a scatter-lookup, so a one-shot BF16 copy is the right call.- The MTP drafter (
google/gemma-4-e4b-it-assistant) carries its own BF16 embed_tokens (262 144 × 256). The patch marks parameters by data_ptr at weight_loader time and only processes parameters that received FP8 data, so the drafter's BF16 table is skipped — leaving it untouched is what keeps MTP acceptance at the W4A4-revision level instead of collapsing to 0/15.
Verified on vLLM 0.28, Python 3.12, CUDA 13, RTX 5090 (32 GB, shared): patch logs F8 load into param (262144, 10752) + F8 load into param (262144, 2560), then PLE F8-resident (262144,10752) scale_max=0.002563 and small table dequanted (262144,2560) scale_max=0.001238. KV cache goes from 45 573 tokens (unpatched, 1.09x concurrency at 42 000) to 260 888 tokens (6.21x) at the same gpu-memory-utilization 0.40 — the 2.62 GiB of freed PLE VRAM plus the FP8-resident tables translates directly into a 2.07× larger KV cache.
What's inside
Table with columns: Component, Format, Notes| Component | Format | Notes |
|---|
| Text transformer (42 layers, 2.56K hidden) | NVFP4 W4A4 (group 16) | ModelOpt 0.46.0, modelopt_fp4 — activations quantized to FP4, native Blackwell tensor-core GEMM |
| Vision tower (16 layers) | BF16 | excluded from quantization |
| Audio encoder (12 layers) | BF16 | excluded from quantization |
lm_head / embed_tokens (262 144 × 2 560, tied) | FP8 e4m3 + per-row F32 scale | dequantized to BF16 once at load by sitecustomize.py |
Files: single model.safetensors (6.2 GB, 3,375 tensors — two of them FP8 e4m3 with accompanying weight_scale tensors) + tokenizer/processor/config + sitecustomize.py (required for serving).
Serve with vLLM (W4A4 + B12X kernel — the fast path)
Verified on vLLM 0.28, Python 3.12, CUDA 13, RTX 5090 (32 GB, shared). W4A4 activates the native FP4 tensor cores via the B12X GEMM kernel, which requires the b12x extra:
# One-time venv setup
uv pip install 'vllm==0.28.0' b12x==1.2.4
uv pip install apache-tvm-ffi==0.1.11 # THE pin — see gotcha below
# Fetch the serving patch from the repo (this revision's version merges the
# B12X pre-import fix AND the FP8-embedding load/forward fix in one file)
mkdir -p ~/.vllm-patches && cd ~/.vllm-patches
wget https://huggingface.co/Adolphsson/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4/resolve/main/sitecustomize.py
# Serve
export CUDA_HOME=$VENV/lib/python3.12/site-packages/nvidia/cu13
export PATH=$VENV/bin:$VENV/lib/python3.12/site-packages/nvidia/cu13/bin:$PATH
export LD_LIBRARY_PATH=$VENV/lib/python3.12/site-packages/nvidia/cu13/lib:$LD_LIBRARY_PATH
export FLASHINFER_CUDA_ARCH_LIST=12.0
export CCCL_DISABLE_CTK_COMPATIBILITY_CHECK=1
export PYTHONPATH=$HOME/.vllm-patches # the sitecustomize.py from the repo
vllm serve Adolphsson/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4 \
--linear-backend b12x \
--port 8000 --host 0.0.0.0 \
--served-model-name gemma4-e4b-nvfp4 \
--max-model-len 42000 \
--gpu-memory-utilization 0.40 \
--max-num-seqs 2 \
--limit-mm-per-prompt '{"image":4,"video":2,"audio":4}' \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--enable-prefix-caching \
--mm-processor-cache-gb 2 \
--spec-model google/gemma-4-e4b-it-assistant \
--spec-tokens 5
--gpu-memory-utilization is a fraction of the card's 32 GB. Of that budget, ~7.5 GB is fixed cost (6.2 GB weights + runtime + CUDA graphs) regardless of setting — everything above that becomes KV cache. That's why the KV-cache column jumps faster than the utilization figure: it scales with the absolute budget, not the percentage.
Table with columns: Setting, --gpu-memory-utilization, --max-model-len, VRAM used (process), KV budget, KV cache| Setting | --gpu-memory-utilization | --max-model-len | VRAM used (process) | KV budget | KV cache |
|---|
| Minimum (responsive voice/assistant) | 0.28 | 42 000 | ~9.6 GB (9 582 MiB) | ~1.4 GB | 66 247 tokens |
| Comfortable (this README's default) | 0.40 | 42 000 | ~13.4 GB (13 432 MiB) | ~5.3 GB |
Measured by booting the server at each setting and reading the vLLM process's own VRAM via nvidia-smi --query-compute-apps (excluding other tenants on the card).
0.28 is the minimum that boots a 42K-context server — 0.26 OOMs ("0.82 GiB needed, 0.38 GiB available"). It is a knife-edge: the model + a 42K ceiling just fits. For anything more than short assistant turns, 0.40 is worth the ~4 GB: 4× the KV cache.- For the full 131K native window, use
0.72 → ~22.5 GB, 839K-token KV cache.
Fitting a full voice pipeline on one 5090. If the model's built-in audio isn't enough and you want to run external STT + this LLM + a high-quality low-latency TTS (e.g. OmniVoice) on the same card: the 0.28 setting holds the model to ~9.6 GB, leaving ~22 GB free of the 32 GB — comfortable headroom for an fp16 Whisper-class STT and an fp16 OmniVoice-class TTS alongside it, all on one card, minimal latency.
With the av (PyAV) + soundfile packages installed in the venv, all three modalities are accepted at the API level (pip install av soundfile).
B12X gotchas (three of them, all hit, all documented)
apache-tvm-ffi must be exactly 0.1.11. The b12x package transitively drags in apache-tvm-ffi 0.1.13.post3, which double-registers the __ffi_repr__ type attribute and crashes the engine after CUDA-graph capture with a C++ abort (tvm::ffi::Error: TypeAttr __ffi_repr__ is already registered for type index 132). vLLM 0.28 pins 0.1.11 — respect the pin.- CCCL toolkit compatibility check. With the
nvidia/cu13 venv layout, FlashInfer's JIT compiles against nvcc 13.3 while the bundled headers are 13.0; CCCL's cuda_toolkit.h aborts on the minor mismatch. The CCCL_DISABLE_CTK_COMPATIBILITY_CHECK=1 env var is not read by the C preprocessor, so patch the header in the venv: in .../flashinfer/data/cccl/libcudacxx/include/cuda/std/__cccl/cuda_toolkit.h, change the guard #ifndef CCCL_DISABLE_CTK_COMPATIBILITY_CHECK to #if 0.
- vLLM 0.28's calls inside the traced path, which fullgraph forbids (). Fix: a on that pre-imports the b12x submodules and rebinds the kernel module's accessor functions to direct module references (a intercepting post-execution).
If you would rather skip all three, the fallback is the W4A16 variant (previous revision of this repo) served with --quantization modelopt_fp4 and no --linear-backend flag — same model, ~27% slower decode (see Benchmarks).
The quantized audio encoder hits a non-contiguous tensor in Gemma4ClippedLinear. In transformers/models/gemma4/modeling_gemma4.py, change the Gemma4ClippedLinear.forward line:
hidden_states = self.linear(hidden_states.contiguous())
This is the only quantized multimodal tower (vision/embeddings are excluded from NVFP4), which is why image input works unpatched but audio input crashes without it.
MTP speculative decoding (~2x decode speed)
This model ships with native MTP support in vLLM 0.28 (Gemma4MTP):
--spec-model google/gemma-4-e4b-it-assistant \
--spec-tokens 5
Measured on the RTX 5090, vLLM 0.28 (fresh server per K, 300-token cap, greedy, streaming with include_usage, 2 runs × 6 prompts). Both revisions measured in the same session, same GPU, same harness — directly comparable:
Table with columns: --spec-tokens, W4A4 long-form, FP8-embeds long-form, W4A4 mixed, FP8-embeds mixed, FP8 accept, W4A4 accept--spec-tokens | W4A4 long-form | FP8-embeds long-form | W4A4 mixed | FP8-embeds mixed | FP8 accept | W4A4 accept |
|---|
| 0 (off) | 241.8 | 242.5 | 244.5 | 245.2 | — | — |
| 1 | 362.4 | 365.1 | 366.4 | 369.1 |
All figures in tok/s. "long-form" = token-weighted average of runs ≥300 tokens (story/long/math prompts); "mixed" = all runs including short-answer prompts. Acceptance = accepted_tokens / draft_tokens from vllm:spec_decode_* Prometheus counters.
W4A4's native FP4 GEMM is the single largest decode-speed win available on this hardware — the activation quantization removes the bf16 dequant/multiply penalty that weight-only NVFP4 carries. Note the win is MTP-dependent: without the drafter, both revisions decode at ~242 tok/s (flat). The FP4 activation gain pays off in the compute-bound draft-verify path, where the speedup is 2.1× (K=5 vs K=0).
FP8-embeds matches or slightly exceeds W4A4 across all K — the FP8 embedding tables are the same size per token as BF16 (after per-row dequant in forward), so decode speed is identical. The KV-cache gain (2.07×) is the practical benefit on shared GPUs.
K=5 is the recommendation for shared GPUs. The acceptance curve is monotonically decreasing in K (0.67 at K=1 → 0.30 at K=5) but the net speed keeps rising because each iteration proposes more tokens than it wastes — diminishing returns above K=5 (the old revision's K=6 data showed +2% over K=5, not worth the extra drafter re-execution on shared hardware).
Previous revision (W4A4 only, old wall-clock harness, 600-token cap)
For reference, the W4A4 revision (this repo, earlier commit) was benchmarked with a different harness (non-streaming wall-clock, 600-token cap, 4 prompts):
Table with columns: --spec-tokens, Long-form decode, Mixed workload--spec-tokens | Long-form decode | Mixed workload |
|---|
| off | 220 tok/s | 198 tok/s |
| 4 | 521 | 410 |
| 5 | 580 | 467 |
| 6 | 591 | 469 |
These are the figures cited in the W4A4 revision's README. The 600-token wall-clock harness reports slightly higher numbers than the 300-token streaming harness (longer runs amortize the prefill better; the wall-clock metric includes prefix-cache hits). Both harnesses agree on the relative K=0 vs K=5 speedup (~2.1–2.6×).
Quick test
curl http://localhost:8000/v1/chat/completions -d '{
"model": "gemma4-e4b-nvfp4",
"messages": [{"role":"user","content":"Write a short poem about cats"}],
"max_tokens": 300, "temperature": 0.7
}'
Multimodal (vision) via the OpenAI-compatible input_image content part; audio via input_audio (see known issue below).
Known issues (be honest section)
sitecustomize.py is required on PYTHONPATH (this revision only). The FP8 embedding tables are loaded through a patched VocabParallelEmbedding (see "Why the checkpoint needs a patch to serve" above). Without it the engine comes up, health returns 200, but the model degenerates: repeated <unused53> + immediate EOS, and the KV cache is ~4× smaller because the FP8 tables get cast to BF16 at load with no rescale. Two failure modes I hit myself, both silent: (a) PYTHONPATH pointing at a directory whose sitecustomize.py is a different file — Python imports only the first sitecustomize.py on the path, so PYTHONPATH=/root/b12x:/root/fp8embed ran the old B12X-only patch and the FP8 fix never loaded; (b) the drafter's own BF16 embed_tokens matching the target's shape — the patch resolves this by marking parameters by data_ptr at weight_loader time and only processing parameters that received FP8 data, so the drafter is skipped and MTP acceptance stays at the W4A4-revision level instead of collapsing to 0/15.- in transformers' gemma4 module — add before the quantized linear. Without it the engine dies on the first request. Verified on the W4A16 revision (transcription, name extraction, word counting all work); the W4A4 text path is kernel-invariant for the audio tower (still BF16), but re-verify on your setup.
Benchmarks & provenance
Raw JSON results live in the author's notes linked below. Methodology: same GPU, same time window, 2 runs per configuration, greedy, 4 fixed prompts, 600-token cap.
- Built: August 2026, on an RTX 5090 LXC container
- W4A4 re-quantization: August 2026, ModelOpt 0.46.0
NVFP4_DEFAULT_CFG, calibration on 24 self-generated sequences (Swedish chat, English technical, code, math)
- FP8 embedding conversion (this revision): September 2026, deterministic per-row
scale = row_max / 448 (F8 e4m3 max), applied to embed_tokens and embed_tokens_per_layer only — all other tensors byte-identical to the W4A4 revision
- Benchmark harness (this revision): 2 runs × 6 fixed prompts (3 short, 3 long-form, 300-token cap), greedy, fresh server per K, streaming with
include_usage (token count from the final usage chunk — vLLM chunks multiple tokens per SSE event under MTP, so per-delta counting undercounts by ~2.9× at K=5; fixed in this revision's harness), acceptance from vllm:spec_decode_* Prometheus counters — same harness run against both the W4A4 and FP8-embeds revisions on the same GPU in the same session
- Base fine-tune: HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive (GGUF Q8_K_P)
License
Lineage
- Original base model:
google/gemma-4-e4b-it
- Source model:
HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive
- Direct contribution: NVFP4 W4A4 quantization, verified B12X native-FP4 serving, verified MTP speculative decoding compatibility, and Blackwell-oriented optimization (previous revision); FP8 e4m3 embedding conversion with per-row scales plus the
sitecustomize.py serving patch (this revision)
This model is a technical optimization/conversion of the upstream HauhauCS model. The uncensored behavior was created by HauhauCS, not by this release.
Modifications
- NVFP4 W4A4 quantization (4-bit weights + activations) using NVIDIA ModelOpt
- Verified B12X native-FP4 GEMM serving on SM120 (RTX 5090)
- Verified compatibility with Gemma 4 E4B MTP speculative decoding
- Blackwell-oriented optimization
- FP8 e4m3 conversion of the two text-embedding tables with deterministic per-row F32 scales, plus the
sitecustomize.py serving patch that loads them (this revision)
Gemma 4 is Apache-2.0 licensed. This release is based on the Gemma 4
E4B model and the HauhauCS derivative described above. Please refer to
the upstream model for its original licensing and attribution information.
Full license text: LICENSE ·
Upstream attribution: NOTICE