GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
aday777
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
aday777
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
At the time it was made, no public checkpoint combined all three of:
NVFP4_AWQ, which stock vLLM rejects), andExisting NVFP4 Gemma-4 builds were either the stock/censored model, the
NVFP4_AWQ variant, or shipped only in ARM-only images. This repo fills that
gap by self-quantizing a bf16 abliterate to plain NVFP4.
This is a modified derivative of google/gemma-4-31B-it. Two changes:
embed_tokens,
per-layer attention o_proj and MLP down_proj). The ablation is baked
into the weights — no inference-time hooks required.Linear layers only. The vision tower,
multimodal projector, audio modules, token embeddings and the (tied)
lm_head are left in higher precision. See recipe.yaml in this repo for
the exact scheme and ignore list.sm_120, e.g. RTX PRO
Blackwell). NVFP4 activation quantization needs hardware FP4 support.compressed-tensors NVFP4 support.Gemma 4 ships an official EAGLE/MTP-style draft,
google/gemma-4-31B-it-assistant,
which vLLM drives as a native multi-token speculator. Use "method": "mtp"
(passing "draft_model" for a Gemma-4 assistant silently disables MTP):
bash
vllm serve aday777/gemma-4-31B-it-abliterated-NVFP4 \--tensor-parallel-size 2 \--max-model-len 8192 \--gpu-memory-utilization 0.88 \--speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 3, "method": "mtp"}' \--port 8003
Drop the --speculative-config line to serve without speculative decoding.
gemma-4-31B-it still apply.Derived from google/gemma-4-31B-it by Google DeepMind, licensed under the
Apache License 2.0 (see Gemma 4 license).
This derivative is distributed under the same license; a copy of the Apache 2.0
License is included as LICENSE. Per the license, note that these files have
been modified from the original (abliterated and NVFP4-quantized as described
above). Please also review Google's Gemma prohibited-use policy.
The model was produced with directional-ablation + llm-compressor NVFP4 PTQ.
The exact quantization recipe is in recipe.yaml. Abliteration calibration used
matched harmful/harmless instruction sets and selected the refusal direction by
the layer that most reduced refusals on a validation split.