GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
xdavxd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
xdavxd
Model Tree
xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega
Input Modalities
Output Modalities
Supported Functionality
k_scale/v_scale per layer)This model is an experts-only NVFP4 quantization of xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega, which is a refusal-ablated version of Google's QAT checkpoint of Gemma 4 26B A4B. It was evaluated on several tasks to assess its quality in comparison to the unquantized base.
The 128 routed experts' gate_proj, up_proj, and down_proj in every MoE layer are quantized to NVFP4 (FP4 weights with group_size=16 and FP8-E4M3 block scales; FP4 activations with dynamic per-group scales) using GPTQ calibration via LLM Compressor. Attention (q/k/v/o_proj), the shared dense MLPs, the router, embeddings, and lm_head stay in BF16. Per-layer FP8 KV cache scales are calibrated and stored in the checkpoint; they take effect with --kv-cache-dtype fp8 and are ignored with auto.
Experts-only quantization keeps the layers every token passes through at full precision and quantizes only the sparse, redundant expert FFNs where 8 of 128 fire per token. The result is a 3× smaller checkpoint with no measurable perplexity, truthfulness, or reasoning-accuracy cost against the BF16 base (see Evaluation). The QAT base's Q4_0 grid alignment exists only in attention, which is not quantized here.
The abliteration underneath is TrevorJS's norm-preserving biprojection plus Expert-Granular Abliteration on the QAT checkpoint; see the base model card for the method and its own evaluation.
This model can be deployed using vLLM. For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 vLLM usage guide.
markdown
vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega-NVFP4 \--max-model-len 32768 \--gpu-memory-utilization 0.90 \--kv-cache-dtype fp8
To enable thinking/reasoning and tool calling:
markdown
vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega-NVFP4 \--max-model-len 32768 \--gpu-memory-utilization 0.90 \--kv-cache-dtype fp8 \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--default-chat-template-kwargs '{"enable_thinking": true}' \--limit-mm-per-prompt '{"image": 4}' \--async-scheduling \--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'
KV cache: this checkpoint ships calibrated FP8 KV scales.
--kv-cache-dtype fp8uses them for 2× KV capacity;autokeeps BF16 KV and ignores the scales.MTP: use the QAT-specific drafter
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant, not the vanilla one. On reasoning workloads it accepts ~78% of drafts vs ~53% for the vanilla drafter, a 2.2× vs 1.6× speedup.Tip: For text-only workloads, pass
--language-model-onlyto skip vision encoder memory allocation and free up GPU memory for a longer context window.
python
from openai import OpenAIopenai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8001/v1"client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)model = "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega-NVFP4"messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]outputs = client.chat.completions.create(model=model,messages=messages,)generated_text = outputs.choices[0].message.contentprint(generated_text)
This model was created by applying experts-only NVFP4 quantization with LLM Compressor to the abliterated base, as presented in the code snippet below.
Stack: llm-compressor git main (0.13.1.dev), transformers 5.16.1, torch 2.13.0+cu130, DGX Spark GB10 (128 GB). Calibration: text and image samples, chat-templated; moe_calibrate_all_experts=True so every expert gets a real Hessian rather than only the tokens routed to it.
python
import os, sysfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom llmcompressor import oneshotfrom llmcompressor.modeling.moe.linearize import load_quantizable_moefrom llmcompressor.modifiers.gptq import GPTQModifierMODEL_ID = os.path.expanduser(sys.argv[1])SAVE_DIR = os.path.expanduser(sys.argv[2])with load_quantizable_moe(AutoModelForImageTextToText):model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="auto", device_map={"": 0})processor = AutoProcessor.from_pretrained(MODEL_ID)n = sum(1 for k, _ in model.named_modules() if "layers.0.experts." in k and k.endswith("down_proj"))print(f"linearized experts in layer 0: {n}", flush=True)assert n == 128, "experts were not linearized - aborting"recipe = GPTQModifier(targets=["re:.*experts.*"],scheme="NVFP4",ignore=["lm_head", "re:.*embed.*", "re:.*vision.*","re:.*self_attn\\..*_proj", "re:.*\\.mlp\\..*", "re:.*router.*"],kv_cache_scheme={"num_bits": 8, "type": "float", "strategy": "tensor", "dynamic": False, "symmetric": True},)for c in (model.config, getattr(model.config, "text_config", None)):if c is not None:c.allow_global_per_layer_attribute_access = Trueoneshot(model=model,processor=processor,recipe=recipe,dataset="ultrachat_200k",splits={"calibration": "train_sft[:512]"},num_calibration_samples=512,max_seq_length=2048,moe_calibrate_all_experts=True,sequential_targets=["Gemma4TextDecoderLayer"],tracing_ignore=["Gemma4VisionModel", "Gemma4MultimodalEmbedder"],)model.save_pretrained(SAVE_DIR, save_compressed=True)processor.save_pretrained(SAVE_DIR)print("saved", SAVE_DIR, flush=True)
After save_pretrained, tokenizer and processor files are copied from the base checkpoint — llm-compressor bakes the calibration truncation length (2048) into tokenizer.json otherwise.
Gemma 4 global-attention layers (5, 11, 17, 23, 29) have no v_proj; those v_proj names are added to the config.json ignore list so vLLM's fused-QKV loader resolves them.
This model was evaluated on IFEval, GSM8K Platinum, MATH-500, GPQA Diamond, WikiText-2, TruthfulQA-MC2, a 300-problem GSM8K termination split, a needle-in-haystack long-context test, and llama-benchy throughput using lm-evaluation-harness, served with vLLM (OpenAI-compatible API).
KL divergence and refusal counts are properties of the abliteration and were measured on the BF16 base; quantization was verified lossless separately (see Accuracy).
| Metric | Base model (BF16) | Original model (google/gemma-4-26B-A4B-it-qat-q4_0-unquantized) |
|---|---|---|
| KL divergence (first-token, 100 harmless_alpaca) | 0.0789 | 0 (by definition) |
| Refusals (mlabonne/harmful_behaviors, 100 prompts, keyword) | 4/100 | 100/100 |
| Refusals (686-prompt cross-dataset audit, keyword) | 11/686 | - |
| Refusals (686-prompt audit, manually audited) | ~1–3/686 | - |
Cross-dataset audit: JailbreakBench 3/100, tulu-harmbench 3/320, NousResearch/RefusalDataset 1/166, mlabonne 4/100. All 11 keyword flags are long responses that answer the prompt; most trip on an "I am an AI, not a doctor/attorney" preamble or on a marker string occurring inside generated content. For reference, TrevorJS/gemma-4-26B-A4B-it-uncensored (same method, vanilla base) scores KL 0.090 and 3/686 on the identical harness.
I ran these to confirm that quantizing experts-only didn't cost capability. It's a sanity check with single seeds, not a statistically rigorous comparison.
Four columns: the original model, Google's QAT release, the abliterated QAT base this checkpoint was quantized from, and this checkpoint. All measured on the same hardware (NVIDIA GB10), same vLLM build, same server config, same seed, with the QAT-specific MTP drafter on every server. The difference between the first two is the cost of QAT; between the second and third is the cost of abliteration; between the last two is the cost of quantization. Recovery is this checkpoint divided by the QAT base (the combined cost of abliteration and quantization). Quantization alone is practically lossless. Protocol follows RedHatAI's: 0-shot, temperature 1.0, top-p 0.95, top-k 64, max_gen_toks=32000, seed 1234, but only 1 repetition.
The vanilla and QAT columns are complete for round 1 (IFEval, wikitext, TruthfulQA, GSM8K split) and show the two bases are equivalent on every chat-templated task; they differ only on raw-text perplexity, where vanilla scores 24% worse for reasons not yet explained. Round 2 (Platinum, MATH-500, GPQA, no-think) was run against QAT as the reference base and not repeated on vanilla.
With thinking
Without thinking
Perplexity, truthfulness, and reasoning stability
On the GSM8K split: answered-only accuracy is unchanged across all four models. The headline gap at temperature 0 is entirely thinking-loop failures — responses that hit the token budget without terminating. Abliteration moved the loop rate from 5.7% to 6.0%; quantization moved it from 6.0% to 9.7%. Under the sampling protocol used in the tables above, this effect disappears. If you serve with greedy decoding and thinking enabled, expect roughly one in ten hard problems to exhaust the budget. Reasoning accuracy on completed problems is intact.
The results were obtained using the following commands:
Single seed (1234) per benchmark. All four models served on NVIDIA GB10 with the same vLLM build and identical server flags apart from the model path and --kv-cache-dtype fp8 (this checkpoint only).
vLLM server:
ghcr.io/timothystewart6/vllm-gb10:latest (v0.28.1.dev0+g2cf0a6915.d20260828, transformers 5.16.1). The QAT-specific MTP drafter is lossless under rejection sampling and only affects throughput.
shell
docker run --rm -it \--gpus all --ipc=host --network host \-v ~/models/quant/gemma-4-26B-qat-heretic-ega-NVFP4:/models/quant:ro \-v ~/.cache/huggingface:/root/.cache/huggingface \-e VLLM_USE_V2_MODEL_RUNNER=1 \ghcr.io/timothystewart6/vllm-gb10:latest \vllm serve /models/quant \--host 0.0.0.0 --port 8001 \--served-model-name quant \--max-model-len 65536 \--gpu-memory-utilization 0.50 \--kv-cache-dtype fp8 \--max-num-seqs 32 \--max-num-batched-tokens 8192 \--language-model-only \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--async-scheduling \--default-chat-template-kwargs '{"enable_thinking": true}' \--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'
To reproduce the without-thinking results, remove
--default-chat-template-kwargs '{"enable_thinking": true}'.Deviations from RedHatAI's protocol:
--max-model-len 65536rather than 32768 — withmax_gen_toks=32000, a 32768 ceiling leaves 768 tokens for the prompt and MATH-500 has longer problems, which vLLM rejects with HTTP 400.timeout=3600rather than 1200 — the BF16 reference models on GB10 need ~31 minutes to exhaust a 32000-token budget.until=[]on all tasks — the default stop sequences truncate thinking traces mid-reasoning. Model's shipped chat template rather thanexamples/tool_chat_template_gemma4.jinja.max_retries=6.
shell
lm_eval --model local-chat-completions \--tasks gsm8k_platinum_cot_llama \--model_args "model=quant,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/quant_gsm8k_platinum \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
shell
lm_eval --model local-chat-completions \--tasks ifeval \--model_args "model=quant,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/quant_ifeval \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
minerva_math500, 0-shot)Requires pip install 'lm-eval[math]' and antlr4-python3-runtime==4.11. Same 500-problem subset as lighteval's math_500, scored with sympy-based answer equivalence (math_verify). The exact_match filter reports 0 on thinking-mode output and is not the reported number.
shell
lm_eval --model local-chat-completions \--tasks minerva_math500 \--model_args "model=quant,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/quant_math500 \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
gpqa_diamond_cot_zeroshot, 0-shot)Dataset is gated; requires an HF token that has accepted the terms for Idavidrein/gpqa. flexible-extract is the reported number.
shell
lm_eval --model local-chat-completions \--tasks gpqa_diamond_cot_zeroshot \--model_args "model=quant,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/quant_gpqa \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
shell
lm_eval --model local-completions \--model_args "model=quant,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \--tasks wikitext --num_fewshot 0 --batch_size 1lm_eval --model local-completions \--model_args "model=quant,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \--tasks truthfulqa_mc2 --num_fewshot 0 --apply_chat_template
shell
lm_eval --model local-chat-completions \--tasks gsm8k \--model_args "model=quant,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=16,max_retries=3,timeout=3600" \--num_fewshot 0 --limit 300 --apply_chat_template \--gen_kwargs "max_gen_toks=8192,until=[]" \--output_path results/quant_gsm8k_300 --log_samples
Rescored from the samples file: a response is empty if it contains no content (thinking never terminated); headline is correct / total; answered-only is correct / (total − empty). Extraction takes the last number in the response after stripping markdown and thousands separators, since lm-eval's flexible-extract filter returns [invalid] on this model's bolded answer formatting.
Custom script: single needle (a random vault code) buried in WikiText-2 filler at depths 0/0.25/0.5/0.75/1.0 for context lengths 4k/8k/16k/32k/64k, thinking off, temperature 0, exact-match on the code. 25 requests per model.