GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
xdavxd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
xdavxd
Model Tree
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
Input Modalities
Output Modalities
Supported Functionality
f1e06dc)This model is a refusal-ablated version of google/gemma-4-26B-A4B-it-qat-q4_0-unquantized. It was evaluated on several tasks to assess its quality in comparison to the original model.
This model was obtained by applying TrevorJS's norm-preserving biprojected abliteration plus Expert-Granular Abliteration (EGA) to the QAT (quantization-aware-trained, Q4_0) checkpoint of Gemma 4 26B A4B, using heretic for model loading, residual extraction and evaluation.
Per-layer refusal directions are computed from winsorized (99.5th percentile) residual activations on 400 harmful + 400 harmless prompts, orthogonalized against the harmless mean, and projected out of attn.o_proj and the shared mlp.down_proj in all 30 layers with row-norm preservation (60 dense tensors). The same projection is then applied to each of the 128 experts' down_proj slices per layer (3,840 expert slices). Scale 1.0, expert scale 1.0. Attention q/k/v, gate/up projections, router, embeddings, vision tower and output head are untouched. Tensor manifest is identical to the base (1,013 tensors, same names).
This is the "o_proj + down_proj" method family (capability-preserving per the E4B abliteration report), extended into the routed experts. Weights are BF16; this is not a quantized model.
This model can be deployed using vLLM. For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 vLLM usage guide.
markdown
vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega \--max-model-len 32768 \--gpu-memory-utilization 0.90
To enable thinking/reasoning and tool calling:
markdown
vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega \--max-model-len 32768 \--gpu-memory-utilization 0.90 \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--default-chat-template-kwargs '{"enable_thinking": true}' \--limit-mm-per-prompt '{"image": 4}' \--async-scheduling \--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'
MTP: use the QAT-specific drafter
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant, not the vanilla one. On reasoning workloads it accepts ~78% of drafts vs ~53% for the vanilla drafter, a 2.2× vs 1.6× speedup. Lossless in distribution.Tip: For text-only workloads, pass
--language-model-onlyto skip vision encoder memory allocation and free up GPU memory for a longer context window.
python
from openai import OpenAIopenai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8001/v1"client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)model = "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega"messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]outputs = client.chat.completions.create(model=model,messages=messages,)generated_text = outputs.choices[0].message.contentprint(generated_text)
This model was created with TrevorJS's gemma-4-abliteration scripts on top of heretic git HEAD (1.4 plugin API, via a small compatibility shim), as presented in the code snippet below.
Stack: transformers 5.16.1, torch 2.13.0+cu130, heretic git HEAD, DGX Spark GB10 (128 GB).
shell
pip install 'heretic-llm @ git+https://github.com/p-e-w/heretic' sentencepiece protobufgit clone https://github.com/TrevorS/gemma-4-abliteration && cd gemma-4-abliterationhf download google/gemma-4-26B-A4B-it-qat-q4_0-unquantized --local-dir ~/models/base/gemma-4-26B-qatHF_DATASETS_CACHE=/tmp/hf_datasets_cache python scripts/ega.py \--model ~/models/base/gemma-4-26B-qat \--strip-topic-markers --skip-prefix --batch-size 4 \--save ~/models/abl/gemma-4-26B-qat-ega# heretic's save drops processor files; restore tokenizer/processor from basecp -L ~/models/base/gemma-4-26B-qat/{tokenizer*,*processor*.json,chat_template*,generation_config.json} ~/models/abl/gemma-4-26B-qat-ega/
Settings passed to heretic: offload_outputs_to_cpu=False, device_map={"": 0}, orthogonalize_direction=True, winsorization_quantile=0.995. Refusal markers: heretic defaults minus the seven topic words (violat, prohibit, illegal, harmful, inappropriate, unethical, ethical boundaries) and minus disclaimer (added to heretic in June 2026; Gemma 4 prepends "Disclaimer:" to compliant answers, which inflates keyword refusal counts ~25×).
This model was evaluated on IFEval, GSM8K Platinum, MATH-500, GPQA Diamond, WikiText-2, TruthfulQA-MC2, a 300-problem GSM8K termination split, and a needle-in-haystack long-context test using lm-evaluation-harness, served with vLLM (OpenAI-compatible API).
| Metric | This model | Original model (google/gemma-4-26B-A4B-it-qat-q4_0-unquantized) |
|---|---|---|
| KL divergence (first-token, 100 harmless_alpaca) | 0.0789 | 0 (by definition) |
| Refusals (mlabonne/harmful_behaviors, 100 prompts, keyword) | 4/100 | 100/100 |
| Refusals (686-prompt cross-dataset audit, keyword) | 11/686 | - |
| Refusals (686-prompt audit, manually audited) | ~1–3/686 | - |
Cross-dataset audit: JailbreakBench 3/100, tulu-harmbench 3/320, NousResearch/RefusalDataset 1/166, mlabonne 4/100. All 11 keyword flags are long responses that answer the prompt; most trip on an "I am an AI, not a doctor/attorney" preamble or on a marker string occurring inside generated content. For reference, TrevorJS/gemma-4-26B-A4B-it-uncensored (same method, vanilla base) scores KL 0.090 and 3/686 on the identical harness.
Three columns: the original model, Google's QAT release, and this checkpoint. All measured on the same hardware (NVIDIA GB10), same vLLM build, same server config, same seed, with the QAT-specific MTP drafter on every server. The difference between the first two is the cost of QAT; between the last two is the cost of abliteration. Recovery is this model divided by the QAT base (abliteration cost only). Protocol follows RedHatAI's: 0-shot, temperature 1.0, top-p 0.95, top-k 64, max_gen_toks=32000, seed 1234, 1 repetition.
The vanilla and QAT columns are complete for round 1 (IFEval, wikitext, TruthfulQA, GSM8K split) and show the two bases are equivalent on every chat-templated task; they differ only on raw-text perplexity, where vanilla scores 24% worse for reasons not yet explained. Round 2 (Platinum, MATH-500, GPQA, no-think) was run against QAT as the reference base and not repeated on vanilla.
I ran these to confirm that abliterating the QAT base didn't cost capability. It's a sanity check with single seeds, not a statistically rigorous comparison. The result: abliteration costs about 5 points on TruthfulQA and 2–4 points on MATH-500, and is within noise everywhere else. Termination and long-context retrieval are unaffected. The NVFP4 quant of this checkpoint is reported on its own model card.
One 26B-specific observation: MATH-500 scores higher without thinking on all three models (QAT 86.8 vs 83.8, EGA 83.2 vs 81.8). The MoE's reasoning traces cost it a few points on this benchmark.
With thinking
Without thinking
Perplexity, truthfulness, and reasoning stability
The results were obtained using the following commands:
Single seed (1234) per benchmark. All three models served on NVIDIA GB10 with the same vLLM build and identical server flags apart from the model path.
vLLM server:
ghcr.io/timothystewart6/vllm-gb10:latest (v0.28.1.dev0+g2cf0a6915.d20260828, transformers 5.16.1). The QAT-specific MTP drafter is lossless under rejection sampling and only affects throughput.
shell
docker run --rm -it \--gpus all --ipc=host --network host \-v ~/models/abl/gemma-4-26B-qat-ega:/models/qat-ega:ro \-v ~/.cache/huggingface:/root/.cache/huggingface \-e VLLM_USE_V2_MODEL_RUNNER=1 \ghcr.io/timothystewart6/vllm-gb10:latest \vllm serve /models/qat-ega \--host 0.0.0.0 --port 8001 \--served-model-name qat-ega \--max-model-len 65536 \--gpu-memory-utilization 0.75 \--max-num-seqs 32 \--max-num-batched-tokens 8192 \--language-model-only \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--async-scheduling \--default-chat-template-kwargs '{"enable_thinking": true}' \--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'
To reproduce the without-thinking results, remove
--default-chat-template-kwargs '{"enable_thinking": true}'.Deviations from RedHatAI's protocol:
--max-model-len 65536rather than 32768 — withmax_gen_toks=32000, a 32768 ceiling leaves 768 tokens for the prompt and MATH-500 has longer problems, which vLLM rejects with HTTP 400.timeout=3600rather than 1200 — BF16 on GB10 needs ~31 minutes to exhaust a 32000-token budget.until=[]on all tasks — the default stop sequences truncate thinking traces mid-reasoning. Model's shipped chat template rather thanexamples/tool_chat_template_gemma4.jinja.max_retries=6.
shell
lm_eval --model local-chat-completions \--tasks gsm8k_platinum_cot_llama \--model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/qat-ega_gsm8k_platinum \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
shell
lm_eval --model local-chat-completions \--tasks ifeval \--model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/qat-ega_ifeval \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
minerva_math500, 0-shot)Requires pip install 'lm-eval[math]' and antlr4-python3-runtime==4.11. Same 500-problem subset as lighteval's math_500, scored with sympy-based answer equivalence (math_verify). The exact_match filter reports 0 on thinking-mode output and is not the reported number.
shell
lm_eval --model local-chat-completions \--tasks minerva_math500 \--model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/qat-ega_math500 \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
gpqa_diamond_cot_zeroshot, 0-shot)Dataset is gated; requires an HF token that has accepted the terms for Idavidrein/gpqa. flexible-extract is the reported number.
shell
lm_eval --model local-chat-completions \--tasks gpqa_diamond_cot_zeroshot \--model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/qat-ega_gpqa \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
shell
lm_eval --model local-completions \--model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \--tasks wikitext --num_fewshot 0 --batch_size 1lm_eval --model local-completions \--model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \--tasks truthfulqa_mc2 --num_fewshot 0 --apply_chat_template
shell
lm_eval --model local-chat-completions \--tasks gsm8k \--model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=16,max_retries=3,timeout=3600" \--num_fewshot 0 --limit 300 --apply_chat_template \--gen_kwargs "max_gen_toks=8192,until=[]" \--output_path results/qat-ega_gsm8k_300 --log_samples
Rescored from the samples file: a response is empty if it contains no content (thinking never terminated); headline is correct / total; answered-only is correct / (total − empty). Extraction takes the last number in the response after stripping markdown and thousands separators, since lm-eval's flexible-extract filter returns [invalid] on this model's bolded answer formatting.
Custom script: single needle (a random vault code) buried in WikiText-2 filler at depths 0/0.25/0.5/0.75/1.0 for context lengths 4k/8k/16k/32k/64k, thinking off, temperature 0, exact-match on the code. 25 requests per model.