GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
xdavxd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
xdavxd
Model Tree
TrevorJS/gemma-4-12B-it-uncensored
Input Modalities
Output Modalities
Supported Functionality
This model is a quantized version of TrevorJS/gemma-4-12B-it-uncensored. It was evaluated on several tasks to assess its quality in comparison to the original model.
This model was obtained by quantizing the weights and activations of TrevorJS/gemma-4-12B-it-uncensored to FP4 data type using the NVFP4 format, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 65%.
Weights are quantized with FP4 (group_size=16, FP8-E4M3 block scales) using GPTQ calibration via LLM Compressor. Activations are quantized with FP4 using dynamic per-group scales computed at inference time. Every nn.Linear in the language decoder is quantized: self_attn.{q,k,v,o}_proj and mlp.{gate,up,down}_proj across all 48 layers. The vision and audio projections, lm_head, token embeddings, and norms are kept in BF16. Per-layer FP8 KV cache scales are calibrated and stored in the checkpoint; they take effect with --kv-cache-dtype fp8 and are ignored with auto.
v2 uses TrevorJS's abliteration rather than coder3101's based on results from Gemma4-12b-it-abliterlitics: it reaches a higher unlock rate (85.8% vs 81.0% ASR over 400 HarmBench behaviours) while doing less capability damage (-0.83pp vs -1.73pp mean across MMLU-Pro/GPQA/BBH), with 8.6x lower median KL divergence over harmless prompts (0.006 vs 0.051), meaning a typical non-refusal prompt behaves near-identically to google/gemma-4-12B-it.
v2 also switches from RTN to GPTQ calibration, which quantizes weights column by column and uses second-order information to push each column's rounding error into the columns not yet quantized. Calibration data is chat-templated rather than raw text, matching the token distribution the model sees at inference.
This model can be deployed using vLLM. For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 vLLM usage guide.
Note: Gemma 4 (gemma4_unified) requires transformers 5.10 or later; older versions fail at config parsing. Tested on vLLM 0.28.0 with transformers 5.16.1.
shell
vllm serve xdavxd/gemma-4-12B-it-heretic-v2-NVFP4 \--max-model-len 32768 \--gpu-memory-utilization 0.90
To enable thinking/reasoning and tool calling:
shell
vllm serve xdavxd/gemma-4-12B-it-heretic-v2-NVFP4 \--max-model-len 32768 \--gpu-memory-utilization 0.90 \--kv-cache-dtype fp8 \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--async-scheduling
KV cache: this checkpoint ships calibrated FP8 KV scales.
--kv-cache-dtype fp8uses them for 2× KV capacity at near-BF16 fidelity.--kv-cache-dtype autokeeps BF16 KV and ignores the scales.
Tip: For text-only workloads, pass
--limit-mm-per-prompt '{"image": 0, "audio": 0}'to skip multimodal memory allocation and free up GPU memory for a longer context window.
python
from openai import OpenAIopenai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8001/v1"client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)model = "xdavxd/gemma-4-12B-it-heretic-v2-NVFP4"messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]outputs = client.chat.completions.create(model=model,messages=messages,)generated_text = outputs.choices[0].message.contentprint(generated_text)
This model was created by applying NVFP4 quantization with LLM Compressor, as presented in the code snippet below.
python
import torchfrom datasets import concatenate_datasets, load_datasetfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierMODEL_ID = "TrevorJS/gemma-4-12B-it-uncensored"SAVE_DIR = "gemma-4-12B-it-heretic-v2-NVFP4"MAX_SEQUENCE_LENGTH = 2048NUM_TEXT, NUM_IMAGE, NUM_AUDIO = 256, 128, 128MAX_AUDIO_MS = 750 * 40 # audio_seq_length x audio_ms_per_token from processor_config.jsonmodel = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="auto")processor = AutoProcessor.from_pretrained(MODEL_ID)# The chat template only reads content-block *type* and emits a placeholder token;# it never reads the media payload. Media must be passed to the processor separately.def render(messages):return processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)def build_text(ex):messages = [{"role": m["role"], "content": [{"type": "text", "text": m["content"]}]} for m in ex["messages"]]return processor.apply_chat_template(messages, return_tensors="pt", padding=False, truncation=True, max_length=MAX_SEQUENCE_LENGTH,tokenize=True, add_special_tokens=False, return_dict=True, add_generation_prompt=False,)def build_image(ex):caption = ex["caption"][0] if isinstance(ex["caption"], list) else ex["caption"]messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this image."}]},{"role": "assistant", "content": [{"type": "text", "text": caption}]},]return processor(text=render(messages), images=ex["image"].convert("RGB"), return_tensors="pt",padding=False, truncation=True, max_length=MAX_SEQUENCE_LENGTH, add_special_tokens=False,)def build_audio(ex):audio = ex["audio"]messages = [{"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Transcribe this audio."}]},{"role": "assistant", "content": [{"type": "text", "text": ex["text"]}]},]return processor(text=render(messages), audio=audio["array"], sampling_rate=audio["sampling_rate"], return_tensors="pt",padding=False, truncation=True, max_length=MAX_SEQUENCE_LENGTH, add_special_tokens=False,)def data_collator(batch):assert len(batch) == 1out = {}for k, v in batch[0].items():if v is None: # concatenate_datasets unifies schemas; text rows carry pixel_values=None, etc.continuet = torch.as_tensor(v)out[k] = t.to(torch.bfloat16) if t.is_floating_point() else treturn outds_text = load_dataset("neuralmagic/calibration", name="LLM", split=f"train[:{NUM_TEXT}]")ds_text = ds_text.map(build_text, remove_columns=ds_text.column_names)ds_image = load_dataset("lmms-lab/flickr30k", split=f"test[:{NUM_IMAGE}]")ds_image = ds_image.map(build_image, remove_columns=ds_image.column_names)ds_audio = load_dataset("MLCommons/peoples_speech", "validation", split=f"validation[:{NUM_AUDIO * 4}]")ds_audio = ds_audio.filter(lambda x: x["duration_ms"] <= MAX_AUDIO_MS).select(range(NUM_AUDIO))ds_audio = ds_audio.map(build_audio, remove_columns=ds_audio.column_names)# Shuffle so GPTQ's sequential calibration interleaves the three distributions.ds = concatenate_datasets([ds_text, ds_image, ds_audio]).shuffle(seed=42)recipe = GPTQModifier(targets="Linear",scheme="NVFP4",ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],kv_cache_scheme={"num_bits": 8, "type": "float", "strategy": "tensor", "dynamic": False, "symmetric": True},)oneshot(model=model,recipe=recipe,dataset=ds,max_seq_length=MAX_SEQUENCE_LENGTH,num_calibration_samples=len(ds),data_collator=data_collator,sequential_targets=["Gemma4UnifiedTextDecoderLayer"],tracing_ignore=["Gemma4UnifiedVisionEmbedder", "Gemma4UnifiedMultimodalEmbedder"],)model.save_pretrained(SAVE_DIR, save_compressed=True)processor.save_pretrained(SAVE_DIR)
Run with llm-compressor 0.13.0, transformers 5.14.1, torch 2.13.0+cu130 on an NVIDIA GB10. Audio decoding requires soundfile, librosa, torchcodec, and a system ffmpeg.
This model is being evaluated on GSM8K Platinum, IFEval, and MATH-500 using lm-evaluation-harness and lighteval, served with vLLM (OpenAI-compatible API).
(as reported by TrevorJS)
| Metric | Base model | Original model (google/gemma-4-12B-it) |
|---|---|---|
| KL divergence | 0.0556 | 0 (by definition) |
| Refusals | 6/100 | 99/100 |
I ran these to make sure that fixing the audio pathway didn't mess up text understanding, it is intended as a sanity check not necessarily to be statistically relevant (which is why I ran each test 1 time not as an average of 3). I'm satisfied with the results.
Three columns: the original model, the abliterated base this checkpoint was quantized from, and this checkpoint. All three measured on the same hardware (NVIDIA GB10), same vLLM build, same server config, same seed. The difference between the first two is the cost of abliteration; between the last two is the cost of quantization. Recovery is this checkpoint divided by the abliterated base (quantization cost only). Protocol follows RedHatAI's: 0-shot, temperature 1.0, top-p 0.95, top-k 64, max_gen_toks=32000, seed 1234, 1 repetition.
With thinking
Without thinking
Perplexity, truthfulness, and reasoning stability
Same three models, same hardware. These use different protocols from the tables above: wikitext and TruthfulQA are loglikelihood (no generation); GSM8K here is a 300-problem subset at temperature 0 with thinking on and an 8192-token budget, split three ways to separate reasoning accuracy from termination failures.
* The single miss is one dropped word-final "s" (linger for lingers); all six sentences are otherwise exact. Under sampling the model produces the correct form roughly half the time. For comparison, RedHatAI/gemma-4-12B-it-NVFP4 — text-only calibration — scores 1/6 on the same clip.
On the GSM8K split: answered-only accuracy is unchanged by quantization. The headline gap at temperature 0 is entirely thinking-loop failures — responses that hit the token budget without terminating — which rose from 8% to 23%. Under the sampling protocol used in the tables above, this effect disappears: sampling breaks the model out of greedy loops. If you serve with greedy decoding and thinking enabled, expect roughly one in four hard problems to exhaust the budget. Reasoning accuracy on completed problems is intact.
Image and audio input are not covered by the benchmarks above
Image input works. Served under vLLM, the model correctly describes simple test images.
Audio transcription works. On a Harvard-sentences clip at temperature 0, this checkpoint transcribes 5/6 sentences exactly; the single miss is one dropped word-final "s" (linger for lingers), with every content word correct. The unquantized base scores 6/6.
The results were obtained using the following commands:
Single seed (1234) per benchmark. All three models served on NVIDIA GB10 with the same vLLM build and identical server flags apart from the model path and --kv-cache-dtype fp8 (this checkpoint only). Protocol follows RedHatAI's 26B/31B cards; deviations noted below.
vLLM server:
Note: this uses ghcr.io/timothystewart6/vllm-gb10:latest (v0.28.1.dev0+g2cf0a6915.d20260828, transformers 5.16.1). VLLM_USE_V2_MODEL_RUNNER=0 is required for MTP speculative decoding on this build; V2 fails during speculator CUDA graph capture (should be fixed upstream in upcoming 0.29.0). MTP is lossless under rejection sampling and only affects throughput.
shell
docker run --rm -it \--gpus all --ipc=host --network host \-v ~/models/gemma-4-12B-it-heretic-v2-NVFP4:/models/heretic:ro \-v ~/.cache/huggingface:/root/.cache/huggingface \-e VLLM_USE_V2_MODEL_RUNNER=0 \ghcr.io/timothystewart6/vllm-gb10:latest \vllm serve /models/heretic \--host 0.0.0.0 --port 8001 \--served-model-name heretic \--max-model-len 65536 \--gpu-memory-utilization 0.50 \--max-num-seqs 32 \--max-num-batched-tokens 8192 \--kv-cache-dtype fp8 \--language-model-only \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--async-scheduling \--default-chat-template-kwargs '{"enable_thinking": true}' \--speculative-config '{"method":"mtp","model":"google/gemma-4-12B-it-assistant","num_speculative_tokens":3}'
To reproduce the without-thinking results, remove
--default-chat-template-kwargs '{"enable_thinking": true}'.Deviations from RedHatAI's protocol:
--max-model-len 65536rather than 32768 — withmax_gen_toks=32000, a 32768 ceiling leaves 768 tokens for the prompt and MATH-500 has longer problems, which vLLM rejects with HTTP 400. Raising the ceiling changes nothing about generation; it only prevents the rejection.timeout=3600rather than 1200 — BF16 on GB10 needs ~31 minutes to exhaust a 32000-token budget.until=[]on all tasks — the default stop sequences truncate thinking traces mid-reasoning. Model's shipped chat template rather thanexamples/tool_chat_template_gemma4.jinja.max_retries=6.
shell
lm_eval --model local-chat-completions \--tasks gsm8k_platinum_cot_llama \--model_args "model=heretic,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/heretic_gsm8k_platinum \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
shell
lm_eval --model local-chat-completions \--tasks ifeval \--model_args "model=heretic,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/heretic_ifeval \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
minerva_math500, 0-shot)Requires pip install 'lm-eval[math]'. RedHatAI used lighteval for this benchmark; lm-eval's minerva_math500 uses the same 500-problem subset with sympy-based answer equivalence (math_verify). The exact_match filter reports 0 on thinking-mode output and is not the reported number.
shell
lm_eval --model local-chat-completions \--tasks minerva_math500 \--model_args "model=heretic,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 --apply_chat_template \--output_path results/heretic_math500 \--seed 1234 \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"
shell
lm_eval --model local-completions \--model_args "model=heretic,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \--tasks wikitext --num_fewshot 0 --batch_size 1lm_eval --model local-completions \--model_args "model=heretic,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \--tasks truthfulqa_mc2 --num_fewshot 0 --apply_chat_template
shell
lm_eval --model local-chat-completions \--tasks gsm8k \--model_args "model=heretic,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=16,max_retries=3,timeout=3600" \--num_fewshot 0 --limit 300 --apply_chat_template \--gen_kwargs "max_gen_toks=8192,until=[]" \--output_path results/heretic_gsm8k_300 --log_samples
Rescored from the samples file: a response is empty if it contains no content (thinking never terminated); headline is correct / total; answered-only is correct / (total − empty). Extraction takes the last number in the response after stripping markdown and thousands separators, since lm-eval's flexible-extract filter returns [invalid] on this model's bolded answer formatting.
Custom script: single needle (a random vault code) buried in WikiText-2 filler at depths 0/0.25/0.5/0.75/1.0 for context lengths 4k/8k/16k/32k/64k, thinking off, temperature 0, exact-match on the code. 25 requests per model.