GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
xdavxd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
xdavxd
Model Tree
coder3101/gemma-4-E4B-it-heretic
Input Modalities
Output Modalities
Supported Functionality
This model is a quantized version of coder3101/gemma-4-E4B-it-heretic. It was evaluated on several tasks to assess its quality in comparison to the original model.
This model was obtained by quantizing the weights and activations of coder3101/gemma-4-E4B-it-heretic to FP4 data type using the NVFP4 format, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 35%.
Weights are quantized with FP4 (group_size=16) using local per-group scaling. Only the weights of the linear operators within transformer blocks are quantized using LLM Compressor. Vision, audio, embedding, and output head layers are kept in their original precision.
This model can be deployed using vLLM. For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 vLLM usage guide.
markdown
vllm serve xdavxd/gemma-4-E4B-it-heretic-NVFP4A16 \--max-model-len 32768 \--gpu-memory-utilization 0.50
To enable thinking/reasoning and tool calling:
markdown
vllm serve xdavxd/gemma-4-E4B-it-heretic-NVFP4A16 \--max-model-len 32768 \--gpu-memory-utilization 0.50 \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--chat-template examples/tool_chat_template_gemma4.jinja \--limit-mm-per-prompt '{"image": 4, "audio": 1}' \--async-scheduling
Tip: For text-only workloads, pass
--limit-mm-per-prompt '{"image": 0, "audio": 0}'to skip vision encoder memory allocation and free up GPU memory for a longer context window.
python
from openai import OpenAIopenai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8001/v1"client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)model = "xdavxd/gemma-4-E4B-it-heretic-NVFP4A16"messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]outputs = client.chat.completions.create(model=model,messages=messages,)generated_text = outputs.choices[0].message.contentprint(generated_text)
This model was created by applying NVFP4 quantization with LLM Compressor, as presented in the code snippet below.
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierMODEL_ID = "./gemma-4-E4B-it-heretic"SAVE_DIR = "gemma-4-E4B-it-heretic-NVFP4A16"model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="auto")tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)recipe = QuantizationModifier(targets="Linear",scheme="NVFP4A16",ignore=["lm_head", "re:.*embed.*", "re:.*vision_tower.*", "re:.*audio_tower.*", "re:.*per_layer.*"],)oneshot(model=model,tokenizer=tokenizer,recipe=recipe,)model.save_pretrained(SAVE_DIR, save_compressed=True)tokenizer.save_pretrained(SAVE_DIR)
This model was evaluated on GSM8K Platinum, IFEval, and MATH-500, using lm-evaluation-harness and lighteval, served with vLLM (OpenAI-compatible API). All evaluations were performed with thinking enabled.
(as reported by coder3101)
| Metric | This model | Original model (google/gemma-4-E4B-it) |
|---|---|---|
| KL divergence | 0.0058 | 0 (by definition) |
| Refusals | 3/100 | 99/100 |
Also, see the Gemma 4 E4B: 23 Abliteration Variants Compared report by DreamFast. coder3101 was rated as "Lightest touch" and noted it "Indistinguishable from base."
The results were obtained using the following commands:
Each benchmark was run 3 times with different judgmental seeds (1234, 2345, 3456) and the scores were averaged; AIME 2025 used 8 seeds.
Ran using a docker image, you will need to tweak these for your own environment. Kept parameters similar to the RedHatAI runs.
vLLM server (all benchmarks):
markdown
sudo docker run --rm -it \--gpus all \--ipc=host \--network host \-v ~/quant-workspace/gemma-4-E4B-it-heretic-NVFP4A16:/models/heretic:ro \-v ~/.cache/huggingface:/root/.cache/huggingface \-e VLLM_USE_V2_MODEL_RUNNER=1 \ghcr.io/timothystewart6/vllm-gb10:latest \vllm serve /models/heretic \--host 0.0.0.0 --port 8001 \--served-model-name heretic \--max-model-len 69632 \--gpu-memory-utilization 0.50 \--enable-auto-tool-choice \--reasoning-parser gemma4 \--tool-call-parser gemma4 \--limit-mm-per-prompt '{"image":0,"audio":0}' \--kv-cache-dtype fp8 \--max-num-seqs 32 \--async-scheduling
markdown
for SEED in 1234 2345 3456; dolm_eval --model local-chat-completions \--tasks gsm8k_platinum_cot_llama \--model_args "model=heretic,max_length=36096,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \--num_fewshot 0 \--apply_chat_template \--output_path "results_gsm8k_seed${SEED}.json" \--seed $SEED \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=${SEED}"echo "Seed $SEED complete"done
markdown
for SEED in 1234 2345 3456; dolm_eval --model local-chat-completions \--tasks ifeval \--model_args "model=heretic,max_length=36096,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \--num_fewshot 0 \--apply_chat_template \--output_path "results_ifeval_seed${SEED}.json" \--seed $SEED \--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=${SEED}"echo "IFEval seed $SEED complete"done
litellm_config.yaml:
yaml
model_parameters:provider: hosted_vllmmodel_name: hosted_vllm/hereticbase_url: http://0.0.0.0:8001/v1api_key: ''timeout: 3600concurrent_requests: 32generation_parameters:temperature: 1.0max_new_tokens: 65536top_p: 0.95top_k: 64seed: 1234
markdown
# MATH-500 (3 seeds)for SEED in 1234 2345 3456; dosed -i "s/seed: .*/seed: $SEED/" litellm_config.yamllighteval endpoint litellm litellm_config.yaml 'math_500|0' \--output-dir "results_math500_seed${SEED}/" --save-detailsecho "MATH-500 seed $SEED complete"done