GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
salve-mundii
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
salve-mundii
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
This checkpoint stacks three independent compression steps. All of them are applied only to the
language model — the vision tower, audio tower, multimodal projectors, embeddings and lm_head
are left at full precision (see ignore list in config.json), because vLLM cannot serve a
quantized vision/audio tower.
| Step | Technique | Detail |
|---|---|---|
| 1. Pruning | SparseGPT, 10% unstructured sparsity | Targets MLP layers only: gate_proj, up_proj, down_proj of every language_model decoder layer. Applied before quantization as a light, Hessian‑aware weight clean‑up. |
| 2. Quantization | GPTQ, W4A16 | 4‑bit integer weights, group size 128, symmetric, static activation ordering. Activations stay BF16. Calibrated on 128 samples of ultrachat_200k (max seq len 256). |
| 3. KV cache | FP8 (E4M3) | KV cache stored in fp8_e4m3 at serve time (calculate_kv_scales: false). Cuts attention‑cache memory and bandwidth. |
Note on the pruning step. The 10% sparsity is unstructured. vLLM ≥0.19 has no sparse kernel, so the weights are served dense — the pruning does not itself accelerate inference. Its role here is regularization/weight clean‑up ahead of GPTQ. The energy and footprint wins below come from the 4‑bit weights + FP8 KV cache, not from the sparsity.
The exact, machine‑readable recipe is in recipe.yaml / recipe.json
and the quantization metadata in quantization.yaml.
Both the baseline and this model were evaluated on a single NVIDIA L4, vLLM 0.19.0, with the
identical serving profile (max_num_seqs=1024, FP8 E4M3 KV cache, prefix caching and
chunked‑prefill on, speculative decoding off) and the same client concurrency (256), using greedy
decoding (temperature=0). Energy was measured by NVML at 10 Hz and integrated over the run window
with the trapezoidal rule, cross‑checked with CodeCarbon. Each configuration was run three times;
values are mean ± 1 std. Because decoding is greedy, that spread is vLLM's system nondeterminism, not
sampling noise.
Energy is the total GPU energy for one complete benchmark run. ΔE and ΔAcc. are relative to BF16 and computed from unrounded means.
| Benchmark | BF16 energy (kJ) | W4A16+P10 energy (kJ) | ΔE (%) | BF16 acc. (%) | W4A16+P10 acc. (%) | ΔAcc. (pp) |
|---|---|---|---|---|---|---|
| MMMU | 92.7 ± 0.3 | 47.7 ± 0.5 | −48.6 ± 0.5 | 58.4 ± 0.7 | 55.5 ± 0.9 | −2.9 |
| MMMU‑Pro | 155.5 ± 0.2 | 75.5 ± 0.2 |
The energy roughly halves on both MMMU benchmarks while accuracy drops by at most 2.9 pp; on MedXpertQA accuracy is unchanged within the noise. Across the four suites the model uses 1.7–2.8× less GPU energy per run (2.07× on MMMU‑Pro, the primary optimisation benchmark).
Suite sizes: MMMU 900 (validation, all 30 subjects) · MMMU‑Pro 3,460 (standard 10‑option + vision) ·
MATH‑Vision 304 (testmini) · MedXpertQA 470 of 500 (30 samples exceed the 3‑image serving cap and
are excluded identically for both models). Generation caps: 1024 tokens, except MATH‑Vision at 2048.
| Metric | BF16 baseline | This model | Δ |
|---|---|---|---|
| Checkpoint size on disk | 14.92 GB | 9.33 GB | −37.5% |
| Peak VRAM during serving | 22.01 GB | 22.34 GB | ≈ equal¹ |
¹ Peak VRAM is essentially unchanged because vLLM reserves ~90% of the GPU regardless; the real memory win is the 37.5% smaller checkpoint and the FP8 KV cache freeing space for KV blocks.
Average board power is essentially the same for both models (≈67–72 W) — the GPU does not draw less power, it finishes sooner. The identical workload completes 1.6–2.8× faster, so proportionally fewer GPU‑seconds (and joules) are spent to produce the same answers.
The saving is therefore entirely a per‑token efficiency win: energy per token drops by 42–65% depending on the suite (−53% on MMMU‑Pro), which tracks the per‑run saving almost exactly. On a synthetic fixed‑length throughput benchmark that isolates this term, W4A16 GPTQ alone cuts energy per generated token by 42.7% (67.2 → 38.5 mJ/token, throughput 398 → 718 tok/s). Serving‑side tuning adds to it, mostly via the FP8 KV cache (≈−15%): it halves the bytes streamed from HBM per decode step, and since decoding is memory‑bandwidth‑bound, that shortens the run at unchanged power.
The trade is at most 2.9 accuracy points, which is mostly the cost of 4‑bit weights; the 10% MLP pruning and the FP8 KV cache add little degradation on top. All four suites stay well above the Resilient AI Challenge's 80%‑of‑baseline accuracy threshold.
The evaluator launches the model as:
bash
vllm serve <model-uri> --config vllm_config.yaml
The bundled vllm_config.yaml pins the tuned profile:
yaml
gpu-memory-utilization: 0.9max-model-len: 20000max-num-seqs: 1024max-num-batched-tokens: 4096enable-prefix-caching: trueenable-chunked-prefill: truequantization: compressed-tensorskv-cache-dtype: fp8_e4m3limit-mm-per-prompt:image: 3
Sampling settings used for all reported evaluations: greedy decoding (temperature=0,
top_p=1.0). The bundled generation_config.json keeps the base model's own defaults
(temperature=1.0, top_p=0.95, top_k=64) for general use.
ultrachat_200k (train_sft), 128 samples, max seq len 256| 38.2 ± 0.6 |
| 35.2 ± 0.5 |
| −2.9 |
| MATH‑Vision | 28.5 ± 0.7 | 10.4 ± 0.1 | −63.7 ± 1.0 | 37.1 ± 2.0 | 35.1 ± 1.8 | −1.9 |
| MedXpertQA | 17.3 ± 0.2 | 10.4 ± 0.1 | −40.0 ± 0.8 | 28.4 ± 1.2 | 28.8 ± 0.9 | +0.3 |