GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
feralcrow
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
feralcrow
Model Tree
google/gemma-4-12B-it-qat-q4_0-unquantized
Input Modalities
Output Modalities
Supported Functionality
In our testing, Google's w4a16-ct lost precise long-context retrieval on vLLM, while
unsloth's GGUF of the same QAT model did not. Serving unsloth's codes on vLLM fixed it.
Long-context tool-calling test (a needle row in a ~194K-token Kubernetes pod listing: extract its exact name into tool arguments, answer from context, pick from a 30-tool menu). 30 trials per depth (3 cases × N=10), temperature 1.0 / top-p 0.95 / top-k 64, thinking off, vLLM v0.30.0, fp8 KV cache, RTX 5060 Ti 16 GB:
| 4-bit weights served on vLLM | ~100K tokens | ~194K tokens | greedy P(correct needle) @194K |
|---|---|---|---|
google/gemma-4-12B-it-qat-w4a16-ct | 23/30 | 16/30 | 0.43 (picks a neighbouring row) |
| Q4_0 round-to-nearest of Google's unquantized QAT weights (= Google's own Q4_0 GGUF codes) | 29/30 | 12/30 | 0.13 (picks a neighbouring row) |
| this repo (unsloth UD Q4_0 codes) | 30/30 | 29/30 | 0.92 |
| reference: llama.cpp, unsloth UD GGUF | 27/30 | 28/30 | 1.00 |
Short-context quality is unchanged (same setup, same sampling, thinking off):
| test | this repo | google/gemma-4-12B-it-qat-w4a16-ct |
|---|---|---|
| tool-calling conformance, 21 cases × N=10 (emission, schema, selection, state, termination, regression) | 210/210 | 210/210 |
| long-context tier at ~15K tokens | 30/30 | 30/30 |
BFCL v4 simple_python (400 entries, AST-checked) | 95.00% (380/400) | 94.25% (377/400) |
Decode speed is unchanged from Google's checkpoint: 95.0 tok/s (single stream, 1024-token
generations, MTP speculative decoding, same GPU) vs 94.4 for w4a16-ct.
What we found about the codes (64 rows × 3 tensors compared):
w4a16-ct uses llm-compressor's min-max scale (absmax / 7.5), a grid ~6.7% coarser
than the Q4_0 grid the model was QAT-trained for (signed_max / -8).Multimodal and concurrency (same setup): image and audio input both work through vLLM's
OpenAI-compatible API — an image with text and shapes was read correctly, and a short speech clip
was transcribed verbatim (vLLM logs a soundfile ImportError for audio unless installed with
vllm[audio], then falls back to another decoder). Concurrent decode scales well: 100.2 / 174.4 /
245.5 tok/s aggregate at 1 / 2 / 3 simultaneous streams (2.45×), with per-request speed down 13% at
3 streams and no preemptions.
Caveats — please read. This is one long-context test family, 30 trials per depth, one GPU, one vLLM version. We have not established why the imatrix codes hold long context better than QAT round-to-nearest, and we have not tested Google's GGUF on llama.cpp at 194K. Short-context behaviour matches Google's checkpoint — see the table below.
All numbers above were measured on this single setup:
| GPU | NVIDIA GeForce RTX 5060 Ti 16 GB (Blackwell, compute capability 12.0 / SM120), 180 W, PCIe gen 3 ×16 — one card, no tensor parallelism |
| Host | AMD Ryzen 7 5800X (8 cores), 62 GiB RAM, Ubuntu 24.04.4 LTS, kernel 6.8.0 |
| Driver / CUDA | NVIDIA 580.126.20, CUDA 13.0 |
| vLLM | 0.30.0 (vllm/vllm-openai:v0.30.0, torch 2.13.0+cu130) |
| vLLM kernels | MarlinLinearKernel for the w4a16 layers; attention forced to TRITON_ATTN (Gemma 4's mixed 256/512 head dims — FA3/FA4 are not available on SM120) |
| vLLM serving | , (5 GiB → 384,522 KV tokens), , , , prefix caching on, MTP drafter with ; ~8.9 GiB model memory |
The decode figures are single-stream (one request at a time), 1024-token generations, averaged over the prompts that hit the cap. Other GPUs, drivers, attention backends (e.g. FlashAttention on Hopper), KV-cache dtypes or vLLM versions were not tested and may behave differently.
bash
vllm serve feralcrow/gemma-4-12B-it-qat-w4a16-ud \ --speculative-config '{"method": "mtp", "model": "google/gemma-4-12B-it-qat-q4_0-unquantized-assistant", "num_speculative_tokens": 4}' \ --kv-cache-dtype fp8 --max-model-len 262144 \ --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4Sampling per the Gemma 4 model card: temperature 1.0, top-p 0.95, top-k 64.
build_from_gguf.py (included) reads each blk.N.{attn_q,attn_k,attn_v,attn_output,ffn_gate,ffn_up,ffn_down}
Q4_0 tensor from the GGUF, copies its 4-bit codes verbatim into compressed-tensors
pack-quantized int32 packing, and converts each block's fp16 scale to bf16 (≈0.18% relative
rounding; codes are 100% identical to the GGUF, verified). About 1/3 of the scales are negative,
which Q4_0 uses to land the block extreme on -8; vLLM's Marlin kernel handles this correctly.
Apache-2.0, as are all sources. Model weights © Google (Gemma 4); quantization codes from unsloth's GGUF. This is an independent derivative, not affiliated with or endorsed by Google or unsloth.
--kv-cache-dtype fp8_e4m3--kv-cache-memory-bytes 5368709120--max-model-len 262144--max-num-seqs 3--max-num-batched-tokens 4096google/gemma-4-12B-it-qat-q4_0-unquantized-assistantnum_speculative_tokens: 4| llama.cpp reference | build b11026, unsloth gemma-4-12B-it-qat-UD-Q4_K_XL.gguf, q8_0 KV cache, --parallel 3 --kv-unified --ctx-size 262144, speculative decoding ngram-mod + MTP draft (mtp-gemma-4-12B-it.gguf), same GPU |