GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
QUASAR-QAT
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
QUASAR-QAT
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
In our vLLM 0.28 test on one H100, group-64 dispatches to the Machete kernel while Google's group-32 goes to Marlin: 4,225 vs 3,111 output tokens/s at the same batch (BF16: 4,823). Single-request latency is at parity.
Higher point estimates on all 8 downstream benchmarks than Google's QAT model.
| Benchmark | QUASAR | Google QAT |
|---|---|---|
| TriviaQA | 23.8 | 20.1 |
| NQ-open | 4.7 | 2.4 |
| TruthfulQA | 58.2 | 55.4 |
| ARC-Easy | 79.8 | 78.3 |
| ARC-Challenge | 56.2 | 56.0 |
| GSM8K | 82.0 | 81.5 |
| IFEval | 82.6 | 81.9 |
| MATH-hard | 60.7 | 59.4 |
| Average | 56.0 | 54.4 |
Same harness and backend for both checkpoints; full configs, BF16 results and paired-bootstrap CIs are in EVAL.md.
Native QAT checkpoint: the INT4 codes come directly from training, rather than a post-training re-quantization. export_receipt.json verifies the export per tensor. The GGUF twin carries the same codes and scales and ships a --verify script that checks the two files against each other.
vllm serve QUASAR-QAT/gemma-4-E4B-it-QUASAR-W4A16-G64 --dtype float16 (tested on vLLM 0.28.0 and 0.29.0; --dtype float16 keeps the stored fp16 scales exact), then any OpenAI-compatible client. Text, image and audio inputs; 131,072-token context.AutoModelForCausalLM.from_pretrained("QUASAR-QAT/gemma-4-E4B-it-QUASAR-W4A16-G64", dtype=torch.bfloat16) with compressed-tensors installed.q/k/v/o/gate/up/down) int4 symmetric, group_size 64, fp16 scales, compressed-tensors pack-quantized; embeddings, norms, per-layer-input tables and the vision/audio towers untouched (bf16). 10.2 GB. Loads wherever Google's W4A16 release loads.QUASAR is loss-aware quantization-aware training. It improves the low-bit reconstruction during training, then exports to standard deployment formats with zero inference overhead.
This checkpoint was healed against the BF16 teacher for 626M tokens: one epoch over ~398k prompts self-distilled from BF16 Gemma 4 E4B-it with thinking on, fp32 master weights, INT4 group-64.
Based on Gemma 4 by Google DeepMind. Upstream LICENSE/NOTICE apply.
bibtex
@article{counathe2026quasar,title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},journal={arXiv preprint arXiv:2608.13966},year={2026}}