axteam

gemma-4-31B-it-NVFP4

README

License: apache-2.0

Quantization details

Scheme — W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations, stored as compressed-tensors nvfp4-pack-quantized.
Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms. These are left unquantized to preserve quality (a small fraction of total parameters).
Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining, so it stays faithful to the original model's behaviour.

RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Used only as an internal baseline here, not published.
AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (256-sample calibration, no gradient).

Evaluation (measured)

Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ attention.

Table with columns: Metric, This model, BF16 original
Metric	This model	BF16 original
Perplexity (lower is better)	1.2326	1.1120 (+10.8%)
Top-1 next-token agreement	92.4%	99.4%

These numbers measure fidelity to the base model, not absolute task scores.

Comparison with other public quantizations

Same eval set and harness for every row — closer to the BF16 original is better.

Table with columns: Model, Perplexity, Top-1
Model	Perplexity	Top-1
BF16 original (upper bound)	1.1120	99.4%
This model (`axteam`)	1.2326	92.4%
RedHatAI/gemma-4-31B-it-NVFP4	1.2753	91.4%

On our harness this model is on par with or ahead of the public same-format quantizations above. It also ships Google's corrected July-2026 chat_template; some earlier community builds were based on the older template.

Usage (vLLM)

bash

vllm serve axteam/gemma-4-31B-it-NVFP4

python

from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-31B-it-NVFP4")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat(    [{"role": "user", "content": "Write a short joke about saving RAM."}],    params,)print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. This is a Blackwell-only format — it needs FP4 tensor cores (B200, RTX PRO) for full speed.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

bash

vllm serve axteam/gemma-4-31B-it-NVFP4 --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.

한국어 요약

google/gemma-4-31B-it 를 NVFP4 (W4A4, FP4) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.

방식 — 가중치·활성화 모두 FP4(W4A4). Blackwell(B200 등) FP4 텐서코어에서 제 속도. 활성화도 FP4라 품질 손실이 가장 큼 — 품질 우선이면 NVFP4A16 권장. 형식은 compressed-tensors nvfp4-pack-quantized.
BF16 유지 — 비전 타워·비전 투영·토큰 임베딩(lm_head와 묶임)·MoE 라우터·노름 (전체의 극히 일부).
토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로). config.json 은 원본 + quantization_config.
양자화 방식(PTQ vs QAT) — 학습 후(PTQ) 방식이라 재학습 없이 원본 가중치를 저비트로 바꿉니다. RTN=보정 없이 반올림만(가장 빠르나 이상치 보상이 안 돼 4비트 품질 최저, 내부 기준용·비공개) · AWQ=소량 보정으로 큰 활성화와 곱해지는 중요 채널을 채널 스케일로 보호한 뒤 반올림(경사 없음, 4비트서 RTN보다 훨씬 좋음) · QAT=양자화를 흉내 내며 재학습해 가중치가 오차에 적응(저비트 최상이나 학습 스택·데이터·GPU 필요, 구글 Gemma QAT 체크포인트는 별도 — 본 레포는 QAT 아님). 저비트 품질 ≈ QAT ≥ AWQ > RTN. 본 빌드: AWQ — 보정 기반 PTQ(256문항 보정, 경사 없음).
품질(실측) — BF16 원본 대비 perplexity 1.2326(원본 1.1120, +10.8%), top-1 일치도 92.4%(원본 99.4%). 내부 영/한 평가셋에서 측정. 변형: AWQ attention.
동형 비교(같은 하네스) — 우리 1.2326/92.4% vs RedHatAI 1.2753/91.4% (BF16 원본 1.1120/99.4%). 같은 포맷 공개본과 동급~상회. 또한 구글이 2026-07에 고친 chat_template 사용(구버전 기반 커뮤니티 빌드와 차별).
사용 — vllm serve axteam/gemma-4-31B-it-NVFP4. 권장 샘플링 .

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

axteam

Model Tree

Base

google/gemma-4-31B-it

Quantized

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quantization details

Scheme — W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations, stored as compressed-tensors nvfp4-pack-quantized.
Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms. These are left unquantized to preserve quality (a small fraction of total parameters).
Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining, so it stays faithful to the original model's behaviour.

RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Used only as an internal baseline here, not published.
AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (256-sample calibration, no gradient).

Evaluation (measured)

Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ attention.

Table with columns: Metric, This model, BF16 original
Metric	This model	BF16 original
Perplexity (lower is better)	1.2326	1.1120 (+10.8%)
Top-1 next-token agreement	92.4%	99.4%

These numbers measure fidelity to the base model, not absolute task scores.

Comparison with other public quantizations

Same eval set and harness for every row — closer to the BF16 original is better.

Table with columns: Model, Perplexity, Top-1
Model	Perplexity	Top-1
BF16 original (upper bound)	1.1120	99.4%
This model (`axteam`)	1.2326	92.4%
RedHatAI/gemma-4-31B-it-NVFP4	1.2753	91.4%

Usage (vLLM)

bash

vllm serve axteam/gemma-4-31B-it-NVFP4

python

from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-31B-it-NVFP4")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat(    [{"role": "user", "content": "Write a short joke about saving RAM."}],    params,)print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. This is a Blackwell-only format — it needs FP4 tensor cores (B200, RTX PRO) for full speed.

Reducing KV-cache memory (optional, serve-time)

bash

vllm serve axteam/gemma-4-31B-it-NVFP4 --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.

한국어 요약

google/gemma-4-31B-it 를 NVFP4 (W4A4, FP4) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.

방식 — 가중치·활성화 모두 FP4(W4A4). Blackwell(B200 등) FP4 텐서코어에서 제 속도. 활성화도 FP4라 품질 손실이 가장 큼 — 품질 우선이면 NVFP4A16 권장. 형식은 compressed-tensors nvfp4-pack-quantized.
BF16 유지 — 비전 타워·비전 투영·토큰 임베딩(lm_head와 묶임)·MoE 라우터·노름 (전체의 극히 일부).
토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로). config.json 은 원본 + quantization_config.
양자화 방식(PTQ vs QAT) — 학습 후(PTQ) 방식이라 재학습 없이 원본 가중치를 저비트로 바꿉니다. RTN=보정 없이 반올림만(가장 빠르나 이상치 보상이 안 돼 4비트 품질 최저, 내부 기준용·비공개) · AWQ=소량 보정으로 큰 활성화와 곱해지는 중요 채널을 채널 스케일로 보호한 뒤 반올림(경사 없음, 4비트서 RTN보다 훨씬 좋음) · QAT=양자화를 흉내 내며 재학습해 가중치가 오차에 적응(저비트 최상이나 학습 스택·데이터·GPU 필요, 구글 Gemma QAT 체크포인트는 별도 — 본 레포는 QAT 아님). 저비트 품질 ≈ QAT ≥ AWQ > RTN. 본 빌드: AWQ — 보정 기반 PTQ(256문항 보정, 경사 없음).
품질(실측) — BF16 원본 대비 perplexity 1.2326(원본 1.1120, +10.8%), top-1 일치도 92.4%(원본 99.4%). 내부 영/한 평가셋에서 측정. 변형: AWQ attention.
동형 비교(같은 하네스) — 우리 1.2326/92.4% vs RedHatAI 1.2753/91.4% (BF16 원본 1.1120/99.4%). 같은 포맷 공개본과 동급~상회. 또한 구글이 2026-07에 고친 chat_template 사용(구버전 기반 커뮤니티 빌드와 차별).
사용 — vllm serve axteam/gemma-4-31B-it-NVFP4. 권장 샘플링 .