HYPR4AI

Gemma-4-12B-MXFP4A16-Text-Only

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Base model	`google/gemma-4-12B-it`
Quantization	MXFP4A16
Quantization tool	`llmcompressor`
Serving tested with	vLLM
Approx. model folder size	~7.4 GB

Text-Only Changes

This checkpoint disables likely image/audio special-token embeddings by zeroing their embedding rows. The tokenizer entries and architecture remain present so the model can still load with Gemma 4 tooling.

Serve with multimodal inputs blocked:

python
limit_mm_per_prompt={"image": 0, "audio": 0}

For vLLM, ensure the checkpoint includes:

text
processor_config.json

Quantization Recipe

Default scheme:

text
MXFP4A16

Targeted layers:

text
Linear

Ignored layers/patterns:

text
lm_head
re:.*embed.*
re:.*router.*
re:.*vision.*
re:.*audio.*
re:.*multimodal.*
re:.*image.*

Intended Use

This checkpoint is intended for:

text-only generation
context-grounded QA
small-footprint serving experiments
comparison against the FP8 checkpoint

It is not intended for image/audio input.

Usage With vLLM

bash
vllm serve /path/to/model \
  --served-model-name fp4-gemma \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --max-model-len 2048 \
  --gpu-memory-utilization 0.45 \
  --enforce-eager

HotpotQA Benchmark

Benchmark:

text
Dataset: HotpotQA Distractor
Slice: validation[0:459]
Model name: fp4-gemma
Successful requests: 459/500

System prompt:

text
Answer questions using only the provided context. Some questions require combining facts from multiple passages. Return only the shortest correct answer. When possible, copy the exact name, date, place, number, or phrase from the context. Do not explain your reasoning or add extra text.

Table with columns: Metric, Value
Metric	Value
Questions	459
Successful	459
Exact Correct	294/459
Exact Match	64.05%
Token F1	78.69%
Exact + Partial	401/459 = 87.36%
Avg Latency	0.54s
P50 Latency	0.48s
Max Latency	1.94s

Limitations

Experimental text-only quantized checkpoint.
Serve with multimodal inputs blocked.
Reported benchmark is HotpotQA Distractor validation[0:459].

License

This checkpoint is derived from google/gemma-4-12B-it. Follow the applicable Gemma 4 terms and Apache 2.0 license information from the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

HYPR4AI

Model Tree

Base

google/gemma-4-12B-it

Quantized

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Base model	`google/gemma-4-12B-it`
Quantization	MXFP4A16
Quantization tool	`llmcompressor`
Serving tested with	vLLM
Approx. model folder size	~7.4 GB

Text-Only Changes

Serve with multimodal inputs blocked:

python
limit_mm_per_prompt={"image": 0, "audio": 0}

For vLLM, ensure the checkpoint includes:

text
processor_config.json

Quantization Recipe

Default scheme:

text
MXFP4A16

Targeted layers:

text
Linear

Ignored layers/patterns:

text
lm_head
re:.*embed.*
re:.*router.*
re:.*vision.*
re:.*audio.*
re:.*multimodal.*
re:.*image.*

Intended Use

This checkpoint is intended for:

text-only generation
context-grounded QA
small-footprint serving experiments
comparison against the FP8 checkpoint

It is not intended for image/audio input.

Usage With vLLM

bash
vllm serve /path/to/model \
  --served-model-name fp4-gemma \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --max-model-len 2048 \
  --gpu-memory-utilization 0.45 \
  --enforce-eager

HotpotQA Benchmark

Benchmark:

text
Dataset: HotpotQA Distractor
Slice: validation[0:459]
Model name: fp4-gemma
Successful requests: 459/500

System prompt:

text
Answer questions using only the provided context. Some questions require combining facts from multiple passages. Return only the shortest correct answer. When possible, copy the exact name, date, place, number, or phrase from the context. Do not explain your reasoning or add extra text.

Table with columns: Metric, Value
Metric	Value
Questions	459
Successful	459
Exact Correct	294/459
Exact Match	64.05%
Token F1	78.69%
Exact + Partial	401/459 = 87.36%
Avg Latency	0.54s
P50 Latency	0.48s
Max Latency	1.94s

Limitations

Experimental text-only quantized checkpoint.
Serve with multimodal inputs blocked.
Reported benchmark is HotpotQA Distractor validation[0:459].

License

This checkpoint is derived from google/gemma-4-12B-it. Follow the applicable Gemma 4 terms and Apache 2.0 license information from the base model.