HYPR4AI

Gemma-4-12B-it-FP8

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Base model	`google/gemma-4-12B-it`
Quantization	FP8 dynamic
Quantization tool	`llmcompressor`
Serving tested with	vLLM
Approx. model folder size	~13 GB
Original BF16 size	~23.9 GB

Quantization Recipe

Targeted layers:

text
Linear

Ignored layers/patterns:

text
lm_head
re:.*embed.*
re:.*router
re:.*vision_tower.*
re:.*audio_tower.*

This was done to reduce the risk of corrupting embeddings, output projection, router behavior, or native multimodal projection paths.

Intended Use

This checkpoint is intended for:

text generation
question answering
retrieval/context-grounded QA
vLLM serving experiments
FP8-vs-FP4 quantization comparison

Although the base architecture is multimodal, the reported benchmark here is text-only.

Usage With vLLM

bash
vllm serve /path/to/model \
  --served-model-name fp8-gemma \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.70 \
  --enforce-eager

For text-only deployments, block multimodal inputs at serving time:

python
limit_mm_per_prompt={"image": 0, "audio": 0}

HotpotQA Benchmark

Benchmark:

text
Dataset: HotpotQA Distractor
Slice: validation[0:500]
Model name: fp8-gemma
Successful requests: 500/500

System prompt:

text
Answer questions using only the provided context. Some questions require combining facts from multiple passages. Return only the shortest correct answer. When possible, copy the exact name, date, place, number, or phrase from the context. Do not explain your reasoning or add extra text.

Table with columns: Metric, Value
Metric	Value
Questions	500
Successful	500
Failures	0
Exact Correct	326/500
Exact Match	65.20%
Token F1	80.38%
Exact + Partial	442/500 = 88.40%
Avg Latency	0.53s
P50 Latency	0.48s

Limitations

Quantized checkpoint; behavior may differ from the original BF16 model.
Reported benchmark is HotpotQA Distractor validation[0:500].

License

This checkpoint is derived from google/gemma-4-12B-it. Follow the applicable Gemma 4 terms and Apache 2.0 license information from the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

HYPR4AI

Model Tree

Base

google/gemma-4-12B-it

Quantized

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Base model	`google/gemma-4-12B-it`
Quantization	FP8 dynamic
Quantization tool	`llmcompressor`
Serving tested with	vLLM
Approx. model folder size	~13 GB
Original BF16 size	~23.9 GB

Quantization Recipe

Targeted layers:

text
Linear

Ignored layers/patterns:

text
lm_head
re:.*embed.*
re:.*router
re:.*vision_tower.*
re:.*audio_tower.*

This was done to reduce the risk of corrupting embeddings, output projection, router behavior, or native multimodal projection paths.

Intended Use

This checkpoint is intended for:

text generation
question answering
retrieval/context-grounded QA
vLLM serving experiments
FP8-vs-FP4 quantization comparison

Although the base architecture is multimodal, the reported benchmark here is text-only.

Usage With vLLM

bash
vllm serve /path/to/model \
  --served-model-name fp8-gemma \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.70 \
  --enforce-eager

For text-only deployments, block multimodal inputs at serving time:

python
limit_mm_per_prompt={"image": 0, "audio": 0}

HotpotQA Benchmark

Benchmark:

text
Dataset: HotpotQA Distractor
Slice: validation[0:500]
Model name: fp8-gemma
Successful requests: 500/500

System prompt:

text
Answer questions using only the provided context. Some questions require combining facts from multiple passages. Return only the shortest correct answer. When possible, copy the exact name, date, place, number, or phrase from the context. Do not explain your reasoning or add extra text.

Table with columns: Metric, Value
Metric	Value
Questions	500
Successful	500
Failures	0
Exact Correct	326/500
Exact Match	65.20%
Token F1	80.38%
Exact + Partial	442/500 = 88.40%
Avg Latency	0.53s
P50 Latency	0.48s

Limitations

Quantized checkpoint; behavior may differ from the original BF16 model.
Reported benchmark is HotpotQA Distractor validation[0:500].

License

This checkpoint is derived from google/gemma-4-12B-it. Follow the applicable Gemma 4 terms and Apache 2.0 license information from the base model.