simaai

Gemma-4-12B-it-GPTQ-Safetensors

README

License: apache-2.0

Quantization

Table with columns: Component, Method, Weight format, Details
Component	Method	Weight format	Details
328 decoder Linear layers	GPTQ	symmetric INT4, group size 256	Static activation ordering, solver block 128, damping 0.01
`lm_head`	GPTQ	symmetric INT4, group size 256	Untied from the original BF16 input embeddings before calibration
Image patch and embedding projections	RTN	symmetric INT8, per output channel	2 Linear layers
Audio embedding projection	RTN	symmetric INT8, per output channel	1 Linear layer
Input embeddings, norms, biases, layer scalars and position embeddings	None	BF16	Preserved from the source

This is weight-only quantization. Media projections remain BF16 while GPTQ calibrates the decoder and head, then receive INT8 RTN. Exact targets are in recipe.yaml. Packed weights and scales are stored in model.safetensors using the compressed-tensors format.

Calibration

512 training examples, seed 42, batch size 1, at most 2048 tokens per example. The examples are shuffled together; image/audio tokens are not truncated. Media samples include reference assistant answers or transcripts. Native masks preserve bidirectional image attention in sliding layers and causal global attention.

Table with columns: Dataset, Split, Samples, Pinned revision
Dataset	Split	Samples	Pinned revision
HuggingFaceH4/ultrachat_200k	train_sft	256 text	`8049631c405ae6576f93f445c6b8166f76f5505a`
lmms-lab/VQAv2	train	64 images	`32665d35052eb4a6d4414851c3c829a72754915a`
HuggingFaceM4/ChartQA	train	64 images	`b605b6e08b57faf4359aeb2fe6a3ca595f99b6c5`

FLEURS contributes 16 clips each in English, German, French, Spanish, Arabic, Hindi, Mandarin and Japanese. Audio is mono 16 kHz, up to 30 seconds. calibration_manifest.json records selection order, source rows, hashes and processed token counts. calibration_exclusions.json excludes evaluation images, waveforms and matching transcripts.

Evaluation

Matched Transformers GPU inference with decompressed BF16 weights, native SDPA, greedy decoding and thinking disabled. Evaluation date: 2026-10-11. These are small held-out suites, not full official vision/audio benchmarks or Modalix measurements.

Table with columns: Check, Original BF16, This checkpoint
Check	Original BF16	This checkpoint
Native vision cases	32/34	33/34
Higher-resolution OCR	6/6	5/6
Short-text final answers	15/20	12/20
Long-text final answers (up to 4K total context)	16/18	18/18
FLEURS subset pooled CER (400 clips, 8 languages)	8.91%	8.07%

The 18 long-text cases are length variants of four tasks. Lower pooled CER does not establish better audio quality than BF16: a BF16 Mandarin outlier affects the aggregate, while typical per-clip errors can increase after quantization. This checkpoint had no audio token-limit hits. Per-language results use the same 50 clips per language:

Table with columns: Language, Metric, Original BF16, This checkpoint
Language	Metric	Original BF16	This checkpoint
English	WER	4.79%	6.61%
German	WER	6.09%	5.62%
French	WER	7.63%	10.49%
Spanish	WER	2.98%	4.77%
Arabic

Full WikiText-2 (EleutherAI/wikitext_document_level, wikitext-2-raw-v1, test split; all 62 documents, no sample limit; 2048-token windows, batch size 1, BOS enabled, no chat template):

Table with columns: Checkpoint, Word perplexity
Checkpoint	Word perplexity
Original BF16	1253.04
This checkpoint	1645.46
Absolute degradation	+392.42
Relative degradation	+31.32%

Absolute perplexities are unusually high even for BF16. Use the matched results alongside task accuracy. Packed-weight, scale-layout, finite-scale, BF16-embedding and checkpoint-reload checks passed. Text, image and audio smoke inference completed; readable-text hallucinations remain in the image answer. Outputs are in smoke_results.json. Detailed methods, failure cases and baseline comparisons are in ACCURACY_REPORT.md and accuracy_summary.json.

Reproduction

The included quantize.py uses public llm-compressor and compressed-tensors with the default GPTQ/RTN backend. No private SiMa repository is required. requirements.txt pins the public upstream revisions closest to the original environment, plus the model and calibration dependencies.

Quantization requires an NVIDIA CUDA GPU. The original run used one A100 80 GB, 192 GiB host RAM and 8 CPUs; quantization and validation took about 28 minutes.

Run from this model directory, keeping the bundled calibration exclusions and smoke assets alongside the script:

bash

python3.12 -m venv quantize_envsource quantize_env/bin/activatepip install --upgrade pipBUILD_TYPE=nightly pip install -r requirements.txt python quantize.py \  --model-path /path/to/original-gemma-4-12B-it \  --output-dir /path/to/new-gemma-4-12B-it-GPTQ-Safetensors

Only --model-path and --output-dir are accepted. The script downloads the pinned training data and builds a writable calibration/ cache beside itself. The output directory must not exist. It saves packed weights, original tokenizer/processor files, recipe, versions, calibration provenance and text/image/audio smoke results.

BUILD_TYPE=nightly supplies version numbers that satisfy upstream dependency constraints when building the pinned public source revisions. It does not select the latest source code. The source model and calibration datasets require Hugging Face access.

quantize.py uses bundled smoke/sjc.jpg and smoke/why_is_the_sky_blue.wav instead of machine-specific paths. The quantization settings and calibration logic are unchanged; quantize_original.py preserves the script submitted for the original run. Public-package installation, decoder tracing, mixed GPTQ/RTN, packed saving, scale validation and reload/inference passed on a small Gemma 4 Unified model on CPU. The saved 12B weights and reported evaluations were produced with the original internal builds. The public recipe has not been rerun on the full 12B model, so bit-identical reproduction is not claimed.

Original checkpoint environment

These versions describe the original internal build; use requirements.txt for public reproduction.

text

torch: 2.11.0+cu128transformers: 5.16.1llmcompressor: 0.13.1.dev80+gfe6f69a5.d20260909compressed-tensors: 0.18.1.dev27+g099fa98.d20260909datasets: 5.0.1Python: 3.12.14CUDA: 12.8

Deployment and limitations

This directory is the pre-LLiMa Hugging Face artifact. LLiMa compilation and Modalix execution have not been validated for this checkpoint. Keep compiler output separate. Quality varies by language, modality, prompt and context length; Mandarin transcription and some arithmetic/OCR cases remain weak. The quantized checkpoint retains the source configuration; the evaluation context budgets do not define a compiled deployment's context limit.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

simaai

Model Tree

Base

google/gemma-4-12B-it

Quantized

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality