Base model and provenance
The immediate parent is Mitchins/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic, revision d62fa836581401beafd08fdd624927e31dcac6ee. That parent is an uncensored/abliterated Heretic derivative of the QAT-trained google/gemma-4-26B-A4B-it-qat-q4_0-unquantized lineage, itself based on Google Gemma 4 26B-A4B-IT. The local export was frozen before conversion, including source-file SHA256 hashes, tokenizer, processor, chat template, generation configuration, and architecture configuration. See provenance/calibration-and-source.json.
The parent model's Apache-2.0 license and Gemma 4 license link apply. This quantization does not change the parent model's intended use, safety guidance, or license terms.
What is quantized
The saved quantization_config is the source of truth:
Linear language-model and MoE expert weights: 4-bit signed integer weights, W4A16, group-wise G64, symmetric, static activation ordering, memoryless_minmax observer, pack-quantized storage.- Dense language-model linears and MoE experts are W4A16. The artifact audit found the expected packed expert tensors and finite weight-scale tensors.
- The vision tower is retained BF16.
- Routers and token/image embeddings are retained BF16/unquantized. This Gemma export has no separate
lm_head tensor; output weights are tied through the BF16 token embedding.
- No reduced-precision KV-cache profile is claimed or recommended here.
The GPTQ modifier also used a 128-token block size and dampening_frac=0.01; the complete public-safe recipe is in provenance/calibration-and-source.json.
Calibration
Calibration was multimodal rather than text-only so the conversion observed both normal language traffic and the image-processing/template path.
- Text: 512 frozen 512-token windows, deterministic seed
20260812.
- 192 windows from
Salesforce/wikitext, wikitext-103-raw-v1 train.
- 192 from
HuggingFaceTB/smoltalk, openhermes-100k.
- 128 from
HuggingFaceTB/smoltalk, apigen-80k.
- Each source is deterministically shuffled; a valid 512-token span is sampled from each selected row. Conversational rows are rendered as role-labelled text before tokenization.
- Images: 1,024 deterministically selected, SHA256-verified images from COCO 2017 validation. The source processor converts each to RGB and applies its native preprocessing.
- Mixed layout: the first 512 calibration samples pair one frozen text window with one image; the remaining 512 use image-instruction prompts. Every selected COCO image and every frozen text window is used once.
The exact datasets, sample counts, seed, selection method, image hash-manifest digest, exclusions, and software versions are in provenance/calibration-and-source.json. Calibration media are deliberately not included in this release.
Quality: held-out perplexity
Table with columns: Model, Perplexity, Delta vs. BF16| Model | Perplexity | Delta vs. BF16 |
|---|
| Tuned QAT/Heretic BF16 parent | 388.150 | — |
| W4A16-G64 | 432.318 | +44.168 / +11.38% |
The result uses held-out Salesforce/wikitext wikitext-2-raw-v1 test text, which was not used for calibration. Both rows use the same frozen tokenizer, four contiguous non-empty 512-token test-corpus windows (2,044 scored next-token positions), next-token NLL arithmetic, and vLLM prompt_logprobs=1 scorer. BF16 is executed with vLLM CPU offload because the parent checkpoint exceeds a 24 GB GPU. See eval/evaluate_gemma4_ppl_vllm.py and benchmarks/perplexity.json.
This is a compact quantization sanity evaluation, not a replacement for task-specific evaluation of the tuned parent.
Verified functional gates
The final artifact was fresh-reloaded and passed:
- deterministic short text generation;
- real COCO image generation through the Gemma 4 processor/template path;
- Marlin W4A16 dense-linear selection;
- Marlin W4A16 MoE selection;
- CUDA graph capture, without eager mode;
- finite weight-scale and packed-tensor audit.
For images, use the Gemma 4 processor or OpenAI-compatible multimodal request format. The source chat template emits Gemma's actual <|image|> placeholder semantics; do not manually write a raw <image> token into a text prompt.
Gemma 4 supports video in its broader architecture, but video was neither required nor validated for this image-focused release profile. Unrestricted renderer warmup needlessly exercises a video path, so the validated image-serving profile explicitly uses:
--limit-mm-per-prompt '{"image":1,"video":0}'
This is a runtime warmup/configuration consideration, not a text or image generation failure.
Peak observed VRAM in the full-checkpoint performance test was 23,914 MiB; the benchmark conditions are below.
Reference setup: physical CUDA:1 RTX 3090, vLLM main commit 7ca49fbe4bab019e55d57cdc4b7fd3d55c67c1a6, CUDA graphs enabled, max_model_len=1024, max_num_seqs=32, max_num_batched_tokens=8192, prefix caching disabled, three repetitions per point, and fixed 128-token greedy ignore_eos outputs. Values are aggregate verified output tokens/s, not per-request rate.
Table with columns: Active requests, Text aggregate output tok/s, One image/request aggregate output tok/s| Active requests | Text aggregate output tok/s | One image/request aggregate output tok/s |
|---|
| 1 | 156.6 | 127.9 |
| 4 | 486.9 | 262.6 |
| 8 | 820.4 | 429.9 |
| 16 | 1,269.7 | 518.5 |
| 32 | 1,948.3 | 461.6 |
The text test is decode-dominant. The one-image/request test includes image processing and multimodal prefill, so it is not directly comparable to the text-only decode number. B32 maximized measured text aggregate throughput. B16 is the measured image-workload knee; B32 was slower for that workload because it exceeds the efficient 8,192-token multimodal batching envelope. vLLM profiling allocated encoder cache for up to 29 maximum-size images in this configuration. Raw public-safe measurements: benchmarks/rtx3090-performance.json.
Long-context validation
Gemma 4 advertises an architectural maximum of 262,144 tokens. That is not the same as what this 24 GB reference deployment can serve.
The validated RTX 3090 operational maximum is 131,072 total tokens (128K). A near-limit request with 130,816 input tokens plus 64 generated tokens completed with CUDA graphs, Marlin W4A16 dense/MoE paths, no OOM, and no allocator warning.
- Validated fixed KV reservation:
6444425401 bytes.
- Observed cache capacity:
135083 tokens.
- Margin above the configured 128K window:
4011 tokens (about 3%).
- A conventional 132K setting does not fit this fixed cache; 128K is the recommended validated ceiling for this reference configuration.
max_model_len includes generation: input_tokens + max_tokens <= 131072. For example, with a 2,048-token generation budget, the maximum input is 129,024 tokens. The exact fixed reservation is hardware- and runtime-specific; do not assume every 24 GB GPU has the same headroom.
Recommended serving configurations
A. Mixed agentic/search and batch service
This is the validated 128K RTX 3090 reference profile:
CUDA_VISIBLE_DEVICES=1 \
vllm serve useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision \
--quantization compressed-tensors \
--dtype bfloat16 \
--max-model-len 131072 \
--kv-cache-memory-bytes 6444425401 \
--max-num-seqs 17 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--enable-prefix-caching \
--scheduling-policy priority \
--limit-mm-per-prompt '{"image":1,"video":0}' \
--host 0.0.0.0 \
--port 8000
Do not combine this exact fixed-KV profile with --gpu-memory-utilization: explicit cache bytes are authoritative. The byte value is a validated RTX 3090 reference setting, not a universal default.
Admission policy matters as much as priority:
- Set
max_num_seqs=17.
- Cap bulk ingress at 16 active requests, normally with
priority: 100.
- Submit interactive/Hermes/search traffic with
priority: -10.
- The free seventeenth slot lets the interactive request enter. Once admitted, priority scheduling can preempt/requeue lower-priority cache users if the long request requires cache. Priority alone should not be expected to create an admission slot when every sequence slot is occupied.
Example OpenAI-compatible high-priority request, reserving a 2K response budget:
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision",
"messages": [{"role": "user", "content": "Summarize the supplied context."}],
"max_tokens": 2048,
"temperature": 0.2,
"priority": -10
}'
B. Text-throughput batch profile
For independent text-only jobs, use up to 32 active requests. The measured B32 point reached 1,948.3 aggregate verified output tok/s under the benchmark conditions above. This is a throughput profile, not the long-context mixed-service reservation.
C. Image-heavy batch profile
For one image per request, B16 is the measured knee (518.5 aggregate output tok/s). Retain the 8,192-token batching envelope and disable video unless it is explicitly required:
--max-num-seqs 16 --max-num-batched-tokens 8192 \
--limit-mm-per-prompt '{"image":1,"video":0}'
Runtime compatibility
Known-good final validation environment:
Table with columns: Component, Version| Component | Version |
|---|
| vLLM | main commit 7ca49fbe4bab019e55d57cdc4b7fd3d55c67c1a6 (0.1.dev1+g7ca49fbe4) |
| Transformers | 5.15.1 |
| PyTorch | 2.13.0+cu130 |
| CUDA runtime | 13.0 |
| compressed-tensors runtime | 0.17.0 |
| Conversion artifact metadata | compressed-tensors 0.18.0 |
vLLM 0.27.1 did not contain the necessary handling for Transformers-5.15-era heterogeneous Gemma 4 attention metadata. The artifact was not the cause. Use the pinned commit above or a later vLLM release that explicitly includes the corresponding Gemma 4 heterogeneous-config support. Transformers-native inference is not claimed as validated for this packed artifact; vLLM is the supported runtime documented here.
Image usage
Use an OpenAI-compatible multimodal request and let vLLM/the model processor apply Gemma's chat template:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://example.org/image.jpg"}},
{"type": "text", "text": "Describe this image accurately and concisely."},
],
}],
max_tokens=256,
)
print(response.choices[0].message.content)
Do not hand-insert raw image-placeholder text; the Gemma 4 processor/template is responsible for the image placement.
Limitations and caveats
- Performance and KV-cache figures are measurements from one RTX 3090 24 GB, not universal guarantees.
- The 128K claim means a validated 131,072-token total window, including output; it does not make 262K feasible on this reference card.
- Video was intentionally disabled and not validated for this image-focused profile.
- Image throughput includes processor and prefill work; do not compare it directly with text decode throughput.
- The long-context reference profile retained the quality-tested BF16/auto KV-cache behavior. Reduced-precision KV cache has not been validated here.
- This is a quantization of the linked QAT/Heretic parent, not a stock-Google checkpoint conversion.
Reproducibility
Public-safe provenance, calibration, artifact-audit, benchmark, and perplexity files are included in this repository:
Citation and license
Please cite the QAT/Heretic parent model card, the original Gemma 4 model card, and technical report for their respective methods and architecture. Google DeepMind developed Gemma; useful-quants packaged this W4A16 artifact from the QAT/Heretic parent checkpoint.
This repository inherits the parent Apache-2.0 license; see LICENSE and the Gemma 4 license.