Quantization details
- Method: round-to-nearest (RTN) with llm-compressor, no calibration data, so no language bias from a calibration set.
- Scheme:
W8A16, symmetric, per-channel INT8 weights, 16-bit activations.
- Quantized: all
Linear layers of the text decoder (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), 258 layers in total.
- Kept in bf16:
- Per-Layer Embeddings (
embed_tokens_per_layer, per_layer_input_gate, per_layer_projection, per_layer_model_projection)
- Vision tower and audio tower
- Multimodal projections (
embed_vision, embed_audio)
- Token embeddings and
lm_head
A large part of the E4B checkpoint is Per-Layer Embeddings, which stay in bf16, so the size reduction is smaller than the "8-bit" label suggests.
The full recipe is in recipe.yaml.
Evaluation
Perplexity and top-1 next-token agreement against the bf16 original on Wikipedia
(wikimedia/wikipedia, dump 20231101), 16,384 tokens per language, window 1024 tokens.
Table with columns: Language, PPL bf16, PPL W8A16, Δ PPL, Top-1 agreement| Language | PPL bf16 | PPL W8A16 | Δ PPL | Top-1 agreement |
|---|
Russian (ru) | 20.27 | 20.26 | -0.07% | 97.8% |
Kyrgyz (ky) | 24.44 | 24.41 | -0.12% | 97.9% |
English (en) | 48.91 | 48.87 | -0.08% | 97.6% |
Perplexity differences are within noise for this sample size: no measurable degradation on any language,
including low-resource Kyrgyz. Top-1 agreement of ~98% means the remaining disagreements are mostly near-ties
in the original model's distribution.
Only the text path was evaluated. Vision and audio encoder weights are identical to the original.
Usage
vLLM
vllm serve Bekten/gemma-4-E4B-it-W8A16 --max-model-len 32768
No --quantization flag is needed: vLLM reads the compressed-tensors config from config.json.
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create( model="Bekten/gemma-4-E4B-it-W8A16", messages=[{"role": "user", "content": "Кыргызстандын борбору кайсы шаар?"}],)print(resp.choices[0].message.content)
import torchfrom transformers import AutoModelForImageTextToText, AutoProcessor model_id = "Bekten/gemma-4-E4B-it-W8A16"processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto") messages = [{"role": "user", "content": [{"type": "text", "text": "Привет! Расскажи о себе."}]}]inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=256)print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Transformers decompresses the weights to bf16 on load, so memory savings apply to vLLM, not to Transformers.
On small GPUs, load with device_map="cpu", run one forward pass to trigger decompression, then move the model to GPU.
License
Same as the base model: apache-2.0. Use of this model is subject to the terms of
google/gemma-4-E4B-it.