GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sachin6624
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sachin6624
Model Tree
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
Input Modalities
Output Modalities
Supported Functionality
The base checkpoint was produced with quantization-aware training (QAT) targeting Q4_0: symmetric int4 with one scale per group of 32 weights. The model's weights are already adapted to that grid, so quantizing with the exact same grid using plain round-to-nearest recovers the QAT quality. No calibration data, GPTQ or AWQ is needed. Using a different scheme (for example the common W4A16 preset with group size 128) would throw away the benefit of QAT.
embed_vision, MoE router, lm_head / embeddings.bash
vllm serve sachin6624/gemma-4-26B-A4B-it-qat-w4a16-ct --max-model-len 8192python
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create( model="sachin6624/gemma-4-26B-A4B-it-qat-w4a16-ct", messages=[{"role": "user", "content": "Explain mixture-of-experts in two sentences."}], temperature=1.0, top_p=0.95,)print(resp.choices[0].message.content)python
from vllm import LLM, SamplingParams llm = LLM(model="sachin6624/gemma-4-26B-A4B-it-qat-w4a16-ct", max_model_len=8192)params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=512)out = llm.chat([{"role": "user", "content": "Write a haiku about quantization."}], params)print(out[0].outputs[0].text)Recommended sampling (from Google's Gemma 4 defaults): temperature=1.0, top_p=0.95, top_k=64.
Tested with vLLM 0.31.0 on a single NVIDIA L40S (48 GB):
max_model_len=8192
and the default gpu_memory_utilization.is_palindrome implementation with tests, and 17 * 23 = 391.Note: these are smoke tests, not a benchmark. No accuracy evaluation (MMLU, GSM8K, etc.) against the bf16 base model has been run yet.
Made with llm-compressor 0.14.0 (transformers 5.x) on an A100-80GB:
python
from compressed_tensors.quantization import QuantizationArgs, QuantizationSchemefrom transformers import AutoModelForImageTextToText, AutoProcessor from llmcompressor import oneshotfrom llmcompressor.modeling.moe.linearize import load_quantizable_moefrom llmcompressor.modifiers.quantization import QuantizationModifier MODEL_ID = "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized" processor = AutoProcessor.from_pretrained(MODEL_ID)with load_quantizable_moe(AutoModelForImageTextToText): # unfuse experts so they get quantized model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="auto") q4_0 = QuantizationArgs( num_bits=4, type="int", symmetric=True, strategy="group", group_size=32, observer="memoryless_minmax",)recipe = QuantizationModifier( config_groups={"group_0": QuantizationScheme(targets=["Linear"], weights=q4_0)}, ignore=["lm_head", "re:.*vision_tower.*", "re:.*embed_vision.*", "re:.*router.*"],)oneshot(model=model, recipe=recipe) model.save_pretrained("gemma-4-26B-A4B-it-qat-w4a16-ct", save_compressed=True)processor.save_pretrained("gemma-4-26B-A4B-it-qat-w4a16-ct")Released under the same terms as the base model: Apache 2.0. All credit for the model and the QAT checkpoint goes to Google DeepMind.
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create( model="sachin6624/gemma-4-26B-A4B-it-qat-w4a16-ct", messages=[{"role": "user", "content": "Explain mixture-of-experts in two sentences."}], temperature=1.0, top_p=0.95,)print(resp.choices[0].message.content)from vllm import LLM, SamplingParams llm = LLM(model="sachin6624/gemma-4-26B-A4B-it-qat-w4a16-ct", max_model_len=8192)params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=512)out = llm.chat([{"role": "user", "content": "Write a haiku about quantization."}], params)print(out[0].outputs[0].text)from compressed_tensors.quantization import QuantizationArgs, QuantizationSchemefrom transformers import AutoModelForImageTextToText, AutoProcessor from llmcompressor import oneshotfrom llmcompressor.modeling.moe.linearize import load_quantizable_moefrom llmcompressor.modifiers.quantization import QuantizationModifier MODEL_ID = "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized" processor = AutoProcessor.from_pretrained(MODEL_ID)with load_quantizable_moe(AutoModelForImageTextToText): # unfuse experts so they get quantized model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="auto") q4_0 = QuantizationArgs( num_bits=4, type="int", symmetric=True, strategy="group", group_size=32, observer="memoryless_minmax",)recipe = QuantizationModifier( config_groups={"group_0": QuantizationScheme(targets=["Linear"], weights=q4_0)}, ignore=["lm_head", "re:.*vision_tower.*", "re:.*embed_vision.*", "re:.*router.*"],)oneshot(model=model, recipe=recipe) model.save_pretrained("gemma-4-26B-A4B-it-qat-w4a16-ct", save_compressed=True)processor.save_pretrained("gemma-4-26B-A4B-it-qat-w4a16-ct")