GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
aleksandard
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
aleksandard
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
Tested on RTX 5090 (Blackwell, sm120), CUDA 13.
Gemma 4 12B "unified" support landed in vllm-project/vllm#44429 and is not yet in a stable release — you need a vLLM nightly build. On Blackwell, the FlashInfer sampler fails to JIT-compile, so disable it.
Install nightly (CUDA 13; use cu129 URLs on CUDA 12.9 hosts):
bash
uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu130 \ --extra-index-url https://download.pytorch.org/whl/cu130 \ --index-strategy unsafe-best-matchServe:
bash
export VLLM_USE_FLASHINFER_SAMPLER=0vllm serve <path-to-this-model> \ --served-model-name gemma4-12b \ --max-model-len 8192 \ --gpu-memory-utilization 0.90 \ --host 0.0.0.0 --port 8000The model loads in ~11 GB, leaving plenty of room on a 32 GB card for KV cache
(raise --max-model-len accordingly). Recommended sampling for Gemma 4:
temperature=1.0, top_p=0.95, top_k=64.
Quick test:
bash
curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4-12b", "messages": [{"role": "user", "content": "Explain quantization in one paragraph."}], "max_tokens": 200, "temperature": 1.0, "top_p": 0.95, "top_k": 64 }'Also loads under transformers (requires gptqmodel):
bash
pip install transformers torch gptqmodel optimumpython
from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "aleksandard/gemma-4-12B-it-int4-MLPonly-AutoRound"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="cuda") messages = [{"role": "user", "content": "Explain quantization in one paragraph."}]ids = tok.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=False).to("cuda")out = model.generate(ids, max_new_tokens=256)print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))VLLM_USE_FLASHINFER_SAMPLER=0 is required to avoid a
FlashInfer JIT-compile failure during sampling.bash
auto-round \ --model google/gemma-4-12B-it \ --scheme W4A16 \ --iters 0 \ --disable_opt_rtn \ --layer_config '{"model.language_model.layers.\d+.self_attn.q_proj":{"bits":16},"model.language_model.layers.\d+.self_attn.k_proj":{"bits":16},"model.language_model.layers.\d+.self_attn.v_proj":{"bits":16},"model.language_model.layers.\d+.self_attn.o_proj":{"bits":16}}' \ --format auto_gptq \ --output_dir ./gemma-4-12B-it-int4-MLPonlyThis is a quantized derivative; it inherits all limitations and biases of the base model and may show additional deviation due to 4-bit quantization. See the base model card for full details. Quantization was calibration-free (RTN); a calibrated build may recover some quality.
Apache 2.0, inherited from the base model. This repository changes only the numeric precision of the weights.
uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu130 \ --extra-index-url https://download.pytorch.org/whl/cu130 \ --index-strategy unsafe-best-matchexport VLLM_USE_FLASHINFER_SAMPLER=0vllm serve <path-to-this-model> \ --served-model-name gemma4-12b \ --max-model-len 8192 \ --gpu-memory-utilization 0.90 \ --host 0.0.0.0 --port 8000curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4-12b", "messages": [{"role": "user", "content": "Explain quantization in one paragraph."}], "max_tokens": 200, "temperature": 1.0, "top_p": 0.95, "top_k": 64 }'from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "aleksandard/gemma-4-12B-it-int4-MLPonly-AutoRound"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="cuda") messages = [{"role": "user", "content": "Explain quantization in one paragraph."}]ids = tok.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=False).to("cuda")out = model.generate(ids, max_new_tokens=256)print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))auto-round \ --model google/gemma-4-12B-it \ --scheme W4A16 \ --iters 0 \ --disable_opt_rtn \ --layer_config '{"model.language_model.layers.\d+.self_attn.q_proj":{"bits":16},"model.language_model.layers.\d+.self_attn.k_proj":{"bits":16},"model.language_model.layers.\d+.self_attn.v_proj":{"bits":16},"model.language_model.layers.\d+.self_attn.o_proj":{"bits":16}}' \ --format auto_gptq \ --output_dir ./gemma-4-12B-it-int4-MLPonly