GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
jcbtc
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
jcbtc
Model Tree
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
Input Modalities
Output Modalities
Supported Functionality
| Component | Format | Notes |
|---|---|---|
| Main model | Hugging Face safetensors, compressed-tensors pack-quantized | Symmetric group-64 INT4 for attention, MLP, and experts; BF16 activations; unquantized lm_head, routers, and vision tensors. |
| MTP assistant | mtp/model.safetensors, BF16 | Matching Google QAT assistant, used at speculative depth 4. |
| Tokenizer and processor | Original Gemma 4 files | Preserved alongside the sharded model. |
The INT4 data plus BF16 group scales cost 4.25 bits per quantized weight. That is not an overall bits-per-weight figure: the repository also has BF16 tensors and a separate BF16 draft. Main-model safetensors total about 15.66 GB (decimal); the assistant safetensors is about 0.84 GB. This is a quantized conversion of Google's instruction-tuned QAT checkpoint, not a new fine-tune.
These are observed output tokens divided by complete request-wave wall time, including prefill and the draining tail, unless noted otherwise. They are not isolated kernel or per-agent decode rates. Hardware is one AMD Ryzen AI MAX+ 395 with 128 GB unified memory for each single-host row; the exact prompt, cap, sampler, and runtime differ by row. Do not compare unlike rows as a controlled speedup.
| Single-system test | Workload | Output rate | Other result |
|---|---|---|---|
| Warmed OpenAI-compatible API, Sozo | 32 executing requests, 512-output cap, 16,384 output tokens | 537.67 tok/s | 32/32 completed; median streamed first token 0.947 s. |
| Sustained offline generation, Sozo | 32 executing requests, 2,048-output cap, natural EOS, 59,914 output tokens | 420.80 tok/s | Zero preemptions; 78.87% MTP draft acceptance. |
| EvalScope serving, Ciru | 32 executing requests, 1,024 input + 2,048 forced output each, 65,536 output tokens | 296.92 tok/s | 32/32 succeeded; full five-second output peak 451.34 tok/s. |
| Cold 64K prefill, Ciru |
The 64K prefill result uses our later V3 long-prefill kernel. The quality suites below ran before that kernel promotion; three exact long-context retrieval checks passed on both old and new kernels, but the full quality suite was not rerun after V3. A 262,144-token context was configured and boundary components were exercised; we do not claim 256K prefill at the 64K rate.
The complete native matched suite used its own deterministic evaluation protocol: GSM8K 95.45% (1,259/1,319), HumanEval 95.73% (157/164), and IFEval prompt-level strict 87.62% (541 prompts). All 2,024 requests were accounted for, with zero execution errors; five responses reached their output limit and remain in the scores. These are scores for this converted model and serving stack, not a claim of improvement over the Google source model.
A separate EvalScope scorecard used the recommended sampler (temperature 1.0, top-p 0.95, top-k 64, repetition penalty 1.0; seed 42; 8,192 output-token cap) and 64 queued clients behind 32 execution slots. Across 941 questions, sampled GSM8K scored 193/200 (96.5%), stratified sampled MMLU-Pro 145/200 (72.5%), and full IFEval 87.99% prompt-level strict. All three evaluations completed with zero execution errors. The GSM8K and IFEval scores in the two protocols should not be collapsed into a single result.
More exact settings and definitions are in the benchmark record.
Open the dual-Strix-Halo 64-app showcase →
Watch the 64-agent showcase on X →
The original generation pass ran 64 independent app requests (32 slots on each machine) in 5 minutes 32 seconds, emitted 219,988 output tokens, and averaged 661.7 combined output tok/s. The highest complete five-second window was 1,166 combined output tok/s. Sixty-three first-pass requests ended normally and one hit its output limit. The linked gallery contains the final 64 playable pages after subsequent repairs and small publication fixes; the speed and time figures exclude that later work. The two hosts averaged about 339 and 323 output tok/s over the shared wall time. This is an end-to-end workflow on two systems, not a one-machine model benchmark.
AMD provided one of the Ryzen AI Halo systems used to create the showcase content. The measurements and model conversion are Ciru's work.
Download the full repository so the mtp/ folder is present. With a Gemma 4-capable vLLM/ROCm build and sufficient available memory:
bash
hf download jcbtc/Gemma4-26B-CIRU-AMD-STRIX-vLLM --local-dir ./gemma4-ciru-strix vllm serve ./gemma4-ciru-strix \ --language-model-only \ --max-model-len 8192 \ --max-num-seqs 8 \ --max-num-batched-tokens 2048 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","model":"./gemma4-ciru-strix/mtp","num_speculative_tokens":4,"quantization":null,"draft_tensor_parallel_size":1}'This is a portable starting profile, not the measured 32-slot AMD configuration. The benchmarked systems used a Ciru-modified ROCm vLLM stack, --max-num-seqs 32, --max-num-batched-tokens 2048, --attention-backend ROCM_AITER_UNIFIED_ATTN, Triton MoE, an auto-selected dense W4A16 kernel, eager execution, and an explicit 64–80 GiB KV reservation depending on the test. Ciru's V3 attention patch was used for the final 64K prefill figure. The custom dense and prefill optimizations are not included in these weights, and unmodified vLLM has not been qualified for the listed Strix Halo speeds. For installation and API examples, see the official vLLM Gemma 4 guide. The measured application and quality runs were text-only (--language-model-only); the retained vision tensors are not a vision-quality claim for this release.
Google's generation defaults are temperature 1.0, top-p 0.95, and top-k 64. Set request parameters deliberately when comparing scores. The source model's chat template supports Gemma 4 thinking and tool calling; prompt formatting and parser support depend on the serving build.
f1e06dc520982d9b9edd76859fdb7ab209449949.9537141506fe8875b3ed45b264af13580cb29166.The upstream source models are published under Apache 2.0; this conversion is released under the same license. Please retain the Google and other upstream attributions when redistributing it.
| 65,504 input + 32 output, zero cache hits, two V3 runs |
| 1,023–1,029 input tok/s |
| Time to first token 63.64–64.01 s; single request, not output tok/s. |