GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
TheUnderscore
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
TheUnderscore
Model Tree
Gryphe/Gemma-4-26B-A4B-StyleTune-V2
Input Modalities
Output Modalities
Supported Functionality
| Layer | Format | Group |
|---|---|---|
| Attention (q/k/v/o) | W4A16 (int4) | 128 |
| MLP / MoE experts | NVFP4 (float4) | 16 |
| Embeddings, norms, router | BF16 | — |
oneshot with config_groups per-layer dispatch.quant_method: "compressed-tensors", two config_groups.Requires 1Cat-vLLM with the feat/gemma4-sm70-native-v3 branch (Gemma-4 model port + SM70 D=512 attention kernels). Untested on standard vLLM.
python
from vllm import LLM, SamplingParams llm = LLM( "TheUnderscore/Gemma-4-26B-A4B-StyleTune-V2-NVFP4-W4ATTN-AWQ", tensor_parallel_size=2, max_model_len=8192,)output = llm.generate("Hello", SamplingParams(temperature=1.0, top_p=0.95, max_tokens=256))print(output[0].text)quantize-nvfp4-gemma4.py — quantization script.