GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Broadnet
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Broadnet
Model Tree
cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
Input Modalities
Output Modalities
Supported Functionality
| Base model | google/gemma-4-26B-A4B-it (Apache-2.0) |
| Quantized checkpoint it was trained and measured on | cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit revision 0ef577a5710035bd2d3a3f27e4f5cb2e86a9a9ba |
| Adapter | rank 16, alpha 256, dropout 0, no bias, PEFT LoRA |
| Target modules | attention q,k,v,o and shared-MLP gate,up,down projections across all 30 decoder layers, 205 trained modules (the packed MoE experts stay frozen) |
| Served as | --lora-target-modules qkv_proj o_proj gate_up_proj down_proj, rank pinned at 16 |
| Training corpus | 965 rows (pilot corpus; English chat/software/stem/tool-calling plus Arabic), 2 epochs, 1,930 updates |
| Checkpoint published | step 1,900 of 1,930 |
adapter_model.safetensors sha256 | cabc83cbb098cd526f07fc872822e5af157820743215951f5c3a22bde4d7642d (75,627,016 bytes) |
The throughput, acceptance, cycle-timing and sampled-gate measurements below used the historical Gemma overlay on the first-generation runner, with a paired plain control in the same serving session on an NVIDIA RTX 3090 under vLLM 0.28.1rc1.dev437 (seed 29). The v0.3.0 release profile, on the second-generation runner, is the one certified and launched below. The throughput rows use the 72-prompt production-sampling workload; cycle timing uses five fixed 384-token greedy requests after two warm-ups.
| quantity | value |
|---|---|
| Accepted draft share at K=4 (real, per-position counters) | 0.4888 (~48.9%) |
| Paired plain control, full72 | 132.279 tok/s |
| Adapter, full72 | 154.261 tok/s |
| Speedup vs paired plain | 1.1662x |
| Survival by draft position (1/2/3/4) | 0.9164 / 0.6069 / 0.3070 / 0.1251 |
| K=8 (same adapter) | 1.0004x, break-even, does not beat K=4 |
| Lossless gate, sampled-distribution permutation test | PASS (4 comparisons incl. the plain-vs-plain floor and the forced all-reject arm; 256 samples × 16 tokens over 3 pinned prefixes, 36 tests, Bonferroni α 2.78e-4, 35,999 permutations) |
| Lossless, in what sense |
The gain over the earlier 400-update point of the same lineage is concentrated in the deep draft positions (position 4: 0.0502 → 0.1251 survival), not position 1. The lossless result reads "no discrepancy detected at this sample size", not proven equivalence, the gate's own qualification.
The adapter is a rank 16 LoRA (alpha 256) on the query, key, value, output, gate, up and down projections of every
layer of google/gemma-4-26B-A4B-it. This adapter was trained and evaluated on the pinned AWQ checkpoint named
above; compatibility, acceptance, and throughput on other checkpoints require validation. On the tested card in
the v0.3.0 release, lossless is used in this sense: Uno is as close to plain as plain is to itself across sessions on this hardware, under greedy and sampled decoding alike. It does not apply to
other Gemma 4 sizes (the 31B dense or E4B models have different shapes).
The adapter is loaded by the vLLM-Uno fork, v0.3.0 or later. The serving conventions below
are the ones the adapter was verified under on the fork's Gemma 4 port (the release's
docs/validation.md records the runs; the refusals such as language-only and single-KV-group
requirements are enforced by the fork):
--dtype bfloat16, --attention-backend TRITON_ATTN (required: the decoder layers mix
sliding-window and full attention, and on this build the FlashInfer path's one-plan window
assertion kills Uno on the first request. FlashInfer's supported head sizes cover this
checkpoint, so the blocking failure is the mixed layer configuration, not head size, and every
published Gemma Uno number is a Triton-attention number)--language-model-only, --disable-hybrid-kv-cache-manager--max-model-len 8192, --max-num-seqs 4, --max-num-batched-tokens 2048,
--gpu-memory-utilization 0.85"num_speculative_tokens": 4, "uno_lora_path": "<adapter dir>",
--max-lora-rank 16bash
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_LORA_ENABLE_DUAL_STREAM=1 python3 -m vllm.entrypoints.cli.main serve cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit --served-model-name uno-gemma4-26b-a4b --host 0.0.0.0 --port 8000 --tensor-parallel-size 1 --api-server-count 1 --async-scheduling --generation-config vllm --dtype bfloat16 --attention-backend TRITON_ATTN --language-model-only --disable-hybrid-kv-cache-manager --enable-prefix-caching --seed 29 --max-model-len 8192 --max-num-seqs 4 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.85 --enable-lora --lora-dtype bfloat16 --max-lora-rank 16 --max-loras 2 --max-cpu-loras 2 --lora-target-modules qkv_proj o_proj gate_up_proj down_proj --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,3,4,5,6,7,8,13,14,15,16]}' --speculative-config '{"method":"uno","uno_lora_path":"/path/to/adapter","uno_mask_token_id":262144,"uno_noise_seed":29,"num_speculative_tokens":4,"uno_noise_low":0}'
This is the command the release launcher's gemma4 profile builds (
markdown
UNO_PROFILE=gemma4 bash release/serve.sh cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit <adapter directory>
UNO_DRAFT_MOE_TOPK=4 in the environment) and is not part of this
command. Run the launcher to reproduce the certified profile exactly rather than rebuilding the
command by hand.
Notes:
--max-num-seqs 1 with the fixed request set; the
port verification ran at --max-num-seqs 4. Both use the same Uno K=4 speculative config.adapter_model.safetensors hash before
starting each arm.num_speculative_tokens 4 is the measured knee; K=8 is break-even and costs more per cycle.If you use this adapter or the measurements above, cite the accompanying paper (BroadNet Research, 2026, https://doi.org/10.5281/zenodo.22820511):
bibtex
@report{al_zubaidi_uno_moe_2026,title = {Self-drafting with Uno on a mixture of experts: serving and training an adapter for Gemma 4 26B A4B},author = {BroadNet Research Team},affiliation = {BroadNet},year = {2026},doi = {10.5281/zenodo.22820511},url = {https://doi.org/10.5281/zenodo.22820511},note = {Preprint}}
The dense-target study, Uno in vLLM: An Independent Implementation and Empirical Serving Study
(DOI 10.5281/zenodo.22652609), covers the Qwen3-8B results and is the implementation reference for
that target; cite it for those results, not for this adapter's measurements.
Uno, the method this adapter implements, and the training recipe it follows are the work of Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing and Zhengzhong Liu: Unlocking Lossless Speedups in LLMs via Discrete Diffusion, arXiv:2609.04010, 2026 (https://arxiv.org/abs/2609.04010; code and adapters https://github.com/ifm-ai/uno), from the Institute of Foundation Models (IFM) and collaborators. Please cite their paper for the method.
Gemma 4 26B A4B and its multi-token-prediction assistant drafter are by Google; the AWQ 4-bit checkpoint this adapter attaches to is by cyankiwi (https://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit); the DFlash drafter used as a comparator is by z-lab (https://huggingface.co/z-lab/gemma-4-26B-A4B-it-DFlash). vLLM is the serving engine this work extends.
This adapter, the Gemma 4 port and every measurement on this card are an independent implementation by the BroadNet Research Team.
The adapter weights are Apache 2.0, like the Gemma 4 base model they attach to. The quantized checkpoint you serve them with belongs to its authors.
The adapter was trained on 965 rows from four public datasets: DataCreatorAI/Multi-Turn-Conversational-SFT (Apache 2.0), nvidia/Nemotron-Post-Training-Dataset-v1 (CC BY 4.0, thanks to NVIDIA), m-a-p/CodeFeedback-Filtered-Instruction (Apache 2.0) and RASHID778/arabic-instruct-curated (Apache 2.0). None of the rows were written by the base model. One thing worth knowing before you build on it: 220 of the 965 rows (22.8%) come from the code-feedback set, whose own card says it contains text generated by OpenAI models. We kept those rows in this pilot and are saying so here so you can decide for yourself.
| On the tested card in the v0.3.0 release that word is used in this sense: Uno is as close to plain as plain is to itself across sessions on this hardware, under greedy and sampled decoding alike |
UNO_DRAFT_MOE_TOPK=4) is part of the measured serving profile: the
measured run's engagement line records moe_top_k=4, and the variant rejects an uncaptured
serving shape by name under traffic rather than falling back to eager routing.
--enforce-eager is a diagnostic flag, not a serving suggestion.--max-loras 2, --max-cpu-loras 2), prefix
caching on, pinned seed 29 and a fixed CUDA-graph capture list; the served model name is
uno-gemma4-26b-a4b.