GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Broadnet
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Broadnet
Model Tree
cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
Input Modalities
Output Modalities
Supported Functionality
| Base model | google/gemma-4-26B-A4B-it (Apache-2.0) |
| Checkpoint it was trained and measured on | cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit revision 0ef577a5710035bd2d3a3f27e4f5cb2e86a9a9ba |
| Adapter | PEFT LoRA, rank 16, alpha 256, dropout 0, no bias |
| Target modules | attention q,k,v,o and shared-MLP gate,up,down in all 30 decoder layers (the packed MoE experts stay frozen) |
| Training | 38,560 rows in six stages: the 10k-prompt adapter (10,000 rows, 2 passes, 20,000 updates), then stages of 5,000, 5,000, 8,560, 5,000 and 5,000 new rows (2 passes each: 10,000, 10,000, 17,120, 10,000 and 10,000 updates), every stage continuing from the previous adapter, learning rate 1e-5; then a seventh stage of 2,000 updates on rows of the sixth stage, with five-row draft blocks (the earlier stages trained four-row blocks) and with the draft passes routed as they are served (4 of the 8 experts, the 64k draft vocabulary); one RTX 3090 (24 GB), trained through the same AWQ 4-bit checkpoint it serves on |
| Checkpoint published | seventh stage, step 2,000 |
adapter_model.safetensors sha256 | 5eda8f7879489737246b86c56f2836e2d4cd359e0bc994ea193ac9f9f658e534 (75,627,016 bytes) |
One RTX 3090, the v0.4.4 container, the recommended settings (UNO_RECOMMENDED=1): K=5 Uno drafts, prompt lookup on
(UNO_PLOOKUP_L=2, UNO_PLOOKUP_MAX_CTX=4096), draft passes routed through 4 of the 8 experts per token
(UNO_DRAFT_MOE_TOPK=4), split-KV attention for steps with several requests (UNO_GEMMA_SPLITKV_MULTI=1) with CUDA
graphs for up to 8 requests, the shipped 64k draft vocabulary. Plain decoding and the DFlash drafter (z-lab, K=8, run
on the stock vllm/vllm-openai:v0.30.0 image with the same serve flags) were measured in the same session, three rounds
each with the arm order rotated, every server started fresh for its workload and sent two warmup requests (the first two
production requests) before the timed ones. Production traffic = 72 real requests. With several requests in flight
each figure is the median of the three rounds, with the range in brackets; one request at a time the table gives each
round, and tokens per second at the median of the three rounds; for long prompts each figure is the median of all 24
requests of the three rounds pooled. Speed-ups over plain are ratios of these medians. This adapter's figures are
rounded so that none is better than measured.
Several requests at a time (the 72 requests with four or eight in flight, wall clock over the whole batch):
| tokens per second, whole batch (median of three rounds, range) | this adapter | DFlash K8, vLLM 0.30.0 | plain |
|---|---|---|---|
| eight requests at a time | 721 (721-723; 1.38x plain) | 679 (673-679; 1.31x) | 520 (517-524) |
| four requests at a time | 523 (521-528; 1.48x plain) | 497 (487-504; 1.41x) | 354 (353-356) |
This adapter took 5.8 %, 5.8 % and 7.0 % less time per token than stock DFlash K8 in vLLM 0.30.0 at eight in flight (rounds 1, 2 and 3), and 6.7 %, 6.0 % and 3.7 % less at four.
One request at a time (HTTP clock: completion tokens over request time):
| one request at a time | this adapter | DFlash K8, vLLM 0.30.0 | plain |
|---|---|---|---|
| ms per output token (three rounds) | 4.463 / 4.411 / 4.420 | 4.650 / 4.786 / 4.820 | 7.506 / 7.471 / 7.469 |
| tokens per second per request (median round) | 226 | 209 | 134 |
| speed-up over plain | 1.69x | 1.56x | 1.00x |
| tokens per cycle (τ) | 4.09 / 4.11 / 4.11 | 3.32 / 3.22 / 3.24 |
This adapter took 4.0 %, 7.8 % and 8.3 % less time per token than stock DFlash K8 in vLLM 0.30.0 (rounds 1, 2 and 3).
Prompt lookup. Since v0.4.3 there is a second source of candidates: after Uno's five drafted tokens, two more are copied from the request itself (the tokens that followed the last place the current text appeared in the prompt or the answer so far). In a separate session with an earlier training stage (K=4; v0.4.3 release evidence), lookup with the length gate below cut the time per token by 2.9 % on this traffic against lookup off. On long prompts lookup did not pay: without the gate it was 4.7 % and 4.3 % slower than lookup off at 14k and 28k tokens. So v0.4.3 switches it off per request once a request's context passes 4,096 tokens: long requests then verify only Uno's drafts, as in v0.4.2.
Long prompts. Open-data documents at three lengths (4 document sets x 2 tasks, a summary and an original story, up to 384 output tokens each, one at a time), same session and settings, on the one-at-a-time servers after the production requests; median decode time per output token, all three rounds pooled (24 requests per cell):
| prompt length | this adapter | DFlash K8, vLLM 0.30.0 | plain |
|---|---|---|---|
| 2k tokens | 4.84 ms (1.53x) | 6.33 ms (1.17x) | 7.43 ms |
| 14k tokens | 6.28 ms (1.37x) | 10.24 ms (0.84x) | 8.60 ms |
| 28k tokens | 7.30 ms (1.31x) | 14.07 ms (0.68x) | 9.63 ms |
Against stock DFlash K8 in vLLM 0.30.0, this adapter took 34.8 %, 43.8 % and 38.0 % less time per token at 14k (rounds 1, 2 and 3) and 48.1 %, 45.5 % and 47.5 % less at 28k. Prompts above 4,096 tokens verify only Uno's drafts (the length gate).
Lossless. Every candidate, drafted by Uno or copied from the prompt, is checked by vLLM's own rejection sampler against the full model before it is emitted, so the output distribution is the model's own; in the Uno authors' words, Uno "accelerates generation without sacrificing the quality of the underlying AR model" (Sahoo et al.). A copied token is a fixed proposal, which the sampler accepts with exactly the model's probability for it. The release's CPU tests run vLLM's real rejection kernels over the served shape (five drafts plus two lookup tokens) and check the emitted tokens against the model's distribution under greedy decoding, temperature 1, temperature 0.7 with top-p 0.9, vocabularies above 8,192 tokens, and batches that mix short requests (with lookup) and long ones (without). The adapter, and routing the draft pass through 4 experts, change only which tokens Uno proposes: the model's verification pass always uses all 8 experts per token, and the sampler checks each proposal against the draft distribution it was actually drawn from. With several requests in flight, v0.4.4 computes their attention in one split-KV launch; a GPU test on the release image checks that each request's attention output is bit for bit the one it gets when it runs alone.
BroadNet serves Gemma 4 26B A4B to its clients from its own GPUs, for agent workloads in English and Arabic: long system prompts, tool calls, structured JSON, multi-step coordination. Faster decoding on the same card means faster answers and more work per GPU, with the model's output distribution unchanged. More open prompts kept making the drafter better, and copying from the prompt adds what a drafter cannot know in advance: the exact strings the request itself contains. This adapter's last training stage drafts as many tokens per block as it is asked to draft when served.
UNO_RECOMMENDED=1, below). Without them this adapter
runs on the v0.4.2 path (K=4, no lookup, 8-expert drafts); its speed there was not measured for this card.UNO_DRAFT_MOE_TOPK=4, part of the recommended settings) runs only on RTX 30-series
class GPUs (compute capability 8.6) and refuses to start elsewhere; there, add UNO_DRAFT_MOE_TOPK= (empty). It runs
only inside captured CUDA graphs, so with it the server refuses to start when no draft graph will be captured
(--enforce-eager, or a CUDA graph mode without full decode graphs), and checks at startup that every draft batch
it can be asked for has a captured graph: with the recommended settings (up to 8 requests, capture sizes up to 64),
--max-num-seqs above 8 stops at startup with a message naming the missing batch sizes. Only UNO_K=5 with lookup
UNO_PLOOKUP_L=2 was measured; with the multi-request path on, the server refuses more than 9 verify rows per
request (UNO_K + UNO_PLOOKUP_L + 1).With the v0.4.4 container, the model cache and this adapter mounted, and the gemma4 profile selected (the profile pins
the checkpoint revision above, serves at 32k context on vLLM's hybrid KV cache manager with split-KV draft attention
and the 64k draft vocabulary on), plus the recommended settings in one switch:
bash
docker run --rm --name vllm-uno-gemma --gpus all --ipc=host \ -p 127.0.0.1:8000:8000 \ -v /path/to/hf-cache:/root/.cache/huggingface \ -v /path/to/this-adapter:/adapter:ro \ -e UNO_PROFILE=gemma4 \ -e UNO_RECOMMENDED=1 \ ghcr.io/brntech/vllm-uno:0.4.4 \ cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit /adapter \ -- --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4The flags after -- turn on Gemma 4's tool-call and reasoning parsers, as on the measured servers.
UNO_RECOMMENDED=1 sets UNO_K=5, UNO_PLOOKUP_L=2, UNO_PLOOKUP_MAX_CTX=4096, UNO_DRAFT_MOE_TOPK=4 and
UNO_GEMMA_SPLITKV_MULTI=1 (with capture sizes up to 64); a variable you set yourself wins. Set to empty,
UNO_PLOOKUP_L= turns prompt lookup off, UNO_DRAFT_MOE_TOPK= drafts with all 8 experts and
UNO_GEMMA_SPLITKV_MULTI= turns the multi-request path off; an empty UNO_K=, or an empty UNO_PLOOKUP_MAX_CTX= while
lookup is on, is refused at startup. The server prints the settings it used at startup.
The served model name is uno-gemma4-26b-a4b. With the recommended settings the KV cache holds 75,731 to 75,768
tokens at 32k context on a 24 GB RTX 3090.
The earlier adapters stay available:
hf download Broadnet/gemma-4-26B-A4B-uno-adapter --revision s5 --local-dir /path/to/s5-adapter (38.6k prompts, trained
with four-row draft blocks, v0.4.3), --revision s3 (28.6k prompts, v0.4.2) or --revision p10k (10k prompts, v0.4.0
and v0.4.1); mount that directory as /adapter.
The adapter weights are Apache 2.0, like the Gemma 4 base model. They were trained on 38,560 prompts drawn from public datasets; the answers the drafter learned from were written by the Gemma 4 model itself. No private or production data was used for training. The 10,000 rows added since the 28.6k-prompt adapter come from the same datasets and subsets; the seventh stage adds no new rows. Dataset licences as published on each dataset card (revisions pinned in the corpus recipe):
| dataset | subsets used | licence |
|---|---|---|
HuggingFaceTB/smoltalk2 (SFT) | SmolLM3 everyday conversations, Magpie Ultra, system chats, Explore-Instruct rewriting, smol-rewrite, smol-summarize (*_no_think); Tulu-derived subsets excluded; prompts only | per the SmolTalk2 card, subsets drawn from existing public datasets keep their original licences (its new datasets are Apache 2.0) |
OpenAssistant/oasst2 | train | Apache 2.0 |
m-a-p/CodeFeedback-Filtered-Instruction | train | Apache 2.0 (its card notes text generated by OpenAI models) |
The 64k draft vocabulary shipped with v0.4.0 is ranked on the first 10,000 of these prompts and the model's answers to them.
Uno is the work of Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing and Zhengzhong Liu: "Unlocking Lossless Speedups in LLMs via Discrete Diffusion" (arXiv:2609.04010; code at ifm-ai/uno). Gemma 4 is by Google; the AWQ checkpoint is by cyankiwi; the DFlash comparator drafter is by z-lab; serving runs on vLLM. This adapter, the vLLM integration and every measurement on this card are BroadNet's independent implementation, by the BroadNet Research Team.
| 1 |
nvidia/Nemotron-Post-Training-Dataset-v1| stem, tool_calling |
| CC BY 4.0 (thanks to NVIDIA) |
Team-ACE/ToolACE | train | Apache 2.0 |
NousResearch/hermes-function-calling-v1 | func_calling, func_calling_singleturn | Apache 2.0 |
open-thoughts/OpenThoughts3-1.2M | train | Apache 2.0 |
nvidia/OpenMathReasoning | cot | CC BY 4.0 (thanks to NVIDIA) |