What was quantized
Table with columns: Part, Format| Part | Format |
|---|
| Routed experts (256 per layer × 40 layers; gate, up and down projections) | NVFP4: FP4 (E2M1) weights, one FP8 (E4M3) scale per 16 weights, one FP32 global scale per matrix |
| Attention, linear attention (Gated DeltaNet), shared expert, router, embeddings, LM head | BF16 (unchanged) |
| Vision encoder and merger | BF16 (unchanged) |
| Activations | BF16 (weight-only quantization: "A16") |
- Format: compressed-tensors
nvfp4-pack-quantized, the same
layout llm-compressor writes for its NVFP4A16 scheme, so vLLM loads it with no extra flags.
- Method: data-free round-to-nearest using compressed-tensors' own NVFP4 functions, one expert matrix at a time
(the original stores the experts fused; they are split per expert here), starting from the published BF16 weights
at revision
ca2b95a. No calibration data was used.
- For each expert, the gate and up projections share one global scale. vLLM fuses them into one matrix and keeps one
scale for both.
- Measured relative weight error (Frobenius norm, sampled experts in every shard): 9.49% on average and 9.55% at
most, which is typical for NVFP4 round-to-nearest.
Running it with vLLM
Tested with vLLM 0.32.0 on one RTX PRO 6000 Blackwell (96 GB). Serve it with Docker:
docker run --gpus all --ipc=host -p 8000:8000 vllm/vllm-openai:v0.32.0 \ willyb23/NorthCUA-Mini-NVFP4A16 \ --max-model-len 65536 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
or, with vLLM installed (pip install vllm==0.32.0), the same flags after vllm serve willyb23/NorthCUA-Mini-NVFP4A16.
vLLM takes 92% of the GPU's memory by default; add --gpu-memory-utilization 0.8 (or lower) if something else uses
the card. vLLM picks its Marlin FP4 MoE kernel for this format by itself.
Thinking is on by default. Mini's chat template switches it with reasoning_effort, not enable_thinking (Nano's
uses enable_thinking). To turn thinking off with vLLM's qwen3 reasoning parser, send both keys:
"chat_template_kwargs": {"reasoning_effort": "none", "enable_thinking": false}. The template reads the first, and the
second tells vLLM's parser to return the reply as content; with reasoning_effort alone, the reply arrives in the
reasoning field and content is empty.
The examples below were run against that server with the openai Python client (pip install openai); only the
served model name differed:
import base64from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")MODEL = "willyb23/NorthCUA-Mini-NVFP4A16" # 1. Ask about a screenshot (thinking off for a short, direct answer)image = base64.b64encode(open("screenshot.png", "rb").read()).decode()resp = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}}, {"type": "text", "text": "What does this dialog ask, and which buttons does it offer?"}, ]}], max_tokens=256, extra_body={"chat_template_kwargs": {"reasoning_effort": "none", "enable_thinking": False}},)print(resp.choices[0].message.content) # 2. Thinking on (the default): the reasoning comes back in its own fieldresp = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "A train covers 240 km in 3 hours. How far does it go in 5 hours?"}], max_tokens=2048,)print(resp.choices[0].message.reasoning) # vLLM 0.32 returns the thinking as `reasoning`print(resp.choices[0].message.content) # "... 80 km/h × 5 h = 400 km." # 3. A tool calltools = [{"type": "function", "function": { "name": "get_weather", "description": "Current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]resp = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, max_tokens=1024,)print(resp.choices[0].message.tool_calls[0].function) # get_weather, arguments '{"city": "Paris"}'
For computer use, follow the original card. Run the model with the official
NorthCUA Harness, and keep every GUI coordinate in the
normalized 0–1000 space (normalized_1000). The Harness converts them to pixels.
Faster generation with speculative decoding (DFlash)
Mini generates up to about twice as fast with
z-lab/Qwen3.5-35B-A3B-DFlash, a 0.77 GB DFlash drafter made for
the base model, Qwen3.5-35B-A3B. The drafter proposes several tokens at a time, and Mini checks them all in one step,
so the output still comes from Mini. Add this to the command above (vLLM 0.32 supports DFlash natively, without
--trust-remote-code):
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-35B-A3B-DFlash", "num_speculative_tokens": 7}'
Measured 2026-11-06 on one RTX PRO 6000 Blackwell with vLLM 0.32.0, with a local copy of the drafter at revision
52cb554. Greedy decoding; tokens per second after the first token:
Table with columns: Output, Without drafter, DFlash, 7 tokens| Output | Without drafter | DFlash, 7 tokens |
|---|
Computer-use step (screenshot + click tool, thinking on), 4 prompts | 184–195 | 190–230 |
| Word problems, thinking on | 179–181 | 359–388 |
| A short Python function | 185 | 414 |
| Copying a text with small edits, thinking off | 179 | 795 |
| All 8 prompts, one at a time | 182 | 339 (1.86×) |
| The same prompts, 8 in flight (total) | 531 | 710 (1.34×) |
The gain is smallest on computer-use steps, because their output is short and specific. With 15 drafted tokens,
generic text got faster still (403 tokens/s one at a time), but computer-use steps and 8 requests in flight got
slower, so 7 is the better setting.
Accuracy is unchanged within measurement error. The same benchmarks as below, rerun with the drafter on: ScreenSpot-Pro
61.10% (62.05% without it; paired McNemar p = 0.20), GSM8K 85.44% (87.04%; p = 0.11), IFEval prompt-level strict
78.74% (78.74%; p = 1.0). Verifying several drafted tokens at once uses slightly different GPU arithmetic than
generating one at a time, so a greedy output can change where two tokens are nearly tied. Without the drafter, the
same request always gave the same text.
What did not help: NorthCUA-Mini ships without the multi-token-prediction (MTP) layer of Qwen3.5, and grafting the
base model's MTP layer back gave only 1.02× (it was not trained on Mini's outputs). Prompt-lookup (n-gram) drafting
gave 1.13×, almost all of it on the copy task.
Other inference engines
Table with columns: Engine, Status| Engine | Status |
|---|
| vLLM 0.32.0 | tested (above) |
| SGLang | not tested yet. It reads compressed-tensors checkpoints, but this NVFP4 weight-only layout has not been tried |
| llama.cpp, Ollama | these load GGUF files, not this format. For them, use a GGUF of the original, such as bartowski/beipo-ai_NorthCUA-Mini-GGUF |
| TensorRT-LLM | not tested. Its NVFP4 support is built around NVIDIA's ModelOpt checkpoint layout, not this one |
Benchmarks
Run 2026-11-06 against the BF16 original, with identical requests to both versions: vLLM 0.32.0, greedy decoding,
thinking off (reasoning_effort: "none" + enable_thinking: false), each version on its own RTX PRO 6000 Blackwell
(96 GB). "Paired p" is an exact McNemar test on the questions where the two versions disagree; below 0.05 would be a
real difference.
Table with columns: Benchmark, This NVFP4 checkpoint, BF16 original, Paired p| Benchmark | This NVFP4 checkpoint | BF16 original | Paired p |
|---|
| ScreenSpot-Pro, all 1,581 (GUI grounding) | 62.05% | 60.78% | 0.11 |
| ScreenSpot-Pro, text targets (977) | 76.97% | 76.56% | |
| ScreenSpot-Pro, icon targets (604) | 37.91% | 35.26% | |
| GSM8K, 1,319 (5-shot, flexible extract) | 87.04% | 86.73% | 0.80 |
| GSM8K, strict match | 82.11% | 82.94% | |
| IFEval, 541 (prompt-level strict) | 78.74% | 80.41% | 0.33 |
| IFEval, instruction-level strict | 85.13% | 86.09% | |
| Single stream, 512 tokens incl. prefill (one run each) | 181.8 tok/s | 181.9 tok/s | |
| Weights in GPU memory | 22.3 GiB | 65.5 GiB | |
No difference is statistically significant: on these benchmarks the quantization costs nothing measurable, and it
cuts the weights' memory to a third.
How they were run:
- ScreenSpot-Pro (likaixin/ScreenSpot-Pro, one prediction
per screenshot, no zoom or crop): a system prompt with the coordinate rules of the NorthCUA Harness (x, y normalized
to 0–1000 relative to the screenshot) and the Harness's
click tool as a native tool. Scored by ScreenSpot-Pro's
rule (the click must fall inside the target box). The NorthCUA authors did not publish their exact grounding prompt,
and their reported 92.41% is for an earlier checkpoint, so the absolute numbers here are not comparable with the
report; the comparison between the two versions is.
- GSM8K and IFEval: lm-evaluation-harness 0.4.13 through
the chat-completions API, with the chat template and few-shot examples as turns, up to 2,048 new tokens.
- vLLM was my own build of 0.32.0; its patches concern multi-machine serving, not this format.
Smoke tests
Run 2026-11-05, same setup, thinking at the template default.
Table with columns: Check, This NVFP4 checkpoint, BF16 original| Check | This NVFP4 checkpoint | BF16 original |
|---|
| Arithmetic probe (17 × 23) | 391 | 391 |
| 10 word problems with exact answers (thinking on) | 10/10 | 10/10 |
| 8 short factual questions | 8/8 | 8/8 |
Tool call (get_weather, qwen3_coder parser) | correct name and arguments | correct name and arguments |
| A fact hidden in a 25,248-token prompt | found | found |
Files
The shard boundaries of the original are kept. checksums.sha256 covers every file
(sha256sum -c checksums.sha256).
License and attribution
Apache-2.0, as for the original. LICENSE and NOTICE are the original files, unchanged.
Changes made: the routed-expert weights of NorthCUA-Mini were converted to NVFP4 (per-expert tensors replace
the fused experts.gate_up_proj / experts.down_proj tensors), and a quantization_config was added to
config.json. All other weights, the tokenizer, the processor configuration and
the chat template are unchanged.
If you use the model, cite the NorthCUA technical report (see the original model card).