What was quantized
Table with columns: Part, Format| Part | Format |
|---|
| Routed experts (512 per layer × 60 layers; gate, up and down projections) | NVFP4: FP4 (E2M1) weights, one FP8 (E4M3) scale per 16 weights, one FP32 global scale per matrix |
| Attention, linear attention (Gated DeltaNet), shared expert, router, embeddings, LM head | BF16 (unchanged) |
| Vision encoder and merger | BF16 (unchanged) |
| Activations | BF16 (weight-only quantization: "A16") |
- Format: compressed-tensors
nvfp4-pack-quantized, the same
layout llm-compressor writes for its NVFP4A16 scheme, so vLLM loads it with no extra flags.
- Method: data-free round-to-nearest using compressed-tensors' own NVFP4 functions, one expert matrix at a time
(the original stores the experts fused; they are split per expert here), starting from the published BF16 weights
at revision
b5d64ab. Every source file was checked byte for byte against the Hub before conversion. No
calibration data was used.
- For each expert, the gate and up projections share one global scale. vLLM fuses them into a single matrix and keeps
one scale for both.
- Measured relative weight error (Frobenius norm, one expert in 64 sampled in each of the 120 shards that hold
experts): 9.2% on average and 9.6% at most, which is typical for NVFP4 round-to-nearest.
Running it with vLLM
Tested 2026-11-06 with the stock vllm/vllm-openai 0.32.0 image at tensor-parallel 4 across four NVIDIA DGX Spark
machines (one GB10 GPU each, linked by 200 GbE RoCE), which picked its Marlin NVFP4 MoE kernel by itself. On one
machine with four GPUs the usual form is:
vllm serve willyb23/NorthCUA-Pro-NVFP4A16 \ --tensor-parallel-size 4 \ --max-model-len 262144 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
(a single four-GPU machine was not tested). Across four machines, run the same command on each with
--nnodes 4 --node-rank <0..3> --master-addr <rank 0's address> --master-port <port>, start rank 0 last, and set
NCCL to your network (NCCL_SOCKET_IFNAME, and NCCL_IB_HCA for RDMA). The tested run used
--max-num-seqs 8 --kv-cache-memory-bytes 8589934592 (8 GiB of KV cache per GPU: 543,820 tokens, two requests at the
full 262,144-token window). Loading took about 10 minutes from a network share.
Thinking is on by default. Pro's chat template switches it with enable_thinking (Mini's uses reasoning_effort):
pass "chat_template_kwargs": {"enable_thinking": false} to turn it off for a request.
The same three requests as the examples below were sent to that server, and the comments show what came back
(pip install openai for the client):
import base64from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")MODEL = "willyb23/NorthCUA-Pro-NVFP4A16" # 1. Ask about a screenshot (thinking off for a short, direct answer)image = base64.b64encode(open("screenshot.png", "rb").read()).decode()resp = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}}, {"type": "text", "text": "What does this dialog ask, and which buttons does it offer?"}, ]}], max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}},)print(resp.choices[0].message.content) # 2. Thinking on (the default): the reasoning comes back in its own fieldresp = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "A train covers 240 km in 3 hours. How far does it go in 5 hours?"}], max_tokens=2048,)print(resp.choices[0].message.reasoning) # vLLM 0.32 returns the thinking as `reasoning`print(resp.choices[0].message.content) # "400 km" # 3. A tool calltools = [{"type": "function", "function": { "name": "get_weather", "description": "Current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]resp = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, max_tokens=1024,)print(resp.choices[0].message.tool_calls[0].function) # get_weather, arguments '{"city": "Paris"}'
For computer use, follow the original card. Run the model with the official
NorthCUA Harness, and keep every GUI coordinate in the
normalized 0–1000 space (normalized_1000). The Harness converts them to pixels.
Faster generation with speculative decoding (DFlash)
Pro generates about twice as fast with
z-lab/Qwen3.5-397B-A17B-DFlash, a 2.6 GB DFlash drafter made
for the base model, Qwen3.5-397B-A17B. The drafter proposes several tokens at a time, and Pro checks them all in one
step, so the output still comes from Pro. Add this to the command above, on every node when running across machines
(vLLM 0.32 supports DFlash natively, without --trust-remote-code):
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-397B-A17B-DFlash", "num_speculative_tokens": 7}'
Measured 2026-11-06 on the four-Spark setup above (vLLM 0.32.0, a local copy of the drafter at revision 2c71e1b).
Greedy decoding, one request at a time; tokens per second after the first token:
Table with columns: Output, Without drafter, z-lab DFlash, 7 tokens, RedHatAI DFlash, 7 tokens| Output | Without drafter | z-lab DFlash, 7 tokens | RedHatAI DFlash, 7 tokens |
|---|
Computer-use step (screenshot + click tool, thinking on), 4 prompts | 24.3–24.6 | 34.9–42.6 | 25.8–30.9 |
| Word problems, thinking on | 24.3–24.5 | 52.9–62.8 | 47.8–48.1 |
| A short Python function | 24.3 | 47.7 | 34.4 |
| Copying a text with small edits, thinking off | 24.6 | 103.0 | 85.3 |
| All 8 prompts | 24.4 | 49.6 (2.03×) | 37.4 (1.53×) |
RedHatAI/Qwen3.5-397B-A17B-speculator.dflash
works too, but its drafts were accepted less often here (24% against 39%). Neither drafter sees the screenshot (vLLM
logs that it uses text-only draft inputs), which is one reason computer-use steps gain least; Pro itself still reads
the image when it checks the drafts.
On Pro only the speed was measured with the drafter. On Mini and Nano, which use the same kind of drafter, rerunning
ScreenSpot-Pro, GSM8K and IFEval with it on showed no significant change (see their cards). Verifying several drafted
tokens at once uses slightly different GPU arithmetic than generating one at a time, so a greedy output can change
where two tokens are nearly tied.
Other inference engines
Table with columns: Engine, Status| Engine | Status |
|---|
| vLLM 0.32.0 | tested (above) |
| SGLang | not tested yet. It reads compressed-tensors checkpoints, but this NVFP4 weight-only layout has not been tried |
| llama.cpp, Ollama | these load GGUF files, not this format |
| TensorRT-LLM | not tested. Its NVFP4 support is built around NVIDIA's ModelOpt checkpoint layout, not this one |
Tests
Run 2026-11-06 on the setup above (vLLM 0.32.0, TP4 over four DGX Sparks), thinking at the template default unless
noted.
Table with columns: Check, Result| Check | Result |
|---|
| Arithmetic probe (17 × 23) | 391 |
| 10 word problems with exact answers (thinking on) | 10/10 |
| 8 short factual questions | 8/8 |
Tool call (get_weather, qwen3_coder parser) | correct name and arguments |
| Click targets on 5 synthetic screenshots (dialog button, toolbar, login form, desktop icon, colour swatch) | 5/5 inside the target box |
| A fact hidden in a 25,251-token prompt (thinking off) | found |
| Single stream, 512 tokens incl. prefill | 24.1 tok/s |
These are smoke tests of this quantized checkpoint, not a re-run of the benchmarks in the NorthCUA report. The BF16
original (794 GB) does not fit on the hardware used, so there is no side-by-side comparison for Pro. The smaller
Mini, a mixture of experts quantized with the same code, showed no significant difference from its BF16 original on
ScreenSpot-Pro, GSM8K and IFEval; the dense Nano, at the same weight error, lost grounding accuracy. That Pro's
quantization costs about as little as Mini's is an inference from their shared design, not a measurement.
The click-target row uses a plain prompt on synthetic screens; it is a smoke test, not a grounding benchmark.
ScreenSpot-Pro
ScreenSpot-Pro: 1,581 screenshots of professional
applications, one click each, which must land inside the target box. The prompt follows the NorthCUA Harness's
coordinate rules and gives the model the Harness's click tool; thinking off, greedy decoding. Run 2026-11-06 on the
setup above, with exactly the requests used for the Mini and Nano cards:
Table with columns: Pro NVFP4 (this), Mini NVFP4, Mini BF16 | Pro NVFP4 (this) | Mini NVFP4 | Mini BF16 |
|---|
| All | 68.56% | 62.05% | 60.78% |
| Text targets | 82.70% | | |
| Icon targets | 45.70% | | |
By application group: Office 80.9%, Creative 68.9%, CAD 67.8%, OS 67.4%, Scientific 65.0%, Dev 63.2%. By target size:
smaller than 16 px 60.2%, 16–32 px 68.4%, 32 px and larger 70.8%. 24 of the 1,581 answers contained no click and are
counted as misses.
Compared question by question with Mini's quantization, Pro is 6.5 points better (95% interval 4.3 to 8.7 points;
exact McNemar p < 0.001). The NorthCUA authors did not publish their exact grounding prompt, so these scores are not
comparable with their report.
Kernel check. A model cut down to two of the 60 layers was run in vLLM 0.32 and compared with a CPU reference
that dequantizes these exact files. vLLM matched the reference to a mean log-probability difference of 0.011 over
each position's top 20 tokens. Deliberately broken references (gate and up swapped, or experts zeroed) differed by
0.35 and 0.11. So vLLM reads these files exactly as written.
Files
model-000NN-of-00122.safetensors keep the original shard boundaries: shards that held routed experts shrink to
about a third of their size, and the others are byte-for-byte BF16 copies. checksums.sha256 covers every file
(sha256sum -c checksums.sha256).
License and attribution
Apache-2.0, as for the original. LICENSE and NOTICE are the original files, unchanged.
Changes made: the routed-expert weights of NorthCUA-Pro were converted to NVFP4 (per-expert tensors replace the
fused experts.gate_up_proj / experts.down_proj tensors), and a quantization_config was added to config.json.
All other weights, the tokenizer, the processor configuration and the chat template are unchanged.
If you use the model, cite the NorthCUA technical report (see the original model card).