Evaluation
Measured 2026-10-14 on a 320-record / 418-question held-out probe set (eval
splits of boolq, SNLI, banking77, CommonsenseQA, ARC-Challenge, go_emotions,
plus synthetic incident and routing records), against Cloudflare's original
bf16 weights. Served with vLLM 0.31.0 as a pooling model
(FlashInferCutlassNvFp4LinearKernel):
Table with columns: bf16 original, DGX Spark GB10, Jetson AGX Thor | bf16 original | DGX Spark GB10 | Jetson AGX Thor |
|---|
| Top-1 accuracy | 93.5% | 93.3% | 92.1% / 91.9% |
| Top-1 agreement with bf16 | — | 97.85% | 95.7% / 96.4% |
| Mean KL from bf16 | — | 0.007 | 1.01 |
| Mean top-1 probability | 0.92 | 0.92 | 0.31 |
Known issue: Jetson AGX Thor (sm_110). On the Thor, the top answer stays
close to bf16 (95.7% and 96.4% agreement, 1.5 to 2.2 points below the Spark's
97.85%), but the probabilities are flattened: the mean top-1 probability is 0.31
instead of 0.92, and the KL from bf16 is about 140 times the Spark's. This holds
under both TRITON_ATTN and FLASH_ATTN attention (the two Thor columns), so the
cause is something else specific to sm_110, not yet identified. On the Thor,
use the top answer only and do not rely on the probabilities.
On the Spark, the median latency per decision is 181 ms for prompts up to 1,024
tokens, about 2.4x faster than bf16.
What changed (Apache-2.0 section 4b)
- The full-attention and MLP Linear layers of the backbone (256 layers) are
quantized to NVFP4: FP4 weights and activations, block size 16, FP8 E4M3
block scales plus FP32 global scales, stored as compressed-tensors
nvfp4-pack-quantized.
- Kept in bf16:
lm_head, embed_tokens, the vision tower, and every
GatedDeltaNet (linear_attn) projection. llm-compressor's own Qwen3.5
NVFP4 recipe documents the fused GatedDeltaNet projections as incompatible
with NVFP4. Norms, conv1d, A_log and dt_bias are not Linear layers and
are unchanged.
- The joint schema head (
joint_head.safetensors) and its code
(joint_schema_model.py) are copied unchanged from the source.
quantization_provenance.json and recipe.yaml record the exact scheme,
calibration and library versions.
Calibration
481 decision records (140,614 tokens), each encoded with Clef's own
encode_record, so activation scales were calibrated on the same token
sequences Clef scores. Records longer than 4,096 tokens were dropped, not
truncated. The set is drawn from public datasets' train splits, pinned by
revision, plus synthetic seeds.
Tooling: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers
5.17.0, torch 2.13.0.
License
Apache-2.0, the same as the source. See LICENSE.