Evaluation
Measured 2026-10-14 on a 320-record / 418-question held-out probe set (eval
splits of boolq, SNLI, banking77, CommonsenseQA, ARC-Challenge, go_emotions,
plus synthetic incident and routing records), against Cloudflare's original
bf16 weights:
Table with columns: bf16 original, this build | bf16 original | this build |
|---|
| Top-1 accuracy | 93.5% | 93.8% |
| Top-1 agreement with bf16 | — | 98.8% |
| Mean / max KL from bf16 | — | 0.003 / 0.22 |
| Failed requests | 0 | 0 |
Served on a Jetson AGX Orin 64GB (sm_87) with vLLM 0.31.0 as a pooling model:
MarlinLinearKernel for the INT4 layers, 14.84 GiB of weights, about 21.7 GiB
for the whole server, median 505 ms per decision for prompts up to 1,024 tokens.
The joint schema head runs in bf16 on the backbone's final hidden states.
This build needs model.safetensors.index.json to load in vLLM: without it, vLLM
also loads joint_head.safetensors and refuses to start.
What changed (Apache-2.0 section 4b)
- The backbone's Linear layers are quantized to INT4, symmetric, group size
64, with GPTQ (
actorder: static), and stored as compressed-tensors
pack-quantized.
- Kept in bf16:
lm_head, embed_tokens, the vision tower, and the
GatedDeltaNet in_proj_a / in_proj_b projections. Norms, conv1d,
A_log and dt_bias are not Linear layers and are unchanged.
- The joint schema head (
joint_head.safetensors) and its code
(joint_schema_model.py) are copied unchanged from the source.
quantization_provenance.json and recipe.yaml record the exact scheme,
calibration and library versions.
Calibration
481 decision records (140,614 tokens), each encoded with Clef's own
encode_record, so the backbone was calibrated on the same token sequences
Clef scores. Records longer than 4,096 tokens were dropped, not truncated.
The set is drawn from public datasets' train splits, pinned by revision, plus
synthetic seeds.
Tooling: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers
5.17.0, torch 2.13.0.
License
Apache-2.0, the same as the source. See LICENSE.