Which build to pick
All rows measured the same way: this repo's server with --gdn-backend flashinfer on one RTX 5070 Ti, the same
1,400 records, 64 concurrent requests.
Agree is the share of questions with the same top answer as BF16; KL is the mean KL divergence from BF16 per
question (lower is closer).
Single-request latency is the same for every build (about 32-46 ms for 240-300 token requests, depending on the card), because at batch
size 1 it is set by per-layer overhead, not by arithmetic. Pick by throughput and fidelity: the all-NVFP4 build is
the fastest, this one is the closest to BF16 among the NVFP4 builds, and alpha-x-ai's sits between them and also
handles images. Their checkpoint ran unmodified through this repo's server; the difference in agreement between it
and this build is within run-to-run noise, the difference in KL is not. The FP8 throughput was measured earlier
without --gdn-backend flashinfer. This repo's throughput was measured before its tables were made 8-bit; a
back-to-back test on the machine's second, slower 5070 Ti gave the same throughput for both versions (12.5k tok/s
each), so the row stands. Sizes are the weight files in each repo.
Quickstart: SystemOne API server
hf download kurcontko/clef-flash-NVFP4-Hybrid --local-dir clef-flash-NVFP4-Hybrid docker run --gpus all --ipc=host -p 8000:8000 -v $PWD/clef-flash-NVFP4-Hybrid:/model:ro \ --entrypoint bash vllm/vllm-openai:v0.28.0 -c \ "cp -r /model/vllm_plugin /tmp/p && pip install -q /tmp/p && clef-systemone --model /model --dp 1"
On RTX 50-series add --gdn-backend flashinfer (about 10% faster; experimental, see the
clef-flash-NVFP4 card). The server warms up before /health
reports ready.
curl localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ "model": "clef-flash", "state": "Our checkout started returning errors and orders are blocked.", "questions": { "department": {"type": "choice", "instructions": "Which team should handle the message?", "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}}, "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]}, "outage": {"type": "noul", "instructions": "Is a service down?"}}}'
Request and response bodies match the systemone() function of the original release. The offline Python API is the
same as in the clef-flash-NVFP4 card.
A clean install from this repo with the command above started in about 4 minutes, left 3.05 GiB for KV cache on a
16 GB card, and measured 98.3% agreement with BF16 (KL 0.0026) at 11.8k tok/s with default settings, or 98.6%
(KL 0.0023) at 12.5k tok/s with --gdn-backend flashinfer (both on the slower of the two test cards).
Results
Accuracy and agreement with BF16, per task
1,400 records, 200 per task, from held-out splits. Acc is against gold labels. BF16 is the original release in
transformers 5.10.2; this model was measured in vLLM with real FP8/FP4 kernels (offline run).
Table with columns: Task, BF16 acc, This model acc, Agree, KL| Task | BF16 acc | This model acc | Agree | KL |
|---|
| BANKING77 (77-way intent) | 96.5 | 97.0 | 99.5 | 0.0008 |
| ARC-Challenge | 99.5 | 99.5 | 100.0 | 0.0006 |
| MMLU | 94.5 | 94.0 | 99.5 | 0.0026 |
| HellaSwag |
Across four runs (offline and through the HTTP server, before and after the tables were made 8-bit) this model
measured 98.1-98.7% agreement and KL 0.0023-0.0028; BF16 itself varies by about 0.2 points between runs. Accuracy
differences under 1 point are within noise at 200 records per task.
Where 4-bit quantization hurts: a layer-by-layer scan
For each layer, one group of projections at a time was switched to NVFP4 (weights and activations) while everything
else stayed BF16, and the mean KL divergence from pure BF16 was measured on 190 eval records. The number is the
damage that group alone does in 4 bits.
Table with columns: Layer, Attention type, Attention projections in NVFP4: KL, MLP in NVFP4: KL| Layer | Attention type | Attention projections in NVFP4: KL | MLP in NVFP4: KL |
|---|
| 0 | linear attention | 0.0015 | 0.0004 |
| 1 | linear attention | 0.0008 | 0.0005 |
| 2 | linear attention | 0.0020 | 0.0010 |
| 3 | full attention | 0.0010 | 0.0004 |
Layers 0-16 account for 97% of the summed damage
(0.0300 of 0.0309); layers 17-31 together add 0.0009. The single-group
numbers add up to about what the all-NVFP4 build measures in practice (KL 0.024 on the full set), so the effects are
close to additive. scripts/sensitivity_scan.py reproduces the scan and scripts/sensitivity_scan.json holds every
number.
Quantization details
- Tool: llm-compressor 0.14.0 (compressed-tensors 0.19.0), one
QuantizationModifier with two config groups;
run on a DGX Spark. scripts/quantize_mixed.py --split 17 reproduces it.
- Layers 0-16: FP8 (E4M3) W8A8, per-channel weight scales, per-token dynamic activation scales.
- Layers 17-31: NVFP4 W4A4, 16-element groups with FP8 group scales and one FP32 global scale per tensor;
activation global scales calibrated on 672 records (96 from the train split of each eval task).
- Covered projections in every layer: MLP
gate/up/down_proj, full-attention q/k/v/o_proj, linear-attention
in_proj_qkv / in_proj_z / out_proj.
- Lookup tables in 8-bit: Clef never multiplies by its token-embedding or
lm_head matrices: embeddings are
looked up by token id, and the joint head only reads the lm_head rows of each request's option tokens. Both are
stored as int8 with one scale per row (scripts/shrink_tables.py): relative error 0.9% and 1.1%, mean row cosine
0.99996 and 0.99994. A back-to-back test against the BF16-table version showed the same agreement, latency and
throughput, with KV cache room on a 16 GB card rising from 0.62 to 3.07 GiB.
Files
Table with columns: Path, Purpose| Path | Purpose |
|---|
model.safetensors, config.json, recipe.yaml | Quantized backbone without the two lookup tables (6.4 GB, compressed-tensors mixed-precision) |
clef_head/tables_int8.safetensors | Token embeddings and lm_head as int8 + per-row scale (2.0 GB) |
clef_head/joint_head* | Joint schema head, BF16; weights unchanged from the original release, config has one added key (requires_clef_vllm) |
joint_schema_model.py |
License and attribution
Apache-2.0, following Cloudflare/clef-flash (post-trained from
Qwen/Qwen3.5-9B). All credit for the model goes to its original authors; this
repository only adds the quantization, the vLLM plugin and the server. The idea of mixing FP8 and NVFP4 by layer for
this model was published first by alpha-x-ai; the layout here
comes from our own scan. Not an official Cloudflare release.
