GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
kurcontko
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
kurcontko
Model Tree
Cloudflare/clef-flash
Input Modalities
Output Modalities
Supported Functionality
bash
hf download kurcontko/clef-flash-NVFP4 --local-dir clef-flash-NVFP4 docker run --gpus all --ipc=host -p 8000:8000 -v $PWD/clef-flash-NVFP4:/model:ro \ --entrypoint bash vllm/vllm-openai:v0.28.0 -c \ "cp -r /model/vllm_plugin /tmp/p && pip install -q /tmp/p && clef-systemone --model /model --dp 1"--dp N runs one engine per GPU behind the one port (recommended over tensor parallelism on PCIe GPUs).
The server warms up its kernels before /health reports ready (about 3-4 minutes from a cold container).
bash
curl localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ "model": "clef-flash", "state": "Our checkout started returning errors and orders are blocked.", "questions": { "department": {"type": "choice", "instructions": "Which team should handle the message?", "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}}, "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]}, "outage": {"type": "noul", "instructions": "Is a service down?"}}}'The request and response bodies are the same as the systemone() function in the original release:
answers keyed by question ID (choice with confidence and probabilities, score with the expected score
and legend, noul with the probability of true) and usage. Also served: GET /v1/models, GET /health.
python
# pip install ./clef-flash-NVFP4/vllm_plugin (in an environment with vllm==0.28.*)import sysfrom transformers import AutoTokenizerfrom vllm import LLM, PoolingParamsfrom clef_vllm import ARCHITECTURE, TASK, schema_layout path = "./clef-flash-NVFP4"sys.path.insert(0, path)from joint_schema_model import encode_record def main(): # Keep `llm` local: vLLM shuts its engine process down when the LLM object is freed. llm = LLM(model=path, hf_overrides={"architectures": [ARCHITECTURE]}, runner="pooling", enable_prefix_caching=False, language_model_only=True, max_model_len=16384) tokenizer = AutoTokenizer.from_pretrained(path) record = { "state": {"invoice": {"vendor": "Acme", "total": 1250.0, "currency": "USD", "status": "overdue"}}, "questions": { "status": {"type": "choice", "instructions": "What is the invoice status?", "criteria": {"paid": "Invoice is paid.", "overdue": "Invoice is past due.", "draft": "Not sent."}}, "large": {"type": "noul", "instructions": "Is the total above 1000 USD?"}, }, } encoded = encode_record(tokenizer, record) params = PoolingParams(task=TASK, extra_kwargs={"clef_questions": schema_layout(encoded)}) output = llm.encode([{"prompt_token_ids": list(encoded.input_ids)}], pooling_params=[params], pooling_task=TASK)[0] logits, offset = output.outputs.data.float(), 0 # every option logit of every question, in order for question in encoded.questions: n = len(question.option_ids) print(question.question_id, dict(zip(question.option_ids, logits[offset:offset + n].softmax(-1).tolist()))) offset += n if __name__ == "__main__": main()Pass a list of prompts and a matching list of PoolingParams to score many records in one call; vLLM batches them.
1,400 records, 200 per task, from held-out splits: BANKING77 (test), ARC-Challenge (test), MMLU (test),
HellaSwag (validation), ANLI R3 (test), BoolQ (validation), and a multi-question ANLI R1 set (3 questions per
record: choice + true/false + score). Accuracy is against gold labels. Agree is top-1 agreement with the
BF16 model and KL is mean KL(BF16 ‖ quantized) per question. BF16 is the original release run with
transformers 5.10.2; the quantized rows were measured in vLLM with real FP8/FP4 kernels.
| Task | BF16 acc | FP8 acc | FP8 agree | FP8 KL | NVFP4 acc | NVFP4 agree | NVFP4 KL |
|---|---|---|---|---|---|---|---|
| BANKING77 (77-way intent) | 96.5 | 97.0 | 99.5 | 0.0005 | 97.5 | 98.0 | 0.013 |
| ARC-Challenge | 99.5 | 99.5 | 100.0 |
Accuracy differences of under 1 point are within noise at 200 records per task. The BF16 model run through the same
plugin in vLLM agrees with transformers on 99.8% of questions (KL 0.0001), so the plugin itself adds no error.
NVFP4 moves most on hard, near-chance cases (ANLI): treat its confidences on borderline inputs with care.
Same 1,400 records (717k tokens, ~512 per record), RTX 5070 Ti 16 GB GPUs.
| Setup | GPUs | Records/s | Tokens/s | vs BF16 transformers |
|---|---|---|---|---|
BF16, transformers 5.10.2 (original release code, model split over 2 GPUs) | 2 | 6.1 | 3,113 | 1.0x |
| BF16, vLLM + plugin, tensor parallel 2 | 2 | 13.0 | 6,633 | 2.1x |
| FP8, vLLM + plugin | 1 | 19.5 | 9,986 |
Through the HTTP server with 64 concurrent clients: NVFP4 on 1 GPU 39.7 req/s (p50 0.95 s, p95 5.1 s), NVFP4 on 2 GPUs 78.7 req/s; with 8 concurrent clients on 2 GPUs, p50 latency is 132 ms. On PCIe GPUs, tensor parallelism was slower than one engine per GPU (FP8 TP2: 8,605 tok/s; NVFP4 TP2: 10,850 tok/s).
One request at a time over HTTP, end to end, on one RTX 5070 Ti with --gdn-backend flashinfer (median of 100):
| Request | Input tokens | Median | p95 |
|---|---|---|---|
| One true/false question | 147 | 34.1 ms | 35.1 ms |
| One 3-option choice (a Snake move) | 242 | 33.8 ms | 35.8 ms |
| 3 questions (the example above) | 300 | 36.9 ms | 38.7 ms |
Two settings in the bundled server account for this; with vLLM 0.28's defaults the same requests took about 70 ms:
max_num_seqs tokens, which fits
chat decoding but leaves every prefill-only Clef request on the slow eager path. The server captures graphs for steps
of up to 2,048 tokens (--max-cudagraph-tokens, about 0.6 GiB of GPU memory): 70 ms to 37 ms.--gdn-backend flashinfer on RTX 50-series. vLLM 0.28 enables FlashInfer's kernel for Qwen3.5's linear attention
only on SM 9.0 / 10.x and falls back to Triton on SM 12.0. Forcing it works on an RTX 5070 Ti with unchanged outputs
(94.4% agreement with BF16): 37 ms to 34 ms, and one-GPU throughput rises from 20.4k to 22.4k tok/s. Experimental and
opt-in; tested on this card only.Responses carry a Server-Timing header (encode, engine, answer). For the lowest latency run one engine (--dp 1);
--dp 2 adds a process hop of a few milliseconds.
QuantizationModifier(targets="Linear", scheme="NVFP4").lm_head (the joint head reads its rows as lexical option embeddings, so it is never quantized),
the vision tower (model.visual.*), the Gated DeltaNet gate projections linear_attn.in_proj_a / in_proj_b
(32 outputs each), embeddings, norms, and the joint schema head (clef_head/).linear_attn.in_proj_qkv and in_proj_z into one GEMM, which needs a
shared NVFP4 weight global scale. llm-compressor 0.14 only shares scales for q/k/v and gate/up, so
adds to its fused groups. Without this, vLLM applies the
larger scale to both halves and the qkv weights come out scaled by about 0.67 (we measured 92.9% instead of 94.3%
agreement). Anyone re-quantizing Qwen3.5-family models to NVFP4 for vLLM should do the same.vllm_plugin/ (pip install-able, package clef-vllm) registers ClefFlashForDecision, a subclass of vLLM's
Qwen3_5ForConditionalGeneration that runs as a pooling model. Its pooler collects each request's final hidden
states across chunked prefill and runs the original JointSchemaHead (from joint_schema_model.py) on the full
prompt. Each request carries its schema layout (question and option token spans from encode_record) in
PoolingParams.extra_kwargs["clef_questions"]; the output is one flat vector of every option logit. Prefix caching
must be off. Under tensor parallelism the vocab-sharded lm_head is gathered once on rank 0 for the head's lookups.
| Path | Purpose |
|---|---|
model.safetensors, config.json, recipe.yaml | Quantized backbone (8.3 GB, compressed-tensors format) |
clef_head/ | Joint schema head, BF16, unchanged from the original release (in a subfolder so vLLM does not load it as backbone weights) |
joint_schema_model.py | Original release code: encode_record, systemone_answer, the head module (unchanged) |
tokenizer*.json, chat_template.jinja, , |
See also kurcontko/clef-flash-FP8-Dynamic (near-lossless, about half the speed).
Apache-2.0, following Cloudflare/clef-flash (itself post-trained from Qwen/Qwen3.5-9B). All credit for the model goes to its original authors; this repository only adds the quantization, the vLLM plugin and the server. Not an official Cloudflare release.
| 0.0006 |
| 99.0 |
| 99.5 |
| 0.009 |
| MMLU | 94.5 | 95.0 | 99.5 | 0.0035 | 93.5 | 95.5 | 0.026 |
| HellaSwag | 99.0 | 99.0 | 100.0 | 0.0003 | 99.0 | 100.0 | 0.002 |
| ANLI R3 | 50.0 | 49.5 | 97.5 | 0.0028 | 49.0 | 86.5 | 0.041 |
| BoolQ | 87.5 | 88.5 | 99.0 | 0.0026 | 87.5 | 98.0 | 0.018 |
| Multi-question ANLI R1 | 73.0 | 74.0 | 98.0 | 0.0023 | 71.5 | 90.3 | 0.034 |
| All | 84.1 | 84.6 | 98.8 | 0.0019 | 83.6 | 94.3 | 0.024 |
| 3.2x |
| NVFP4, vLLM + plugin | 1 | 39.8 | 20,359 | 6.5x |
FP8, one engine per GPU (--dp 2) | 2 | ~39 | ~20,000 | 6.4x |
NVFP4, one engine per GPU (--dp 2) | 2 | ~81 | ~41,000 | 13x |
scripts/quantize.py("in_proj_qkv", "in_proj_z")recipe.yaml is the exact recipe llm-compressor saved.processor_config.jsongeneration_config.json| Tokenizer and processor (unchanged) |
vllm_plugin/ | vLLM plugin and clef-systemone server |
scripts/ | build_data.py (calibration/eval sets) and quantize.py (how this checkpoint was made) |