Files
The layout matches Cloudflare's Clef releases, so Clef's own code loads it unchanged.
Table with columns: File, Purpose| File | Purpose |
|---|
model.safetensors, config.json, generation_config.json | Qwen3.5-0.8B backbone with the distillation LoRA merged in, including the vision encoder |
joint_head.safetensors, joint_head_config.json | Joint schema head (hidden 1024, width 1024, 2 routing + 2 layers, 16 heads; 69M params) |
joint_schema_model.py | Cloudflare's Clef module, unchanged (Apache-2.0): record encoding, batching, load_release_model, systemone |
tokenizer.json, tokenizer_config.json, chat_template.jinja, processor_config.json | Tokenizer and image processor |
LICENSE | Apache-2.0 |
Usage
Needs torch, transformers ≥ 5.10 (Qwen3.5) and pillow. On CUDA, install causal-conv1d and
flash-linear-attention for the fast Gated DeltaNet kernels; without them transformers falls back to slower
reference code.
import sys import torchfrom huggingface_hub import snapshot_downloadfrom PIL import Image path = snapshot_download("FluidInference/clef-vision-0.8b")sys.path.insert(0, path)from joint_schema_model import collate_records, encode_record, load_release_model model, processor = load_release_model(path, device="cuda") record = { "state": {"task": "Check the camera frame."}, "images": [Image.open("frame.jpg").convert("RGB")], # The student was trained at this image size; use the same on inference. "media_kwargs": {"size": {"shortest_edge": 16384, "longest_edge": 200704}}, "questions": { "person": {"type": "noul", "instructions": "Is there a person in the image?"}, "scene": { "type": "choice", "instructions": "Where was this taken?", "criteria": {"indoor": "Inside a building.", "outdoor": "Outside."}, }, },} encoded = encode_record(processor.tokenizer, record, processor=processor)batch = collate_records([encoded], processor.tokenizer.pad_token_id, torch.device("cuda"))with torch.inference_mode(): logits = model(batch)[0] for question, question_logits in zip(encoded.questions, logits): probabilities = question_logits.float().softmax(-1).tolist() print(question.question_id, dict(zip(question.option_ids, probabilities)))
systemone(model, processor, request) from the same module takes a Jev/SystemOne POST /v1/systemone request
body and returns the response body; see the Clef-flash card for
the full input format.
Verified by loading this exact folder with load_release_model (CPU, fp32): token ids identical to the reference
encoder on 4 image records, 8/8 argmax agreement and max probability difference 4.3e-3 against the shipped
student. On Apple silicon, use the Core ML build; loading this checkpoint onto MPS through device_map crashed
in our tests.
Quality
Held-out test, 1,500 records / 4,094 questions over Oxford-IIIT Pets, Food-101 and COCO val images
(classification, reference matching, tile grids):
Table with columns: Teacher agreement, Gold accuracy, KL to teacher | Teacher agreement | Gold accuracy | KL to teacher |
|---|
| this model (0.92B) | 92.5% | 92.2% | 0.095 |
| teacher Clef-flash 9B (MLX 8-bit) | — | 94.1% | — |
Per task (gold): COCO object presence 98%, Food-101 dish 97%, pets breed 95%, reference-match grids 96–100%,
tile grids 88–93%. The training images cover pets, food and COCO objects; expect lower accuracy on other domains
(e.g. surveillance or industrial camera feeds) until you check it on your own data.
Training
14,246 records (train 5,648 / dev 2,167 / test 6,431, hash-split by image) labelled by Clef-flash running locally
(MLX 8-bit backbone, exact port of the head). Student: LoRA r64 on the language model (merged into this
checkpoint) + a fresh joint schema head; loss = KL(T=2) to the teacher + 0.3 × label-smoothed CE on dataset gold;
530 steps × 16 records on a 24 GB Apple M5 Pro (6.2 h). Vision tower frozen.
Licenses
Apache-2.0 (this model, Qwen3.5-0.8B, Clef and joint_schema_model.py © Cloudflare). Training images:
Oxford-IIIT Pet (CC BY-SA 4.0), Food-101 (research use), COCO 2017 val (CC BY 4.0).