clef-omni-int4-autoround API & Inference Endpoint

Quick start

Requirements: one GPU with ≥ 40 GB of VRAM (measured peak: 35.87 GiB), CUDA 12.8+, and Python 3.13. CPU-only is not supported by this build.

bash

# 1. environment (clean-room verified; see "Install" below for the exact pins)uv venv venv --python 3.13uv pip install --python venv/bin/python \    torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu130uv pip install --python venv/bin/python \    "transformers==5.20.1" "auto-round==0.16.1" accelerate safetensors huggingface_hub pillow av # 2. download the buildhf download moeshawky/clef-omni-int4-autoround --local-dir ./clef-omni-int4 # 3. run itcat > run.py <<'PY'import syssys.path.insert(0, "./clef-omni-int4")from joint_schema_model import load_release_model, systemone model, processor = load_release_model("./clef-omni-int4", device="cuda")response = systemone(model, processor, {    "model": "clef-omni",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {        "department": {            "type": "choice",            "instructions": "Which team should handle the message?",            "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"},        },        "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},        "outage": {"type": "noul", "instructions": "Is a service down?"},    },})print(response["answers"])PYvenv/bin/python run.py

Measured output of step 3 (deterministic — the same values on every run):

json

{  "department": {    "choice": "technical",    "confidence": 0.7628,    "probabilities": {"billing": 0.2372, "technical": 0.7628},    "type": "choice"  },  "outage": {    "noul": 0.945,    "type": "noul"  },  "urgency": {    "confidence": 0.9336,    "legend": {"0": "Can wait", "1": "This week", "2": "Today"},    "probabilities": {"0": 0.0401, "1": 0.0263, "2": 0.9336},    "score": 1.8936,    "type": "score"  }}

joint_schema_model.py ships inside this repository, so the consumer path needs nothing else from the source release.

Install

auto-round is required because transformers routes the auto-round quant_method to AutoRoundQuantizer, which calls auto_round.inference.convert_model.convert_hf_model. torchvision is required because the Qwen3-Omni processor imports Qwen2VLVideoProcessor.

torch and torchvision must be installed together and from the same index, or pip will pull a second copy of torch and change its version.

Table with columns: Package, Pin, Why this pin
Package	Pin	Why this pin
`torch`	`2.11.0`	tested with the CUDA 13.0 wheel; `sm_120` supported
`torchvision`	`0.26.0`	the build that pairs with `torch` 2.11.0
`transformers`	`5.20.1`	first release with the `auto-round` quantizer path
`auto-round`	`0.16.1`	the version this build was produced with

If python -m venv fails with No module named ensurepip (common on Debian/Ubuntu images that ship without python3-venv), use uv venv as shown above, or install python3-venv.

Serving with vLLM

vLLM 0.32.0 ships a native implementation for this model (ClefOmniForJointSchema). It serves clef-omni's actual decision path — not the backbone as a chat model — and exposes the upstream POST /v1/systemone request/response contract, loading the joint schema head from joint_head.safetensors in this repository.

1. Install vLLM in its own environment

bash

uv venv ~/venvs/vllm --python 3.13uv pip install --python ~/venvs/vllm/bin/python vllm~/venvs/vllm/bin/vllm --version     # -> vLLM 0.32.0

2. Patch one gap in vLLM's quantized path

vLLM 0.32.0's clef-omni pooler reads the LM head's output embedding through VocabParallelEmbedding.forward, bypassing ParallelLMHead.forward. The call therefore lands on the quantization method's embedding(), which no quantization method in that release implements — every request fails with NotImplementedError and takes the engine down.

This build's LM head is BF16 (it sits outside the quantized scope), so gathering rows of the layer's own weight is exactly what the unquantized path would have done. The patch makes that one substitution and nothing else.

vllm_patch.py ships in this repository:

bash

~/venvs/vllm/bin/python ./clef-omni-int4/vllm_patch.py

text

patched:  /home/you/venvs/vllm/lib/python3.13/site-packages/vllm/model_executor/layers/quantization/base_config.pybackup:   /home/you/venvs/vllm/lib/python3.13/site-packages/vllm/model_executor/layers/quantization/base_config.py.bakundo with: cp .../base_config.py.bak .../base_config.py

It is idempotent (re-running it reports already patched) and it writes a backup you can restore.

3. Serve

bash

~/venvs/vllm/bin/vllm serve ./clef-omni-int4 \  --served-model-name clef-omni-int4 \  --tensor-parallel-size 1 \  --dtype bfloat16 \  --max-model-len 8192 \  --gpu-memory-utilization 0.90

Measured on this build:

text

Using MarlinLinearKernel for mixed-precision linearUsing MarlinLinearKernel for AutoGPTQLinearMethodUsing 'MARLIN' WNA16 MoE backend.Available KV cache memory: 65.4 GiBGPU KV cache size: 714,288 tokens, Maximum concurrency for 8,192 tokens per request: 87.19xinit engine (profile, create kv cache, warmup model) took 17.47 s (compilation: 0.16 s)

4. Ask it something

bash

curl -s -X POST http://127.0.0.1:8000/v1/systemone \  -H "Content-Type: application/json" \  -d '{    "model": "clef-omni-int4",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {      "department": {        "type": "choice",        "instructions": "Which team should handle the message?",        "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}      },      "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},      "outage": {"type": "noul", "instructions": "Is a service down?"}    }  }'

The model field must equal the --served-model-name you passed, or the server answers 404.

Measured response:

json

{  "id": "decision-a51195c0d396227d",  "object": "structured_decision",  "model": "clef-omni-int4",  "answers": {    "department": {      "type": "choice",      "choice": "technical",      "probabilities": {"billing": 0.2069, "technical": 0.7931},      "confidence": 0.7931    },    "urgency": {      "type": "score",      "score": 1.8874872494488955,      "legend": {"0": "Can wait", "1": "This week", "2": "Today"},      "probabilities": {"0": 0.0441, "1": 0.0243, "2": 0.9316},      "confidence": 0.9316    },    "outage": {      "type": "noul",      "noul": 0.9437636137008667,      "probabilities": {"yes": 0.9438, "no": 0.0562},      "confidence": 0.9438    }  },  "usage": {"input_tokens": 295, "output_tokens": 0}}

The request contract

Table with columns: Field, Type, Meaning
Field	Type	Meaning
`model`	string	must match `--served-model-name`
`state`	string	the application state to read
`questions`	object	one entry per question, keyed by question id
`questions.<id>.type`	`choice` \| `score` \| `noul`	how the answer is shaped
`questions.<id>.criteria`	object \| list	the allowed options
`questions.<id>.instructions`	string	optional guidance for the model

choice returns the winning option id; score returns a weighted ordinal over the listed criteria; noul returns a yes/no probability. Every answer also carries a probabilities map over the allowed options.

Two behaviours worth knowing

The vLLM path is not bit-reproducible run to run. The same request sent three times to the same running server returned technical = 0.821044, 0.831691, 0.831143. This is the MoE router plus vLLM's autotuned kernels (the server logs

text

Warming up ll_bf16 router GEMM kernels

), not a configuration difference. The transformers path is deterministic and returned byte-identical probabilities on every run. If you need reproducible scores, use the transformers path.

The extra_config keys in config.json are exact module names, not regexes. vLLM resolves the 16-bit MoE-router exemption against the internal layer name by exact equality, so regex keys are silently ignored and the load then fails on the router gate. This repository ships the exact-name form (68 keys: 48 thinker.model.layers.N.mlp.gate, 20 talker.model.layers.N.mlp.gate), which is understood by vLLM, by auto-round and by transformers alike.

Serving on 2× Tesla T4 (constrained, language-only) — measured

The card above assumes one GPU with ≥ 40 GB (measured peak 35.87 GiB on a Blackwell). On two 15 GB T4s (compute capability 7.5, no bf16) the same build serves in language-only mode (--limit-mm-per-prompt zeros out image/video/audio, so the vision/audio towers become StageMissingLayer and are never allocated), with tensor parallelism across both cards and float16 compute:

bash

# 1. install vLLM in its own env (already done above)# 2. patch (idempotent)~/venvs/vllm/bin/python /path/to/clef/vllm_patch.py # 3. serve~/venvs/vllm/bin/vllm serve /path/to/clef \  --served-model-name clef-omni \  --tensor-parallel-size 2 \  --dtype half \  --max-model-len 2048 \  --gpu-memory-utilization 0.90 \  --limit-mm-per-prompt '{"image":0,"video":0,"audio":0}'

Measured on this build (2× T4, 15,360 MiB each, driver 580.178.04, CUDA 13.0):

text

VRAM at idle: ~13,659 MiB per cardRoute registered: POST /v1/systemone

Ask it something (the model field must equal --served-model-name):

bash

curl -s -X POST http://127.0.0.1:8000/v1/systemone \  -H "Content-Type: application/json" \  -d '{    "model": "clef-omni",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {      "department": {"type": "choice", "instructions": "Which team should handle the message?", "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}},      "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},      "outage": {"type": "noul", "instructions": "Is a service down?"}    }  }'

Measured response:

json

{  "id": "decision-ab64218d5b4d8af8",  "object": "structured_decision",  "model": "clef-omni",  "answers": {    "department": {"type": "choice", "choice": "technical", "probabilities": {"billing": 0.2077, "technical": 0.7923}, "confidence": 0.7923},    "urgency": {"type": "score", "score": 1.8905, "legend": {"0": "Can wait", "1": "This week", "2": "Today"}, "probabilities": {"0": 0.0426, "1": 0.0242, "2": 0.9331}, "confidence": 0.9331},    "outage": {"type": "noul", "noul": 0.9460, "probabilities": {"yes": 0.9460, "no": 0.0540}, "confidence": 0.9460}  },  "usage": {"input_tokens": 295, "output_tokens": 0}}

Notes for this path:

--dtype half is required: T4 is compute capability 7.5 and does not support bf16.
Language-only mode disables the vision/audio towers; no image or audio input is accepted on this path. The decision path (thinker + joint schema head) is unchanged.
FA2 is unavailable on cc 7.5; vLLM falls back to a supported attention backend automatically.
The vLLM path is not bit-reproducible run to run (MoE router + autotuned kernels); the transformers path is deterministic.

Troubleshooting

Table with columns: Symptom, Cause, Fix
Symptom	Cause	Fix
`ValueError: There is no module or parameter named 'layers.0.mlp.gate.weight' in Qwen3MoeLLMModel.`	`config.json`'s `extra_config` keys are regexes; vLLM matches by exact equality and ignores them	use a build whose `extra_config` keys are exact module names (this one)
`NotImplementedError` at `base_config.py:64 in embedding`, every request fails with HTTP 500	vLLM 0.32.0 has no `embedding()` for the quantization method	run `vllm_patch.py` (step 2 above)
HTTP 404 from `/v1/systemone`	the `model` field in the body does not match `--served-model-name`	make them identical
`ImportError: Qwen2VLVideoProcessor requires the Torchvision library`	`torchvision` is not installed	install the pinned `torchvision` from the same index as `torch`
`No module named ensurepip` from `python -m venv`	the system Python has no `ensurepip`	use `uv venv`, or install `python3-venv`
`torch.OutOfMemoryError: CUDA out of memory`	a server is still holding the GPU, or the build is on a card with < 40 GB	stop the other server (`nvidia-smi` shows the holder), or lower `--gpu-memory-utilization`
`WARNING backend.py: Better backend is found, please install ... "gptqmodel>=2.0"`	auto-round prefers the gptqmodel backend when it is present	benign; the build was measured on the `tritonv2_zp` backend and works as-is
`[transformers] Unrecognized keys in 'rope_parameters' for 'rope_type'='default'`	upstream `Qwen3-Omni` rope config uses keys the installed `transformers` does not know	benign; it does not affect the decision path

If you are running the server in the background and need to stop it, do not use pkill -f "vllm serve" — that pattern matches the shell issuing the command and will kill your own session. Use the PID file instead.

What was and was not quantized

Measured from the exported safetensors headers — bytes, not estimates.

Table with columns: Component, State, Size
Component	State	Size
`thinker.model.layers` (48-layer MoE decoder)	4-bit weight-only (packed int32 + fp16 scales)	15,065,284,608 B packed + 467,140,608 B scales
`thinker.model.embed_tokens` + norms	BF16	648,441,856 B
`thinker.lm_head`	BF16	622,854,144 B
`thinker.visual` (vision tower)	BF16	1,077,262,816 B
`thinker.audio_tower`	BF16	1,295,854,336 B
`talker.model`	BF16	6,334,582,784 B
`talker.code_predictor`	BF16	283,140,608 B
`talker.hidden_projection` / `text_projection` / `codec_head`	BF16	31,469,568 B
`code2wav` (pre_transformer, decoder, code_embedding, upsample)	BF16	432,033,154 B
`joint_head.safetensors`	unchanged, BF16	218,364,000 B

The talker and code2wav are not used by load_release_model at all. The vision and audio towers were deliberately left in BF16: they are outside the model's decision path, and quantizing them would mean calibrating a modality-specific tower against a text corpus. The MoE router gates are retained at full precision in every layer.

Quantization

Table with columns: Value
	Value
Tool	`auto-round` 0.16.1
Output format	`auto_round` (native safetensors), `packing_format: auto_round:auto_gptq`
Realized bits	4 (read back from the exported `config.json`)
Group size	128
Symmetric	yes (`sym: true`)
Data type	`int`
`quant_method` in `config.json`	`auto-round`
Quantized scope	`thinker.model.layers` (all 48 layers)
Weight-only exception	MoE router `mlp.gate` kept at 16-bit float in all 48 thinker layers
Effective bits/weight	4.13 (4-bit weights + fp16 scales at group 128)
Tuning	`iters=200`, `nsamples=128`, `seqlen=2048`, `batch_size=8`
Calibration corpus	`NeelNanda/pile-10k` (auto-round default), seed 42
Source	`Cloudflare/clef-omni`, source `model.safetensors.index.json` SHA-256 `9bd2cd0d13e8ba283e69634d6be634340915dfa1bc9abcaa5dc445f1bf5cf8fe`
Hardware	1x NVIDIA RTX PRO 6000 Blackwell (97,887 MiB), driver 580.82.07, CUDA 13.0
Software	torch 2.11.0+cu130, transformers 5.20.1
Wall clock	27 min total (1449 s of tuning)

Note: the exported quantization_config does not record the iteration count. iters=200 above is the value this build was run with.

Files

Table with columns: File, Size
File	Size
`model-00001-of-00005.safetensors`	5,370,427,616 B
`model-00002-of-00005.safetensors`	5,370,447,968 B
`model-00003-of-00005.safetensors`	5,362,263,536 B
`model-00004-of-00005.safetensors`	5,368,941,008 B
`model-00005-of-00005.safetensors`	4,794,414,570 B
`model.safetensors.index.json`	6,403,352 B
`joint_head.safetensors`	218,364,000 B
`joint_head_config.json`, `joint_schema_model.py`	119 B, 29,376 B
`vllm_patch.py`	3,778 B
`config.json`, `quantization_config.json`	18,873 B, 6,610 B
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`	11,424,262 B, 907 B, 6,519 B
`processor_config.json`, `preprocessor_config.json`, `generation_config.json`	2,356 B, 679 B, 204 B
`LICENSE`	11,544 B
Total	26.50 GB

joint_head.safetensors SHA-256: d5ee8d7e75c9a503edce94400dda9074fc0f2f4d62729012bf0a827df947ba9c — identical to the source release, so consumers can verify they received the same head.

Verification

Table with columns: Check, Result
Check	Result
Loads via `load_release_model` from this directory	pass (20.4 s, clean environment)
Consumer's own weight-integrity gate	pass — no missing, unexpected, or mismatched keys
Linears actually quantized (not a silent BF16 fallback)	18,634 of 19,014
Peak VRAM, consumer path	35.87 GiB
Joint head SHA-256 vs source	identical
Top-option agreement vs the BF16 source (transformers)	15 / 15
Max abs. probability delta vs BF16 (transformers)	0.1037
Mean abs. probability delta vs BF16 (transformers)	0.0126
Served under vLLM 0.32.0 via `POST /v1/systemone`	pass — all 6 records answered
Top-option agreement vs the BF16 source (vLLM)	15 / 15
Max abs. probability delta vs BF16 (vLLM)	0.0834
Mean abs. probability delta vs BF16 (vLLM)	0.0097
Top-option agreement across BF16, this build, and the 8-bit build, in both stacks	15 / 15
Record set	6 hand-written records, 15 questions, synthetic

The comparison runs the same records through this build and through the unquantized BF16 release, via the same consumer path, and compares the per-option softmax probabilities. The two stacks use different dequantization kernels — auto-round's tritonv2_zp and vLLM's Marlin — so their probabilities differ slightly while the decisions do not. Label vocabularies are aligned before comparing (yes/no vs true/false).

Limitations

The evaluation is a smoke test, not a benchmark. 6 hand-written synthetic records covering choice, score and noul questions. It shows the decision path works and the top option is preserved; it does not establish task accuracy at any meaningful resolution.
One question moved 0.10 in probability (refund_request.sla, a 3-way ordinal) without flipping its top option. Weighted ordinal decisions operating near a boundary are the most likely place this build diverges from the BF16 source.
Calibration corpus mismatch. Tuning used NeelNanda/pile-10k, a general text corpus, while the model also has a vision and an audio tower. Those towers were not quantized, so no audio or image data was needed, but the calibration distribution is not the model's own decision workload.
Weight-only. Activations are BF16. Memory use is dominated by the components that were left unquantized, not by the quantized decoder.
vLLM 0.32.0 needs the patch described above to serve a quantized clef-omni. Unpatched, the model loads but every request fails in the pooler.
The vision and audio towers were never exercised in any measurement here. They are unquantized, so this build does not change them, but no image or audio input was run end to end.

Reproduce

bash

auto-round --model "Cloudflare/clef-omni" \    --bits 4 --group_size 128 --iters 200 \    --nsamples 128 --seqlen 2048 --batch_size 8 \    --format "auto_round" \    --output_dir ./clef-omni-int4-autoround

License

Apache-2.0. Derived from Cloudflare/clef-omni, which is post-trained from Qwen/Qwen3-Omni-30B-A3B-Instruct.

Quick start

Requirements: one GPU with ≥ 40 GB of VRAM (measured peak: 35.87 GiB), CUDA 12.8+, and Python 3.13. CPU-only is not supported by this build.

bash

# 1. environment (clean-room verified; see "Install" below for the exact pins)uv venv venv --python 3.13uv pip install --python venv/bin/python \    torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu130uv pip install --python venv/bin/python \    "transformers==5.20.1" "auto-round==0.16.1" accelerate safetensors huggingface_hub pillow av # 2. download the buildhf download moeshawky/clef-omni-int4-autoround --local-dir ./clef-omni-int4 # 3. run itcat > run.py <<'PY'import syssys.path.insert(0, "./clef-omni-int4")from joint_schema_model import load_release_model, systemone model, processor = load_release_model("./clef-omni-int4", device="cuda")response = systemone(model, processor, {    "model": "clef-omni",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {        "department": {            "type": "choice",            "instructions": "Which team should handle the message?",            "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"},        },        "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},        "outage": {"type": "noul", "instructions": "Is a service down?"},    },})print(response["answers"])PYvenv/bin/python run.py

# 1. environment (clean-room verified; see "Install" below for the exact pins)uv venv venv --python 3.13uv pip install --python venv/bin/python \    torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu130uv pip install --python venv/bin/python \    "transformers==5.20.1" "auto-round==0.16.1" accelerate safetensors huggingface_hub pillow av # 2. download the buildhf download moeshawky/clef-omni-int4-autoround --local-dir ./clef-omni-int4 # 3. run itcat > run.py <<'PY'import syssys.path.insert(0, "./clef-omni-int4")from joint_schema_model import load_release_model, systemone model, processor = load_release_model("./clef-omni-int4", device="cuda")response = systemone(model, processor, {    "model": "clef-omni",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {        "department": {            "type": "choice",            "instructions": "Which team should handle the message?",            "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"},        },        "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},        "outage": {"type": "noul", "instructions": "Is a service down?"},    },})print(response["answers"])PYvenv/bin/python run.py

Measured output of step 3 (deterministic — the same values on every run):

json

{  "department": {    "choice": "technical",    "confidence": 0.7628,    "probabilities": {"billing": 0.2372, "technical": 0.7628},    "type": "choice"  },  "outage": {    "noul": 0.945,    "type": "noul"  },  "urgency": {    "confidence": 0.9336,    "legend": {"0": "Can wait", "1": "This week", "2": "Today"},    "probabilities": {"0": 0.0401, "1": 0.0263, "2": 0.9336},    "score": 1.8936,    "type": "score"  }}

{  "department": {    "choice": "technical",    "confidence": 0.7628,    "probabilities": {"billing": 0.2372, "technical": 0.7628},    "type": "choice"  },  "outage": {    "noul": 0.945,    "type": "noul"  },  "urgency": {    "confidence": 0.9336,    "legend": {"0": "Can wait", "1": "This week", "2": "Today"},    "probabilities": {"0": 0.0401, "1": 0.0263, "2": 0.9336},    "score": 1.8936,    "type": "score"  }}

joint_schema_model.py ships inside this repository, so the consumer path needs nothing else from the source release.

Install

torch and torchvision must be installed together and from the same index, or pip will pull a second copy of torch and change its version.

Table with columns: Package, Pin, Why this pin
Package	Pin	Why this pin
`torch`	`2.11.0`	tested with the CUDA 13.0 wheel; `sm_120` supported
`torchvision`	`0.26.0`	the build that pairs with `torch` 2.11.0
`transformers`	`5.20.1`	first release with the `auto-round` quantizer path
`auto-round`	`0.16.1`	the version this build was produced with

If python -m venv fails with No module named ensurepip (common on Debian/Ubuntu images that ship without python3-venv), use uv venv as shown above, or install python3-venv.

Serving with vLLM

1. Install vLLM in its own environment

bash

uv venv ~/venvs/vllm --python 3.13uv pip install --python ~/venvs/vllm/bin/python vllm~/venvs/vllm/bin/vllm --version     # -> vLLM 0.32.0

uv venv ~/venvs/vllm --python 3.13uv pip install --python ~/venvs/vllm/bin/python vllm~/venvs/vllm/bin/vllm --version     # -> vLLM 0.32.0

2. Patch one gap in vLLM's quantized path

vllm_patch.py ships in this repository:

bash

~/venvs/vllm/bin/python ./clef-omni-int4/vllm_patch.py

text

patched:  /home/you/venvs/vllm/lib/python3.13/site-packages/vllm/model_executor/layers/quantization/base_config.pybackup:   /home/you/venvs/vllm/lib/python3.13/site-packages/vllm/model_executor/layers/quantization/base_config.py.bakundo with: cp .../base_config.py.bak .../base_config.py

patched:  /home/you/venvs/vllm/lib/python3.13/site-packages/vllm/model_executor/layers/quantization/base_config.pybackup:   /home/you/venvs/vllm/lib/python3.13/site-packages/vllm/model_executor/layers/quantization/base_config.py.bakundo with: cp .../base_config.py.bak .../base_config.py

It is idempotent (re-running it reports already patched) and it writes a backup you can restore.

3. Serve

bash

~/venvs/vllm/bin/vllm serve ./clef-omni-int4 \  --served-model-name clef-omni-int4 \  --tensor-parallel-size 1 \  --dtype bfloat16 \  --max-model-len 8192 \  --gpu-memory-utilization 0.90

~/venvs/vllm/bin/vllm serve ./clef-omni-int4 \  --served-model-name clef-omni-int4 \  --tensor-parallel-size 1 \  --dtype bfloat16 \  --max-model-len 8192 \  --gpu-memory-utilization 0.90

Measured on this build:

text

Using MarlinLinearKernel for mixed-precision linearUsing MarlinLinearKernel for AutoGPTQLinearMethodUsing 'MARLIN' WNA16 MoE backend.Available KV cache memory: 65.4 GiBGPU KV cache size: 714,288 tokens, Maximum concurrency for 8,192 tokens per request: 87.19xinit engine (profile, create kv cache, warmup model) took 17.47 s (compilation: 0.16 s)

Using MarlinLinearKernel for mixed-precision linearUsing MarlinLinearKernel for AutoGPTQLinearMethodUsing 'MARLIN' WNA16 MoE backend.Available KV cache memory: 65.4 GiBGPU KV cache size: 714,288 tokens, Maximum concurrency for 8,192 tokens per request: 87.19xinit engine (profile, create kv cache, warmup model) took 17.47 s (compilation: 0.16 s)

4. Ask it something

bash

curl -s -X POST http://127.0.0.1:8000/v1/systemone \  -H "Content-Type: application/json" \  -d '{    "model": "clef-omni-int4",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {      "department": {        "type": "choice",        "instructions": "Which team should handle the message?",        "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}      },      "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},      "outage": {"type": "noul", "instructions": "Is a service down?"}    }  }'

curl -s -X POST http://127.0.0.1:8000/v1/systemone \  -H "Content-Type: application/json" \  -d '{    "model": "clef-omni-int4",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {      "department": {        "type": "choice",        "instructions": "Which team should handle the message?",        "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}      },      "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},      "outage": {"type": "noul", "instructions": "Is a service down?"}    }  }'

The model field must equal the --served-model-name you passed, or the server answers 404.

Measured response:

json

{  "id": "decision-a51195c0d396227d",  "object": "structured_decision",  "model": "clef-omni-int4",  "answers": {    "department": {      "type": "choice",      "choice": "technical",      "probabilities": {"billing": 0.2069, "technical": 0.7931},      "confidence": 0.7931    },    "urgency": {      "type": "score",      "score": 1.8874872494488955,      "legend": {"0": "Can wait", "1": "This week", "2": "Today"},      "probabilities": {"0": 0.0441, "1": 0.0243, "2": 0.9316},      "confidence": 0.9316    },    "outage": {      "type": "noul",      "noul": 0.9437636137008667,      "probabilities": {"yes": 0.9438, "no": 0.0562},      "confidence": 0.9438    }  },  "usage": {"input_tokens": 295, "output_tokens": 0}}

{  "id": "decision-a51195c0d396227d",  "object": "structured_decision",  "model": "clef-omni-int4",  "answers": {    "department": {      "type": "choice",      "choice": "technical",      "probabilities": {"billing": 0.2069, "technical": 0.7931},      "confidence": 0.7931    },    "urgency": {      "type": "score",      "score": 1.8874872494488955,      "legend": {"0": "Can wait", "1": "This week", "2": "Today"},      "probabilities": {"0": 0.0441, "1": 0.0243, "2": 0.9316},      "confidence": 0.9316    },    "outage": {      "type": "noul",      "noul": 0.9437636137008667,      "probabilities": {"yes": 0.9438, "no": 0.0562},      "confidence": 0.9438    }  },  "usage": {"input_tokens": 295, "output_tokens": 0}}

The request contract

Table with columns: Field, Type, Meaning
Field	Type	Meaning
`model`	string	must match `--served-model-name`
`state`	string	the application state to read
`questions`	object	one entry per question, keyed by question id
`questions.<id>.type`	`choice` \| `score` \| `noul`	how the answer is shaped
`questions.<id>.criteria`	object \| list	the allowed options
`questions.<id>.instructions`	string	optional guidance for the model

Two behaviours worth knowing

text

Warming up ll_bf16 router GEMM kernels

), not a configuration difference. The transformers path is deterministic and returned byte-identical probabilities on every run. If you need reproducible scores, use the transformers path.

Serving on 2× Tesla T4 (constrained, language-only) — measured

bash

# 1. install vLLM in its own env (already done above)# 2. patch (idempotent)~/venvs/vllm/bin/python /path/to/clef/vllm_patch.py # 3. serve~/venvs/vllm/bin/vllm serve /path/to/clef \  --served-model-name clef-omni \  --tensor-parallel-size 2 \  --dtype half \  --max-model-len 2048 \  --gpu-memory-utilization 0.90 \  --limit-mm-per-prompt '{"image":0,"video":0,"audio":0}'

# 1. install vLLM in its own env (already done above)# 2. patch (idempotent)~/venvs/vllm/bin/python /path/to/clef/vllm_patch.py # 3. serve~/venvs/vllm/bin/vllm serve /path/to/clef \  --served-model-name clef-omni \  --tensor-parallel-size 2 \  --dtype half \  --max-model-len 2048 \  --gpu-memory-utilization 0.90 \  --limit-mm-per-prompt '{"image":0,"video":0,"audio":0}'

Measured on this build (2× T4, 15,360 MiB each, driver 580.178.04, CUDA 13.0):

text

VRAM at idle: ~13,659 MiB per cardRoute registered: POST /v1/systemone

Ask it something (the model field must equal --served-model-name):

bash

curl -s -X POST http://127.0.0.1:8000/v1/systemone \  -H "Content-Type: application/json" \  -d '{    "model": "clef-omni",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {      "department": {"type": "choice", "instructions": "Which team should handle the message?", "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}},      "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},      "outage": {"type": "noul", "instructions": "Is a service down?"}    }  }'

curl -s -X POST http://127.0.0.1:8000/v1/systemone \  -H "Content-Type: application/json" \  -d '{    "model": "clef-omni",    "state": "Our checkout started returning errors and orders are blocked.",    "questions": {      "department": {"type": "choice", "instructions": "Which team should handle the message?", "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}},      "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},      "outage": {"type": "noul", "instructions": "Is a service down?"}    }  }'

Measured response:

json

{  "id": "decision-ab64218d5b4d8af8",  "object": "structured_decision",  "model": "clef-omni",  "answers": {    "department": {"type": "choice", "choice": "technical", "probabilities": {"billing": 0.2077, "technical": 0.7923}, "confidence": 0.7923},    "urgency": {"type": "score", "score": 1.8905, "legend": {"0": "Can wait", "1": "This week", "2": "Today"}, "probabilities": {"0": 0.0426, "1": 0.0242, "2": 0.9331}, "confidence": 0.9331},    "outage": {"type": "noul", "noul": 0.9460, "probabilities": {"yes": 0.9460, "no": 0.0540}, "confidence": 0.9460}  },  "usage": {"input_tokens": 295, "output_tokens": 0}}

{  "id": "decision-ab64218d5b4d8af8",  "object": "structured_decision",  "model": "clef-omni",  "answers": {    "department": {"type": "choice", "choice": "technical", "probabilities": {"billing": 0.2077, "technical": 0.7923}, "confidence": 0.7923},    "urgency": {"type": "score", "score": 1.8905, "legend": {"0": "Can wait", "1": "This week", "2": "Today"}, "probabilities": {"0": 0.0426, "1": 0.0242, "2": 0.9331}, "confidence": 0.9331},    "outage": {"type": "noul", "noul": 0.9460, "probabilities": {"yes": 0.9460, "no": 0.0540}, "confidence": 0.9460}  },  "usage": {"input_tokens": 295, "output_tokens": 0}}

Notes for this path:

--dtype half is required: T4 is compute capability 7.5 and does not support bf16.
Language-only mode disables the vision/audio towers; no image or audio input is accepted on this path. The decision path (thinker + joint schema head) is unchanged.
FA2 is unavailable on cc 7.5; vLLM falls back to a supported attention backend automatically.
The vLLM path is not bit-reproducible run to run (MoE router + autotuned kernels); the transformers path is deterministic.

Troubleshooting

Table with columns: Symptom, Cause, Fix
Symptom	Cause	Fix
`ValueError: There is no module or parameter named 'layers.0.mlp.gate.weight' in Qwen3MoeLLMModel.`	`config.json`'s `extra_config` keys are regexes; vLLM matches by exact equality and ignores them	use a build whose `extra_config` keys are exact module names (this one)
`NotImplementedError` at `base_config.py:64 in embedding`, every request fails with HTTP 500	vLLM 0.32.0 has no `embedding()` for the quantization method	run `vllm_patch.py` (step 2 above)
HTTP 404 from `/v1/systemone`	the `model` field in the body does not match `--served-model-name`	make them identical
`ImportError: Qwen2VLVideoProcessor requires the Torchvision library`	`torchvision` is not installed	install the pinned `torchvision` from the same index as `torch`
`No module named ensurepip` from `python -m venv`	the system Python has no `ensurepip`	use `uv venv`, or install `python3-venv`
`torch.OutOfMemoryError: CUDA out of memory`	a server is still holding the GPU, or the build is on a card with < 40 GB	stop the other server (`nvidia-smi` shows the holder), or lower `--gpu-memory-utilization`
`WARNING backend.py: Better backend is found, please install ... "gptqmodel>=2.0"`	auto-round prefers the gptqmodel backend when it is present	benign; the build was measured on the `tritonv2_zp` backend and works as-is
`[transformers] Unrecognized keys in 'rope_parameters' for 'rope_type'='default'`	upstream `Qwen3-Omni` rope config uses keys the installed `transformers` does not know	benign; it does not affect the decision path

What was and was not quantized

Measured from the exported safetensors headers — bytes, not estimates.

Table with columns: Component, State, Size
Component	State	Size
`thinker.model.layers` (48-layer MoE decoder)	4-bit weight-only (packed int32 + fp16 scales)	15,065,284,608 B packed + 467,140,608 B scales
`thinker.model.embed_tokens` + norms	BF16	648,441,856 B
`thinker.lm_head`	BF16	622,854,144 B
`thinker.visual` (vision tower)	BF16	1,077,262,816 B
`thinker.audio_tower`	BF16	1,295,854,336 B
`talker.model`	BF16	6,334,582,784 B
`talker.code_predictor`	BF16	283,140,608 B
`talker.hidden_projection` / `text_projection` / `codec_head`	BF16	31,469,568 B
`code2wav` (pre_transformer, decoder, code_embedding, upsample)	BF16	432,033,154 B
`joint_head.safetensors`	unchanged, BF16	218,364,000 B

Quantization

Table with columns: Value
	Value
Tool	`auto-round` 0.16.1
Output format	`auto_round` (native safetensors), `packing_format: auto_round:auto_gptq`
Realized bits	4 (read back from the exported `config.json`)
Group size	128
Symmetric	yes (`sym: true`)
Data type	`int`
`quant_method` in `config.json`	`auto-round`
Quantized scope	`thinker.model.layers` (all 48 layers)
Weight-only exception	MoE router `mlp.gate` kept at 16-bit float in all 48 thinker layers
Effective bits/weight	4.13 (4-bit weights + fp16 scales at group 128)
Tuning	`iters=200`, `nsamples=128`, `seqlen=2048`, `batch_size=8`
Calibration corpus	`NeelNanda/pile-10k` (auto-round default), seed 42
Source	`Cloudflare/clef-omni`, source `model.safetensors.index.json` SHA-256 `9bd2cd0d13e8ba283e69634d6be634340915dfa1bc9abcaa5dc445f1bf5cf8fe`
Hardware	1x NVIDIA RTX PRO 6000 Blackwell (97,887 MiB), driver 580.82.07, CUDA 13.0
Software	torch 2.11.0+cu130, transformers 5.20.1
Wall clock	27 min total (1449 s of tuning)

Note: the exported quantization_config does not record the iteration count. iters=200 above is the value this build was run with.

Files

Table with columns: File, Size
File	Size
`model-00001-of-00005.safetensors`	5,370,427,616 B
`model-00002-of-00005.safetensors`	5,370,447,968 B
`model-00003-of-00005.safetensors`	5,362,263,536 B
`model-00004-of-00005.safetensors`	5,368,941,008 B
`model-00005-of-00005.safetensors`	4,794,414,570 B
`model.safetensors.index.json`	6,403,352 B
`joint_head.safetensors`	218,364,000 B
`joint_head_config.json`, `joint_schema_model.py`	119 B, 29,376 B
`vllm_patch.py`	3,778 B
`config.json`, `quantization_config.json`	18,873 B, 6,610 B
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`	11,424,262 B, 907 B, 6,519 B
`processor_config.json`, `preprocessor_config.json`, `generation_config.json`	2,356 B, 679 B, 204 B
`LICENSE`	11,544 B
Total	26.50 GB

joint_head.safetensors SHA-256: d5ee8d7e75c9a503edce94400dda9074fc0f2f4d62729012bf0a827df947ba9c — identical to the source release, so consumers can verify they received the same head.

Verification

Table with columns: Check, Result
Check	Result
Loads via `load_release_model` from this directory	pass (20.4 s, clean environment)
Consumer's own weight-integrity gate	pass — no missing, unexpected, or mismatched keys
Linears actually quantized (not a silent BF16 fallback)	18,634 of 19,014
Peak VRAM, consumer path	35.87 GiB
Joint head SHA-256 vs source	identical
Top-option agreement vs the BF16 source (transformers)	15 / 15
Max abs. probability delta vs BF16 (transformers)	0.1037
Mean abs. probability delta vs BF16 (transformers)	0.0126
Served under vLLM 0.32.0 via `POST /v1/systemone`	pass — all 6 records answered
Top-option agreement vs the BF16 source (vLLM)	15 / 15
Max abs. probability delta vs BF16 (vLLM)	0.0834
Mean abs. probability delta vs BF16 (vLLM)	0.0097
Top-option agreement across BF16, this build, and the 8-bit build, in both stacks	15 / 15
Record set	6 hand-written records, 15 questions, synthetic

Limitations

The evaluation is a smoke test, not a benchmark. 6 hand-written synthetic records covering choice, score and noul questions. It shows the decision path works and the top option is preserved; it does not establish task accuracy at any meaningful resolution.
One question moved 0.10 in probability (refund_request.sla, a 3-way ordinal) without flipping its top option. Weighted ordinal decisions operating near a boundary are the most likely place this build diverges from the BF16 source.
Calibration corpus mismatch. Tuning used NeelNanda/pile-10k, a general text corpus, while the model also has a vision and an audio tower. Those towers were not quantized, so no audio or image data was needed, but the calibration distribution is not the model's own decision workload.
Weight-only. Activations are BF16. Memory use is dominated by the components that were left unquantized, not by the quantized decoder.
vLLM 0.32.0 needs the patch described above to serve a quantized clef-omni. Unpatched, the model loads but every request fails in the pooler.
The vision and audio towers were never exercised in any measurement here. They are unquantized, so this build does not change them, but no image or audio input was run end to end.

Reproduce

bash

auto-round --model "Cloudflare/clef-omni" \    --bits 4 --group_size 128 --iters 200 \    --nsamples 128 --seqlen 2048 --batch_size 8 \    --format "auto_round" \    --output_dir ./clef-omni-int4-autoround

auto-round --model "Cloudflare/clef-omni" \    --bits 4 --group_size 128 --iters 200 \    --nsamples 128 --seqlen 2048 --batch_size 8 \    --format "auto_round" \    --output_dir ./clef-omni-int4-autoround

License

Apache-2.0. Derived from Cloudflare/clef-omni, which is post-trained from Qwen/Qwen3-Omni-30B-A3B-Instruct.

clef-omni-int4-autoround

README

Quick start

Install

Serving with vLLM

1. Install vLLM in its own environment

2. Patch one gap in vLLM's quantized path

3. Serve

4. Ask it something

The request contract

Two behaviours worth knowing

Serving on 2× Tesla T4 (constrained, language-only) — measured

Troubleshooting

What was and was not quantized

Quantization

Files

Verification

Limitations

Reproduce

License

Explore FriendliAI today

README

Quick start

Install

Serving with vLLM

1. Install vLLM in its own environment

2. Patch one gap in vLLM's quantized path

3. Serve

4. Ask it something

The request contract

Two behaviours worth knowing

Serving on 2× Tesla T4 (constrained, language-only) — measured

Troubleshooting

What was and was not quantized

Quantization

Files

Verification

Limitations

Reproduce

License