JRamirez-UAB

clef-GPTQ-W4A16-g64-embed-int8

README

License: apache-2.0

Results

Test: 50 sysadmin records in Spanish, 68 questions with known answers: the risk of a shell command (read-only / reversible / destructive, 32 questions), and for log lines the affected component (GPU / disk / network / memory / access / service, 18) and whether someone must act (18). "KL" is the KL divergence of each question's option probabilities (BF16 → this checkpoint), averaged over the 68 questions.

Table with columns: Checkpoint, Size, Correct, Risk, Component, Act?, KL vs BF16 (mean / max), Time per record
Checkpoint	Size	Correct	Risk	Component	Act?	KL vs BF16 (mean / max)	Time per record
Cloudflare/clef BF16 (transformers, partly offloaded to RAM on 48 GB)	54 GB	65/68	32/32	18/18	15/18	—	1.51 s
This repo, transformers (dequantized copy)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.007	—
This repo, vLLM + head (`scripts/clef_vllm.py`)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.009	0.16 s
Vishva007/clef-W4A16-AutoRound (for comparison, same test and code)	19.4 GB	65/68	32/32	18/18	15/18	0.0011 / 0.013	0.16 s

68 questions is a small test: treat differences of ±0.001 in KL as indicative. All measured on one NVIDIA L40S (48 GB).

Quantization recipe

Table with columns: Part, Treatment
Part	Treatment
Linear layers of the language model (attention, MLP)	GPTQ W4A16, group size 64, symmetric (llm-compressor)
`embed_tokens`	INT8, group size 128 (saves ≈1.3 GB; needs the vLLM patch below)
`lm_head`	BF16 (unquantized): Clef's head builds the option vectors from the output embedding matrix
Joint schema head (`joint_head.safetensors`)	BF16, unchanged
Vision tower, GatedDeltaNet `in_proj_a/b`, norms	BF16, unchanged
Calibration

Requirements & install (vLLM with a small patch)

Stock vLLM does not pass quant_config to the embedding layer of qwen3_5-family models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Apply the 4-line patch in patches/qwen3_5-embed-quant.patch (from syv-ai/HyperQwen, Apache-2.0). Tested with vLLM 0.31:

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=autoVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')hf download JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 --local-dir clef-W4A16patch -p1 -d "$VLLM_DIR" < clef-W4A16/patches/qwen3_5-embed-quant.patch

Quick start

bash

cat > record.json <<'EOF'{"state": "Comando propuesto por el agente: rm -rf /var/lib/docker/volumes/*", "questions": {"riesgo": {"type": "choice", "instructions": "¿Qué riesgo tiene ejecutar este comando?",                          "criteria": {"lectura": "Solo lee", "reversible": "Cambia algo que se puede deshacer", "destructivo": "Borra o rompe algo sin vuelta atrás"}}}}EOFvenv/bin/python clef-W4A16/scripts/clef_vllm.py clef-W4A16 record.json

scripts/clef_vllm.py loads the backbone in vLLM's pooling runner (runner="pooling", convert="embed", PoolerConfig(task="token_embed", tok_pooling_type="ALL", use_activation=False)), takes the final hidden states of every token and applies the joint schema head with the BF16 lm_head matrix, exactly as Cloudflare's ClefModel does. For a server, wrap decide() in your HTTP framework of choice. For the original API (/v1/systemone) and multimodal records, see Cloudflare's model card.

Validation of the vLLM path: on the BF16 9B sibling (clef-flash), vLLM's hidden states match transformers' with a per-token cosine of 0.9995; served this way, the BF16 model gives the same answers as Cloudflare's loader (KL 0.00003).

Limitations

Text records only: the vision tower is kept in BF16 but images and video were not tested.
Evaluated on our own small Spanish sysadmin test, not on Cloudflare's Decision Index.
The joint_schema_model.py file is Cloudflare's code, unchanged; it is imported by scripts/clef_vllm.py.

Feedback

Reports are welcome in the Community tab: GPU, vLLM version, your use case, and whether the decisions matched the BF16 model.

License

Apache License 2.0, as Cloudflare/clef (and its base, Qwen3.8-27B). See NOTICE. Unofficial quantization by JRamirez-UAB; not affiliated with Cloudflare, Alibaba Cloud or the Qwen team.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

Cloudflare/clef

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results

Table with columns: Checkpoint, Size, Correct, Risk, Component, Act?, KL vs BF16 (mean / max), Time per record
Checkpoint	Size	Correct	Risk	Component	Act?	KL vs BF16 (mean / max)	Time per record
Cloudflare/clef BF16 (transformers, partly offloaded to RAM on 48 GB)	54 GB	65/68	32/32	18/18	15/18	—	1.51 s
This repo, transformers (dequantized copy)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.007	—
This repo, vLLM + head (`scripts/clef_vllm.py`)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.009	0.16 s
Vishva007/clef-W4A16-AutoRound (for comparison, same test and code)	19.4 GB	65/68	32/32	18/18	15/18	0.0011 / 0.013	0.16 s

68 questions is a small test: treat differences of ±0.001 in KL as indicative. All measured on one NVIDIA L40S (48 GB).

Quantization recipe

Table with columns: Part, Treatment
Part	Treatment
Linear layers of the language model (attention, MLP)	GPTQ W4A16, group size 64, symmetric (llm-compressor)
`embed_tokens`	INT8, group size 128 (saves ≈1.3 GB; needs the vLLM patch below)
`lm_head`	BF16 (unquantized): Clef's head builds the option vectors from the output embedding matrix
Joint schema head (`joint_head.safetensors`)	BF16, unchanged
Vision tower, GatedDeltaNet `in_proj_a/b`, norms	BF16, unchanged
Calibration

Requirements & install (vLLM with a small patch)

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=autoVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')hf download JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 --local-dir clef-W4A16patch -p1 -d "$VLLM_DIR" < clef-W4A16/patches/qwen3_5-embed-quant.patch

Quick start

bash

cat > record.json <<'EOF'{"state": "Comando propuesto por el agente: rm -rf /var/lib/docker/volumes/*", "questions": {"riesgo": {"type": "choice", "instructions": "¿Qué riesgo tiene ejecutar este comando?",                          "criteria": {"lectura": "Solo lee", "reversible": "Cambia algo que se puede deshacer", "destructivo": "Borra o rompe algo sin vuelta atrás"}}}}EOFvenv/bin/python clef-W4A16/scripts/clef_vllm.py clef-W4A16 record.json

Limitations

Text records only: the vision tower is kept in BF16 but images and video were not tested.
Evaluated on our own small Spanish sysadmin test, not on Cloudflare's Decision Index.
The joint_schema_model.py file is Cloudflare's code, unchanged; it is imported by scripts/clef_vllm.py.

Feedback

Reports are welcome in the Community tab: GPU, vLLM version, your use case, and whether the decisions matched the BF16 model.

License

Apache License 2.0, as Cloudflare/clef (and its base, Qwen3.8-27B). See NOTICE. Unofficial quantization by JRamirez-UAB; not affiliated with Cloudflare, Alibaba Cloud or the Qwen team.