jlancaster

clef-27b-nf4

README

License: apache-2.0

What changed, and what didn't

Changed: the backbone's linear layers are quantised to NF4 with this config. config.json and the weight shards are the only files that differ from Cloudflare's.

python

BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,                   bnb_4bit_compute_dtype=torch.bfloat16, llm_int8_skip_modules=["lm_head"])

Unchanged: the joint schema head (joint_head.safetensors, joint_head_config.json), the loader (joint_schema_model.py), the tokenizer and processor files, and the license are Cloudflare's, copied byte for byte.
lm_head stays in bf16 on purpose. The joint schema head reads the output-embedding weights directly, so quantising lm_head would hand it packed 4-bit data. If you quantise Clef yourself, skip lm_head.

Usage

Tested with torch 2.11.0 (CUDA 13.0), transformers 5.10.2, bitsandbytes 0.50.2, torchvision 0.26.0 and accelerate, on an RTX 4090.

For about a third more speed, install the fast kernels for Qwen 3.5's linear-attention layers too. Without them transformers falls back to plain PyTorch, which works but is slower.

bash

pip install flash-linear-attention   # pure Triton, nothing to compile# causal-conv1d compiles a CUDA extension. There's no prebuilt wheel for torch 2.11 + CUDA 13,# so it builds from source and needs the CUDA 13 toolkit's nvcc (12.x won't do: torch refuses# to build across a major-version mismatch).CUDA_HOME=/usr/local/cuda CAUSAL_CONV1D_FORCE_BUILD=TRUE pip install --no-build-isolation causal-conv1d

python

import sysfrom huggingface_hub import snapshot_download path = snapshot_download("jlancaster/clef-27b-nf4")sys.path.insert(0, path)from joint_schema_model import load_release_model, systemone # The quantisation settings come from config.json; no extra arguments needed.model, processor = load_release_model(path, device="cuda") answer = systemone(model, processor, {    "model": "clef",    "state": {"title": "...", "transcript_excerpt": "..."},    "questions": {"video_type": {        "type": "choice",        "instructions": "Classify this video.",        "criteria": {"sermon": "A complete sermon.", "worship_music": "Music is the main content."},    }},})print(answer["answers"]["video_type"])  # choice, confidence, probabilities

The request and response shapes are Cloudflare's System One format; see their model card for noul, choice and score questions.

How close it is to the bf16 model

I ran the same 152 requests through the full bf16 model and through this quantisation, and compared their answers request by request. Each request was one 13-option choice question about a church video (title, description, transcript excerpts and metadata), about 4–5k tokens.

Table with columns: Compared with bf16 Clef 27B, Result
Compared with bf16 Clef 27B	Result
Same top-level decision (the 7 options we keep vs the 6 we hide)	151 of 152
Same top choice	149 of 152
Difference in the summed "hide" probability	mean 0.011, max 0.092

The one flipped decision was a near tie that moved from 0.51 to 0.49. Against our own blind labels, the two scored within a point of each other: 97% vs 98% keep/hide agreement and Brier 0.034 vs 0.033.

Resources

GPU memory: 18.2 GB after loading. It ran our 4–5k-token requests one at a time within 24 GB. I didn't test batching or longer inputs.

Speed on an RTX 4090, the same 152 requests (4–5k tokens each), one at a time:

Table with columns: Kernels installed, Median, 90th percentile
Kernels installed	Median	90th percentile
`flash-linear-attention` 0.5.2 + `causal-conv1d` 1.7.0	1.70 s	1.79 s
`flash-linear-attention` only	1.78 s	1.88 s
Neither (PyTorch fallback)

What I didn't test

Other tasks: these numbers come from one classification task and 152 requests. They're evidence for that task, not a general quality claim.
Images and video inputs: text-only requests only.
Other GPUs and library versions: only the setup listed above.
score and noul questions: only choice.

License

Apache 2.0, the same as Clef. This is a derivative of Cloudflare/clef; the only change is the quantisation described above.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

jlancaster

Model Tree

Base

Cloudflare/clef

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What changed, and what didn't

Changed: the backbone's linear layers are quantised to NF4 with this config. config.json and the weight shards are the only files that differ from Cloudflare's.

python

BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,                   bnb_4bit_compute_dtype=torch.bfloat16, llm_int8_skip_modules=["lm_head"])

Unchanged: the joint schema head (joint_head.safetensors, joint_head_config.json), the loader (joint_schema_model.py), the tokenizer and processor files, and the license are Cloudflare's, copied byte for byte.
lm_head stays in bf16 on purpose. The joint schema head reads the output-embedding weights directly, so quantising lm_head would hand it packed 4-bit data. If you quantise Clef yourself, skip lm_head.

Usage

Tested with torch 2.11.0 (CUDA 13.0), transformers 5.10.2, bitsandbytes 0.50.2, torchvision 0.26.0 and accelerate, on an RTX 4090.

For about a third more speed, install the fast kernels for Qwen 3.5's linear-attention layers too. Without them transformers falls back to plain PyTorch, which works but is slower.

bash

pip install flash-linear-attention   # pure Triton, nothing to compile# causal-conv1d compiles a CUDA extension. There's no prebuilt wheel for torch 2.11 + CUDA 13,# so it builds from source and needs the CUDA 13 toolkit's nvcc (12.x won't do: torch refuses# to build across a major-version mismatch).CUDA_HOME=/usr/local/cuda CAUSAL_CONV1D_FORCE_BUILD=TRUE pip install --no-build-isolation causal-conv1d

python

import sysfrom huggingface_hub import snapshot_download path = snapshot_download("jlancaster/clef-27b-nf4")sys.path.insert(0, path)from joint_schema_model import load_release_model, systemone # The quantisation settings come from config.json; no extra arguments needed.model, processor = load_release_model(path, device="cuda") answer = systemone(model, processor, {    "model": "clef",    "state": {"title": "...", "transcript_excerpt": "..."},    "questions": {"video_type": {        "type": "choice",        "instructions": "Classify this video.",        "criteria": {"sermon": "A complete sermon.", "worship_music": "Music is the main content."},    }},})print(answer["answers"]["video_type"])  # choice, confidence, probabilities

The request and response shapes are Cloudflare's System One format; see their model card for noul, choice and score questions.

How close it is to the bf16 model

Table with columns: Compared with bf16 Clef 27B, Result
Compared with bf16 Clef 27B	Result
Same top-level decision (the 7 options we keep vs the 6 we hide)	151 of 152
Same top choice	149 of 152
Difference in the summed "hide" probability	mean 0.011, max 0.092

Resources

GPU memory: 18.2 GB after loading. It ran our 4–5k-token requests one at a time within 24 GB. I didn't test batching or longer inputs.

Speed on an RTX 4090, the same 152 requests (4–5k tokens each), one at a time:

Table with columns: Kernels installed, Median, 90th percentile
Kernels installed	Median	90th percentile
`flash-linear-attention` 0.5.2 + `causal-conv1d` 1.7.0	1.70 s	1.79 s
`flash-linear-attention` only	1.78 s	1.88 s
Neither (PyTorch fallback)

What I didn't test

Other tasks: these numbers come from one classification task and 152 requests. They're evidence for that task, not a general quality claim.
Images and video inputs: text-only requests only.
Other GPUs and library versions: only the setup listed above.
score and noul questions: only choice.

License

Apache 2.0, the same as Clef. This is a derivative of Cloudflare/clef; the only change is the quantisation described above.