Hardware
- About 25.3 GiB of weights and 30.7 GiB peak GPU memory in our test, so it does not fit one
24 GB card. Tested on two RTX 3090s: 16.4 GiB peak on GPU 0, 14.3 GiB on GPU 1.
- A single GPU with more than about 32 GB should also work (not tested).
What was quantized
Table with columns: Part, Precision| Part | Precision |
|---|
MLP (gate_proj, up_proj, down_proj) in all 64 decoder layers | int4, group size 128, symmetric |
Full attention (q/k/v/o_proj) in the 16 full-attention layers | int4, group size 128, symmetric |
| Gated DeltaNet linear attention (48 layers) | BF16 (the most rounding-sensitive part of a Qwen3.5-style stack; the runtime also reads in_proj_qkv.weight directly) |
Embeddings, lm_head | BF16 (the runtime reads lm_head.weight for its readout) |
| Vision encoder | BF16 |
Typed answer heads (clair_head.safetensors) | BF16, copied unchanged |
mtp.* tensors | BF16, carried over unchanged (not used by the runtime) |
- 256 of 607 linear layers are int4.
- Tool: AutoRound 0.16.1, 100 iterations, 256 calibration
samples × 2048 tokens (text only, not extraction-specific),
auto_round:auto_gptq packing.
About 75 minutes of tuning on one RTX 3090.
Results
A sanity check, not a benchmark: 13 extraction requests with 49 scored fields, run through the
model's own load_release_model + systemone split over two RTX 3090s. One request is the example
from Cloudflare's model card; the other 12 are synthetic (support ticket, lease, shipping notice,
tax computation, relative date, meeting email, bank statement, a request with no answer present,
estimate/forecast, two receipt images, one invoice image). Two passes; the second gave identical
answers to the first on all 13 requests.
No BF16 reference for the 27B: the 53 GB original does not fit 48 GB of VRAM, so the quant is
compared against ground truth only. For comparison, the same test on Clairic Flash (9B):
Table with columns: Clairic Flash BF16, Clairic Flash W4A16, Clairic 27B W4A16 (this repo) | Clairic Flash BF16 | Clairic Flash W4A16 | Clairic 27B W4A16 (this repo) |
|---|
| Fields correct vs. ground truth | 46 / 49 | 47 / 49 | 47 / 49 |
| GPUs | 1× RTX 3090 | 1× RTX 3090 | 2× RTX 3090 |
| Peak GPU memory | 21.2 GiB | 14.2 GiB | 30.7 GiB (16.4 + 14.3) |
| All 13 requests, warm | 62.0 s | 32.6 s | 75.0 s |
The two fields the 27B gets wrong are the same two multi-step arithmetic fields that Flash gets
wrong, both with low confidence:
subtotal + 8.5% tax + shipping (truth 272.40): returned 266.40, confidence 0.18. The single-step
tax (20.40) was right at confidence 0.92.
- due date as "issue date + net 30" (truth 2026-04-14): returned the issue date 2026-03-15,
confidence 0.28.
Extract the parts and do the arithmetic in code. Unlike Flash BF16, the 27B also read the vendor of
an invoice image with no "From:" label (confidence 0.999).
The 27B is slower than Flash here: it is three times the size and runs without CUDA graphs when
split (see Patches). Absolute latencies are slower than Cloudflare's published numbers
because of the fallback kernels on this setup (see Limitations).
Usage
Python 3.11+; install the packages in requirements.txt plus auto-round for the int4 kernels:
pip install -r requirements.txtpip install "auto-round==0.16.1"
Split over two GPUs. Keep embeddings, vision, final norm and lm_head on GPU 0, where the runtime
places the typed heads; put the first 22 decoder layers on GPU 0 and the rest on GPU 1:
import sysfrom huggingface_hub import snapshot_download path = snapshot_download("stuchapin/Clairic-W4A16-AutoRound")sys.path.insert(0, path)from clairic_model import load_release_model, systemone n_layers, 22device_map = { "model.visual": 0, "model.language_model.embed_tokens": 0, "model.language_model.rotary_emb": 0, "model.language_model.norm": 0, "lm_head": 0, **{f"model.language_model.layers.{i}": 0 if i < on_gpu0 else 1 for i in range(n_layers)},}model, processor = load_release_model(path, device="cuda:0", device_map=device_map) response = systemone( model, processor, { "state": ( "Invoice from Northwind Supply. Issued 2026-09-02. " "Items: Pens; Paper. Total: $1080.00." ), "fields": { "vendor": {"type": "text", "instructions": "The invoice vendor."}, "total": {"type": "number", "decimals": 2}, "issued": {"type": "date"}, "items": {"type": "list", "items": "text", "instructions": "The purchased item names, in order."}, }, },)print(response["answers"])
On a single GPU with enough memory, load_release_model(path, device="cuda") is enough.
The full request and response format (field types, images, PDFs, estimates, forecasts, scores) is
documented on the original model card.
Patches
clairic_runtime/math.py and clairic_runtime/native.py are modified from Cloudflare's original
(marked at the top of each file). Everything else in clairic_runtime/ and clairic_model.py is
unchanged. On a single GPU the multi-GPU changes do nothing.
- Ampere support (
math.py). _matmul sends every contiguous BF16 GEMM to DeepGEMM, which
raises Unsupported architecture below compute capability 9.0. DeepGEMM is now used only on
compute capability ≥ 9; older GPUs use the runtime's own Triton matmul fallback.
- Quantized layers (
math.py). install() read gate_proj.weight and q_proj.weight only for
their dtype; quantized linear layers have no .weight. A _dtype_ref() helper falls back to the
embedding weight's dtype.
- Multi-GPU split (
math.py). The runtime replaces each layer's forward with fused kernels,
which also removes accelerate's hook that moves tensors between GPUs, so a split model failed with
Mismatched device index. When the model spans several GPUs, each replaced forward now moves its
tensor inputs to that layer's GPU and makes it the current device first.
- No CUDA graphs when split (
native.py). Per-layer CUDA-graph capture assumes one device and ran
GPU 0 out of memory on a 2 × 24 GB split. It is skipped when the model spans several GPUs; the
decoder runs eagerly.
Limitations
- First run is slow. AutoRound's
tritonv2_zp kernels compile on first use. Triton caches them
on disk, so later runs are fast.
- AutoRound picks its Triton backend here and suggests
gptqmodel>=2.0 for a faster one; not tested.
- Tested without
causal_conv1d, so linear attention used the PyTorch fallback.
- No BF16 comparison for the 27B (it does not fit the test machine); quality is measured against
ground truth only.
- Not tested on Hopper or on a single large GPU. On Ampere the runtime uses its Triton GEMM fallback
instead of DeepGEMM, so answers on Hopper may differ slightly.
- 13 requests and 3 images is a small test. Validate on your own documents and set your own
confidence and
p_null thresholds, as Cloudflare's card recommends.
License
Apache-2.0, the same as the original model. See LICENSE. Model weights, heads and runtime are
© Cloudflare, Inc. and the Qwen team, under that licence; this repository modifies weight
precision and clairic_runtime/math.py and clairic_runtime/native.py only.