Quantization
- Tool: llm-compressor 0.14.0.1 (compressed-tensors 0.19.0, transformers 5.17.0).
- Scheme: W4A16 GPTQ, stored in the compressed-tensors
pack-quantized format.
- Weights are int4 and activations stay 16-bit.
- Symmetric, group size 128, static activation ordering, dampening 0.01, block size 128.
- What is quantized: 200 linear layers.
- all 96 MLP projections;
- the 32 full-attention q/k/v/o projections;
- the 72 Gated DeltaNet (linear-attention) projections
in_proj_qkv, in_proj_z and out_proj.
- What is kept unquantized: 538 tensors, all byte-identical to the bf16 student, in the source's dtypes.
- In bf16: the embeddings and the tied LM head,
in_proj_a / in_proj_b (32 outputs each), conv1d, dt_bias, the RMSNorm weights, the vision tower and the MTP head (mtp.*, 15 tensors).
- In F32, as in the source: the 24
linear_attn.A_log tensors and the 24 Gated DeltaNet norm weights (linear_attn.norm.weight).
- Corrections after the build: llm-compressor 0.14.0.1's save changed two things, and both were put back from the source and checked tensor by tensor.
- Its offset-norm conversion changed 81 RMSNorm weights by bf16 rounding, by up to 0.016.
- It stored the 48 F32 tensors as BF16.
- Calibration data: 512 chats from the student's own training set, stratified by task, seed 0.
- Each chat is a bibr extraction prompt plus the teacher's answer, rendered with this chat template with thinking off.
- They come from 158 CC BY papers from PubMed Central, are at most 4,096 tokens each, and total 789,503 tokens.
- None of the evaluation papers is among them (checked by DOI and title).
Evaluation
The scores come from bibr's extraction harness of 9 Oct 2026: bibr 9621739, v5 scorer, 252 dev papers (gate192 + ci60), OCR replayed from cache.
- Comparisons are paired, with a paper bootstrap of 10,000 reps and a non-inferiority margin of −4 pp.
- One paper is 0.4 pp of the 252.
Table with columns: model, serving, passes / 252, vs bf16 4B, same box: Δ pp [95% CI], vs reference run: Δ pp [95% CI]| model | serving | passes / 252 | vs bf16 4B, same box: Δ pp [95% CI] | vs reference run: Δ pp [95% CI] |
|---|
| bibr-extract-qwen35-4b, bf16: the merged checkpoint this build was made from, same card and harness setup | vLLM 0.31.0, RTX 5090 | 217 | – | +0.4 [0.0, +1.2] |
| bibr-extract-qwen35-4b reference run: the unmerged LoRA on Qwen3.5-4B, from a separate run on another GPU | vLLM 0.31.0 | 216 | – | – |
| this model | vLLM 0.31.0, RTX 5090 | 218 | +0.4 [−0.8, +1.6]; 2 gained, 1 lost | +0.8 [−0.8, +2.4]; 3 gained, 1 lost |
| this model + MTP speculative decoding (k=3) | vLLM 0.32.0, RTX 5090 | 218 | +0.4 [−0.8, +1.6]; 2 gained, 1 lost | +0.8 [−0.8, +2.4]; 3 gained, 1 lost |
- Which comparison to use: the same-box bf16 row is like for like, with the same checkpoint, card and setup. The 216 reference is the run bibr's model comparisons have used since October.
- Subsets on vLLM 0.31, against the same-box bf16 model:
- gate192 (n=192): +0.5 [0.0, +1.6], 1 gained.
- ci60 (n=60): +0.0 [−5.0, +5.0], 1 gained and 1 lost. The lower bound is below −4 pp, so ci60 alone does not show non-inferiority (one paper is 1.7 pp here).
- Against the reference run the ci60 line is the same, and gate192 is +1.0 [0.0, +2.6].
- Subsets with MTP on vLLM 0.32, against the same-box bf16 model:
- gate192: +0.0 [0.0, 0.0], no change.
- ci60: +1.7 [−3.3, +6.7], 2 gained and 1 lost.
- Against the reference run: gate192 +0.5 [0.0, +1.6], and ci60 the same as above.
- Floor fields (title, DOI, authors, references), all 252 papers, vLLM 0.31, against the same-box bf16 model:
- DOI is identical.
- Authors gain 1 paper and references gain 2.
- Title is level: 2 gained, 2 lost, +0.0 [−1.6, +1.6]. Both title losses are ci60 papers, so on ci60 alone title is −3.3 pp [−8.3, 0.0]. One of them is the paper this model loses overall; the other paper fails in both models anyway.
- With MTP on vLLM 0.32: title −0.4 [−2.0, +0.8], 1 gained and 2 lost (on ci60 alone −1.7 [−6.7, +3.3]). DOI is identical, authors gain 2 papers and references gain 3.
- Abstract: mean ROUGE-L 0.955 on vLLM 0.31, the same as the bf16 model, and 0.951 with MTP on vLLM 0.32.
- Noise: two identical runs of this harness agree on about 96% of LLM answers, because bibr picks a random fence boundary for each call. Differences of one or two papers are within run-to-run noise.
Speed
Measured on an RTX 5090, replaying the same 351 extraction requests. The bf16 numbers on vLLM 0.31 come from the same session and card.
Table with columns: bf16 4B, this model | bf16 4B | this model |
|---|
| decode tok/s per request, concurrency 1, vLLM 0.31 | 164.5 | 322.6 (1.96×) |
| aggregate tok/s, concurrency 8, vLLM 0.31 | 705 | 1,118 (1.59×) |
| decode tok/s per request, concurrency 1, vLLM 0.32 + MTP k=3 | 320 * | 469 (2.85×) |
| aggregate tok/s, concurrency 8, vLLM 0.32 + MTP k=3 | 1,214 * | 1,377 (1.95×) |
weights in GPU memory (--language-model-only) | 7.99 GiB | 3.11 GiB |
* From an earlier run on the same card with the same 351 requests. The ratios in brackets are against the bf16 model on vLLM 0.31 without MTP.
- MTP acceptance at concurrency 1, from vLLM's counters:
- accepted / drafted tokens = 51,525 / 52,844 = 0.975. vLLM drafted 2.94 tokens per step on average, not always 3.
- Mean acceptance length is 3.87 tokens per step (1 + accepted tokens per drafting step).
- Acceptance per draft position: 99.4% for the first, 95.9% for the second and 91.2% for the third.
Memory
The budgets below were emulated on a 32 GB card by lowering --gpu-memory-utilization. No 8–11 GB card was run. The CUDA context and any other GPU use come on top of the budget:
- vLLM's own CUDA context took about 0.5 GiB outside its budget;
- vLLM refuses to start unless the whole budget is free once that context exists.
How the probes were run:
- budget = share × 31.45 GiB, the total that vLLM sees on this card;
- every probe used
--max-model-len 32768 --max-num-seqs 64;
- one 32,768-token request needs 1.11 GiB of KV cache.
Table with columns: start, vLLM, budget (share), result, KV-cache tokens, vLLM process at ready| start | vLLM | budget (share) | result | KV-cache tokens | vLLM process at ready |
|---|
| cold | 0.31 | 5.98 GiB (0.19) | fails: room for 26,928 tokens | – | – |
| cold | 0.31 | 6.46 GiB (0.2054) | serves | 42,265 | 5,854 MiB |
| cold | 0.31 | 6.71 GiB (0.2135) | serves | 49,389 | 6,106 MiB |
| cold | 0.32 + MTP k=3 | 6.71 GiB (0.2135) | fails: room for 19,584 tokens | – | – |
| cold | 0.32 + MTP k=3 | 7.86 GiB (0.25) | serves | 48,720 | 7,168 MiB |
| warm | 0.31 | 5.35 GiB (0.17) | serves | 41,316 | 5,818 MiB |
| warm | 0.31 | 5.98 GiB (0.19) | serves | 59,837 | 6,482 MiB |
| warm | 0.32 + MTP k=3 | 7.11 GiB (0.226) | serves | 58,206 | 7,618 MiB |
- Cold: a first start. The vLLM, TorchInductor and Triton compile caches were empty, and another process held 1.6 GiB of the card (a 1 GiB buffer plus its own CUDA context).
- Warm: the compile cache from an earlier start was present. Those probes ran on the first build of these weights, whose tensors differ only in the 48 F32 tensors listed above (7,680 bytes of 4.03 GB).
- Why cold needs more: compiling during vLLM's memory profiling raises peak activation memory from 0.34 to 1.43 GiB on vLLM 0.31, and from 0.82 to 1.98 GiB on 0.32 with MTP. So a first start needs about 1.1 GiB more budget than a later start.
- Budget needed for one 32,768-token request, from vLLM's own memory report (weights and non-torch memory, plus peak activation, plus that request's KV cache). These are estimates; the probes above bracket them.
- vLLM 0.31: about 6.1 GiB cold (3.59 + 1.43 + 1.11) and 5.0 GiB warm.
- vLLM 0.32 + MTP k=3: about 7.2 GiB cold (3.75 + 1.98 + 1.42; the draft layer needs more KV cache per request) and 6.0 GiB warm.
- Little free memory: another process held all but 7.5 GiB, then all but 7.0 GiB, of the card. vLLM 0.31 started at the 6.71 GiB budget with 7.5 GiB free. With 7.0 GiB free it refused to start, because only 6.51 GiB was left after its own context.
- bibr's default: bibr gives vLLM 85% of the card. vLLM computes that from the CUDA-reported total, which is lower than nvidia-smi's figure (31.45 vs 31.84 GiB on the 5090), so an 8 GB card's budget is below 6.8 GiB. With an assumed CUDA total of 7.6 GiB it is 6.46 GiB. A cold start at exactly that budget served on vLLM 0.31 (the 0.2054 row, emulated). MTP needs about 7.2 GiB for a first start, so it does not fit that budget.
Usage
vllm serve scienceverse/bibr-extract-qwen35-4b-W4A16 --language-model-only \ --default-chat-template-kwargs '{"enable_thinking": false}' --generation-config vllm \ --max-model-len 32768 --max-num-seqs 64# vLLM >= 0.32, faster, about 1 GiB more memory: add# --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --no-async-scheduling
Use the model with thinking off, as it was trained and scored. The memory figures above were measured with --max-num-seqs 64. The 252-paper scores were run with vLLM's default for that setting.
Scored on an RTX 5090 (Blackwell) with vLLM 0.31.0 and 0.32.0. It has also run in bibr's managed mode on an RTX 3090 (Ampere) with vLLM 0.31.0 and --max-num-seqs 64, with another process holding 8 GiB or 12 GiB of the card. Both runs of 48 papers finished with no failed starts. On those 48 papers it passed 39, against 41 for the bf16 model on the same card. It lost 2 papers: one title kept the journal's running head, and in the other, an author's family name became the affiliation. It has not been run on Ada cards.
Licence
Apache-2.0, the same as the bf16 student and its base model Qwen/Qwen3.5-4B (Apache-2.0, revision 851bf6e).