Also serves autotrust/JEV-27B (the text model)
autotrust/JEV-27B is the same model without the vision tower. I checked: all
851 of its backbone tensors are byte-identical to Qwen3.8-27B, and its adapter's 802 tensors are byte-identical to
the one in this repo. Its decision head and calibration file are identical too. So this repo is the B70 port of both.
PubMedQA below is in fact compared against JEV-27B's own BF16 run.
One trap if you mix files: JEV-27B's adapter names its layers model.layers.*, while this backbone loads as the
multimodal class (model.language_model.layers.*). vLLM then applies only the adapter's lm_head part, without any
error, and System 1 quietly gets worse (PubMedQA 74.4% with 16 confident answers flipped, instead of 78.4% and 0).
Use the adapter_vllm/ shipped here, which already has the right names.
Does it still give the same answers?
I wrote the pass rules down before running anything. All checks are text-only.
Table with columns: check, int4 on B70, reference, rule, result| check | int4 on B70 | reference | rule | result |
|---|
| card example (billing / shipping / tech support) | 0.9954 / 0.0000 / 0.0046 | 0.9969 / 0.0000 / 0.0031 (card) | – | matches |
| PubMedQA, 500 questions, accuracy | 78.4% | 77.8% (JEV's BF16 run) | within 2 points | pass |
| PubMedQA, same answer as BF16 | 97.6% | – | – | – |
| PubMedQA, flipped answers where BF16 was ≥ 80% sure | 0 of 333 | – | ≤ 1% | pass |
| held-out yes/no (500 rows), AUROC on rows whose target isn't near 50/50 | 1.000 | 0.995 (card, full 30K, rows not stated) | ≥ 0.985 | pass |
| held-out ratings 0-5 (500 rows), MAE of expected rating | 0.107 | 0.098 (card) | ≤ 0.15 | pass |
| held-out choice, all 9,199 rows, top-1 agreement on clear rows | 94.6% (7,305 / 7,723) | 95.8% (card) | ≥ 93% | pass |
Notes, so nothing is hidden:
- The PubMedQA reference is the BF16 run that ships in JEV's demo repo (
08-biomedical-qa/results.json). It was made
with the text model JEV-27B, so the 2.4% disagreement mixes int4 error with the VL-vs-text difference (the JEV card
puts that at a mean |Δp| of 0.010).
- Choice was first checked on a 500-row sample and came out at 92.9%, one item under the bar. Before rerunning, I wrote
down that the full set would replace the sample, with the same bar and no further reruns. The full set is the 94.6%.
- The yes/no AUROC only counts rows where the target is clearly yes or no (more than 10 points away from 50/50), so it
isn't directly comparable to the card's 0.995.
- "Clear" here means the target's top option has at least 50%. The JEV card doesn't say how it defines clear, so the
1.2-point gap to 95.8% is part int4 and part definition, and I can't separate the two.
- KL to the held-out targets is about 0.020 (weighted to the full set's mix). The card gives ≈ 0.017, but only on
"Jev-labelled rows", a subset it doesn't define.
- Not tested: image inputs, thinking-mode accuracy (
thinking: "on" / "auto"; only its speed was measured),
contexts above 131K.
Raw per-item outputs and the eval script are in eval/. The full-set choice run and the 1,500-row check used eager
mode; PubMedQA and the 1,500-row check were repeated in the fast mode with the same results.
Speed and memory (one B70, stock power)
- System 1: about 11 decisions/s with 8 requests at a time on the held-out rows (9,199 in 853 s, eager mode).
PubMedQA abstracts are longer: 500 in 163 s. Graphs and MTP give the same System 1 results (PubMedQA 78.4%, 0 flips).
- System 2 / thinking: 76 tok/s for one request (code; 69 on prose), with XPU graphs + MTP (k=4, mean acceptance
~3.3). With
thinking: "on" on 32 PubMedQA questions, 8 at a time: median 469 thinking tokens and 17.7 s per
question, 170 tok/s across the batch. Eager mode without MTP (FAST=0) does 8 tok/s.
- Memory: 14.8 GiB of weights on the GPU. The embeddings and the vision tower (3.2 GiB) stay in host RAM and are read
over PCIe; plan on about 12 GB of free host RAM in total.
The KV cache holds 10.1 GiB (≈ 308K tokens in fp8) at
GPU_UTIL=0.90 with a 131K max context.
- 24 GB cards:
GPU_UTIL=0.72 keeps vLLM at 21.8 GiB and still leaves 4.6 GiB of KV cache (≈ 138K tokens), so
the full 131K context fits. Only 16 of the 64 layers keep a KV cache, the rest are linear attention. Checked on the
B70 under that cap (the example decision gives the same probabilities to every digit); not run on a real 24 GB card.
Run it
Needs Docker with /dev/dri, one Arc Pro B70 (or another 32 GB Battlemage card, untested), about 12 GB of free host
RAM, and my B70 cookbook for the vLLM patches and attention
kernels. Everything is pinned to the vllm/vllm-openai-xpu:v0.31.0 image. On a 24 GB card add GPU_UTIL=0.72. The launcher starts in the fast mode (XPU graphs + MTP); FAST=0 gives eager mode.
git clone -b b70.6 https://github.com/riceharvest/b70-qwen38-cookbook # tag b70.6 = the v0.31 patch scripts this launcher uses(cd b70-qwen38-cookbook && gh release download v3 -p 'libattn_kernels_xe_2.b70.so' -D kernels) # v2+: fixes a packed-MTP-verify bug in v1hf download darioooooo0o/JEV-27B-VL-GPTQ-Int4-asym --local-dir ~/models/jev-b70COOKBOOK=$PWD/b70-qwen38-cookbook MODEL_DIR=~/models/jev-b70 bash ~/models/jev-b70/serve-b70.sh # :8000
curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{"kind":"choice", "state":"Customer: my card was charged twice for one coffee.","question":"Which team should handle this?", "options":["billing","shipping","tech support"]}'
The prebuilt image ghcr.io/riceharvest/b70-vllm:v0.31.0-b70.3 also serves it (use that tag; :latest moved to vLLM v0.32, not tested with JEV yet): it detects a JEV repo
(-e MODEL=darioooooo0o/JEV-27B-VL-GPTQ-Int4-asym) and starts System 1 + System 2 with the LoRA.
The API is JEV's own (/v1/decide, or model jev-decision through /v1/completions), so the
JEV card and its demo repo
apply as they are. Keep --max-num-seqs 8: JEV notes that with more than 8 sequences per batch, vLLM's LoRA path
returns wrong System 1 probabilities.
Reproduce the evals
cd evalgit clone https://github.com/yuhai-china/JEV-27B-DEMO demomkdir -p data && hf download SargeDev/jev-distill-corpus-v3 test_set_30k.jsonl --repo-type dataset --local-dir dataJEV_URL=http://localhost:8000 python3 eval_jev.py pubmedqaJEV_URL=http://localhost:8000 python3 eval_jev.py s1 500JEV_URL=http://localhost:8000 python3 eval_jev.py choice_full
Credits
JEV-27B-VL, its adapter, decision head, calibration and serve_decide.py are by
autotrust (apache-2.0). The backbone is Qwen3.8-27B by the Qwen team (apache-2.0).
The MTP and prefix-cache patches in the cookbook are from
SergiioB's B70 cookbook (MIT).
Questions or issues: reply to @imdariotoo on X or open a discussion on this repo.