What is quantized
Table with columns: part, weights, activations, kernel| part | weights | activations | kernel |
|---|
| routed experts (40 x 256 MLPs) | int4, group 128, symmetric | fp8 (e4m3), per-token dynamic | Humming W4A8 MoE |
| attention / linear-attention / shared expert Linears | int4, group 128, symmetric | int8, per-token dynamic | Marlin W4A8 (WNA8Int) |
lm_head (the 264-slot decision head) | bf16 | — | — |
vision tower, MTP layer, routers, shared_expert_gate | bf16 | — | — |
Weights are stored pack-quantized (8 int4 per int32), so the expert tensors keep the
_packed names vLLM's MoE loader expects.
How it was built
- AWQ smoothing (duo scaling, complete mappings on the always-on path: the 10 full-attention
input_layernorm -> q/k/v, the 30 linear-attention input_layernorm -> GDN projections, and
shared_expert.up_proj -> shared_expert.down_proj) followed by GPTQ with per-module
activation schemes. The routed experts are deliberately not smoothed: llm-compressor's AWQ grid
search needs a forward pass per expert per grid point (~10k passes per layer, ~20 h for the model),
and a partial smoothing of a shared norm is measurably harmful (see the journal).
- Calibration: the provider's own calibration split,
SargeDev/jev-distill-corpus-v3
(calibration, 13,766 rows), sampled kind-balanced (45% choice / 30% noul / 25% score) into
the checkpoint's exact System 1 prompt format, 1024 samples x 2048 tokens, all 256 experts
calibrated (moe_calibrate_all_experts).
- Built with llm-compressor 0.14 / compressed-tensors 0.19 / transformers 5.17 / torch 2.14
(single RTX 4090; sequential onloading keeps one layer on the GPU at a time).
Validation
Validation runs the checkpoint's own serve_decide.py (POST /v1/decide) against the
provider's held-out test split; metrics follow the same convention as the JEV-27B-VL AWQ build
(KL(target||model), top-1 against the teacher argmax, ECE on the teacher argmax).
Full reports: eval/validation-w4a8-awq2.json (this build), eval/validation-w4a8.json
(GPTQ-only variant), eval/kernel-selection.txt (the vLLM startup lines proving the W4A8 kernels).
Quality — w4a8-awq2
Table with columns: slice, n, KL mean, KL median, top-1, ECE, 95% CI| slice | n | KL mean | KL median | top-1 | ECE | 95% CI |
|---|
| ALL | 500 | 0.03 | 0.0081 | None | 0.1548 | [0.0228, 0.0403] |
| confident | 258 | 0.0339 | 0.0077 | 0.9884 | 0.1802 | [0.0211, 0.0528] |
Card example (w4a8-awq2): {"choice": "billing", "probabilities": [0.9997, 0.0001, 0.0001, 0.0001], "seconds": 0.286}
Bench (concurrency 1) (w4a8-awq2): {"requests": 24, "throughput_req_s": 17.44, "median_ms": 57.3, "p95_ms": 57.9}
Long decision (prefill) (w4a8-awq2): {"state_chars": 26200, "input_tokens": 4644, "seconds_cold_then_cached": [0.29, 0.06, 0.06], "ms_per_token_cold": 0.062, "choice": "technical support"}
Quality — w4a8
Table with columns: slice, n, KL mean, KL median, top-1, ECE, 95% CI| slice | n | KL mean | KL median | top-1 | ECE | 95% CI |
|---|
| ALL | 500 | 0.0278 | 0.0078 | None | 0.1774 | [0.0223, 0.0347] |
| confident | 258 | 0.0286 | 0.0075 | 0.9884 | 0.1822 | [0.0194, 0.0415] |
Card example (w4a8): {"choice": "billing", "probabilities": [0.9998, 0.0, 0.0001, 0.0001], "seconds": 0.292}
Bench (concurrency 1) (w4a8): {"requests": 24, "throughput_req_s": 17.33, "median_ms": 57.5, "p95_ms": 58.5}
Long decision (prefill) (w4a8): {"state_chars": 26200, "input_tokens": 4644, "seconds": 0.29, "ms_per_token": 0.063, "choice": "technical support"}
Serving shape (measured on one RTX 4090, 24 GB)
Table with columns: this build | this build |
|---|
| weights in VRAM | 19.04 GiB |
| KV cache | 53,833 tokens (fp8 KV, --max-num-seqs 8, --max-model-len 16384) |
| latency, concurrency 1 (24 short decisions) | 57.3 ms median / 57.9 ms p95 / 17.4 req/s |
| latency, concurrency 8 | 34.8 req/s |
| cold prefill, 4,644-token decision | 0.29 s (0.062 ms/token) |
| image decision | red circle -> circle, p = 0.99995 |
| wide choice |
Serve
hf download aioutfitters/JEV-35B-AWQ-INT4-W4A8 --local-dir JEV-35B-AWQ-INT4-W4A8cd JEV-35B-AWQ-INT4-W4A8 && bash serve.sh # vLLM on :8000, one 24 GB GPU
serve.sh is the provider's own server (serve_decide.py: OpenAI endpoints + POST /v1/decide).
On a 24 GB card add --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 and pick
--max-model-len for your workload; the decision model reads up to 256 options in one pass.
License
Apache-2.0, same as autotrust/JEV-35B.