What is quantized
Table with columns: part, precision| part | precision |
|---|
every text Linear, including the GatedDeltaNet projections | int4, group 128, symmetric + int8 per-token dynamic activations (Marlin W4A8 / CompressedTensorsWNA8Int) |
lm_head (decision head) | bf16 |
| vision tower | bf16 (untouched; spliced back after quantization) |
Recipe: AWQ smoothing (hybrid-attention mappings, duo scaling) followed by GPTQ, calibrated on
1024 kind-balanced serving-format prompts (45% choice) from the provider corpus
(SargeDev/jev-distill-corpus-v3, calibration split), max_seq_length 2048. Built with
llm-compressor 0.14 / compressed-tensors 0.19 / transformers 5.17 / torch 2.14.
Validation
Served with the same stack as the deployed W4A16 build (--kv-cache-dtype fp8, --max-num-seqs 8,
one RTX 4090) and scored on the same 500 held-out corpus rows (seeded shuffle), through the same
POST /v1/decide shim:
Table with columns: slice, W4A8 + AWQ (this build), deployed W4A16 g64 (v2)| slice | W4A8 + AWQ (this build) | deployed W4A16 g64 (v2) |
|---|
| KL mean, all rows | 0.0147 | 0.0143 |
| noul | 0.0022 | 0.0021 |
| score | 0.0186 | 0.0180 |
| choice | 0.0385 | 0.0374 |
| top-1 noul | 0.732 | 0.774 |
| top-1 choice | 0.825 | 0.851 |
card example (billing) | p = 0.9963 | p = 0.9976 |
Image decisions verified (red circle -> circle, p = 0.99999), so the bf16 vision tower is intact.
Cold prefill on a 5,098-token decision: 1.76 s (0.345 ms/token) — the deployed W4A16 build measures
0.350 ms/token on the same harness, i.e. parity: at these batch sizes this workload is
bandwidth-bound, not GEMM-bound, so 8-bit activations do not buy prefill throughput here.
Serve
hf download aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8 --local-dir JEV-27B-VL-AWQ-INT4-W4A8
serve.sh / serve_decide.py are the provider's own server; start vLLM with
--enable-lora --max-lora-rank 32 --lora-modules jev-decision=<dir>/adapter_vllm
(see the base model card) plus --kv-cache-dtype fp8 --gpu-memory-utilization 0.95 --max-num-seqs 8
on a 24 GB card. decision_head/ carries the head bias + per-kind temperatures.
License
Apache-2.0, same as autotrust/JEV-27B-VL.