JevBench231
187/231 correct · 80.95% accuracy · 68.16 Capability
Table with columns: Metric, Value| Metric | Value |
|---|
| Intelligence | 55.02 |
| Calibration | 81.30 |
| Capability | 68.16 |
Measured on JevBench's 231 public tasks using the official CLI, with public O1S Capability calculated from its predictions. Temperature is 1; no probability postprocessing. These are local public-task results. The scores above use the Triton evaluation backend. With the FlashInfer deployment command below, the same weights measured 189/231 (81.82%), Intelligence 55.66, Calibration 80.61 and Capability 68.14.
Full-attention layers use bidirectional attention in training and inference; Qwen3.5 linear-attention layers retain their original behavior. Inputs contain prompts only. The merge script sets text_config.is_causal=false in the merged model configuration.
Download and merge
hf download Qwen/Qwen3.5-4B --local-dir ./Qwen3.5-4Bhf download AsystemAReno/AReno-Jev-4B --local-dir ./jev python ./jev/merge_weights.py \ --base-model ./Qwen3.5-4B --checkpoint ./jev --output ./jev-merged
Use the original floating-point base model as the merge input.
Serve with SGLang
sglang serve --model-path ./jev-merged \ --host 0.0.0.0 --port 8000 \ --dtype bfloat16 --context-length 4096 \ --attention-backend flashinfer --chunked-prefill-size -1
The SGLang configuration above was tested with version 0.5.23.
Serve with AReno
Install AReno with --bi-attn support (implementation), then select your backend.
CUDA:
areno serve --sgl-jev --bi-attn --backend cuda --model-path ./jev-merged \ --world-size 1 --tp-size 1 --host 127.0.0.1 --port 8000
Mac:
areno serve --sgl-jev --bi-attn --backend mlx --model-path ./jev-merged \ --world-size 1 --tp-size 1 --port 8000
Fine-tuning
CUDA:
areno train --sgl-jev --bi-attn --backend cuda --qlora \ --ckpt ./Qwen3.5-4B --lora-adapter-path ./jev \ --dataset-path ./train.jsonl --save-path ./jev-ft \ --lora-rank 16 --lora-alpha 32 --lr 2e-4 --max-context-len 4096 \ --world-size 1 --tp-size 1 --mini-bs 1 --epochs 1
MLX:
areno train --sgl-jev --bi-attn --backend mlx --q4 \ --ckpt ./Qwen3.5-4B --lora-adapter-path ./jev \ --dataset-path ./train.jsonl --save-path ./jev-ft \ --lora-rank 16 --lora-alpha 32 --lr 2e-4 --max-context-len 4096 \ --world-size 1 --tp-size 1 --mini-bs 1 --epochs 1
API example
curl http://localhost:8000/v1/systemone \ -H 'Content-Type: application/json' \ -d '{"state":"The meeting starts at 14:30. Travel takes 45 minutes.","questions":{"departure":{"type":"choice","instructions":"Choose the latest departure that arrives by the meeting.","criteria":{"a":"13:45","b":"14:00"}}}}'