Highlights
All numbers below were measured on this checkpoint, on the setup above, on 2026-11-13.
- KLD against BF16: 0.01069 on turboderp's public trace (median 0.00042, p90 0.03216), 0.00945 on an opencode trace, 0.01294 on our own sysadmin trace; perplexity 1.4461 vs 1.4322 (public), 2.7462 vs 2.7317 (opencode), 1.7081 vs 1.6852 (own).
- Tool calls with the real opencode request: 60/60 valid and 60/60 correct (3 cases × 20 tries,
xhigh reasoning, 4 concurrent requests, no errors, no answers cut by the limit, no tool call left inside the reasoning).
- Vision: 15/15 checks on four image tasks (terminal screenshot, chart, stress-test output, process/port screenshot), and a follow-up turn on a cached image answered correctly.
- MTP acceptance: mean acceptance length 2.93 / 3.07 / 3.45 in the last three 10-second windows of the tool-call run; per-position acceptance 0.82 / 0.63 / 0.48 up to 0.88 / 0.81 / 0.76.
- Weights 15.9 GB (decimal), 14.78 GiB in GPU memory; 65,536-token sessions with an fp8 KV cache and 4 concurrent sequences ran with no preemption reported in the server log.
Our focus
Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling)
for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality
checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect
that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.
What is in this repository
Which one to use
Table with columns: GPU, Validated configuration, Status| GPU | Validated configuration | Status |
|---|
| 48 GB (A40) | 65,536-token sessions, fp8 KV, --kv-cache-memory-bytes 12000000000, 4 sequences, MTP 3, vision | Validated (Quick start) |
| 24 GB | — | Not measured. The weights take 14.78 GiB, so a 24 GB card should fit them, but we did not run it |
| 96 GB | — | Not measured (KLD only) |
A single revision (main). The native context of the model is 262,144 tokens; we only validated 65,536 (the model card of Arete says it was trained at 64k).
Files and reproducibility
Table with columns: File, What it is| File | What it is |
|---|
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json | EXL3 weights: backbone, vision tower, MTP head (mtp.*), BF16 embeddings |
config.json, quantization_config.json | Model config (quant_method: exl3, 3.85 bits, head 5, vision 6, MTP 4, mtp_num_hidden_layers: 1) and the per-tensor allocation |
generation_config.json | Sampling defaults: temperature 1.0, top-p 0.95, top-k 20 |
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, chat_template.jinja | Tokenizer and chat template, unchanged from Arete |
preprocessor_config.json, processor_config.json, video_preprocessor_config.json | Image/video processor config. image_processor_type is Qwen2VLImageProcessorFast (Arete declares Qwen2VLImageProcessor, which exllamav3 1.6.6 rejects) |
LICENSE, NOTICE | Apache-2.0 and the list of changes made in this quantization |
SHA256SUMS | SHA-256 of every file above |
Not included: the BF16 reference, the calibration and evaluation data (internal, not published), the vllm-quantpack plugin and exllamav3 (install them from their own repositories).
Results
Hardware and stack for every table: A40 48 GB, vLLM 0.33.0, exllamav3 1.6.6, vllm-quantpack 0.1.5, the Quick start configuration (MTP 3, fp8 KV, 4 sequences, 65,536 tokens), unless stated.
Quantization quality (KLD against the BF16 model)
Measured with exllamav3's qbench on an RTX PRO 6000 Server Edition; the BF16 reference is Arete-27B itself. Tokenizer unchanged from Qwen3.8, so the public trace needed no re-tokenization.
Table with columns: Trace, KLD mean, KLD median, KLD p90, Perplexity (this quant / BF16)| Trace | KLD mean | KLD median | KLD p90 | Perplexity (this quant / BF16) |
|---|
turboderp's public qbench trace | 0.01069 | 0.00042 | 0.03216 | 1.4461 / 1.4322 |
| opencode trace (24 rows from real opencode sessions, not part of the calibration set) | 0.00945 | 0.00293 | not recorded | 2.7462 / 2.7317 |
| our own sysadmin trace | 0.01294 | 0.00298 | not recorded | 1.7081 / 1.6852 |
Measured bits per weight: 3.866.
The real opencode request (its long system prompt and its tools, captured and replayed), 3 cases × 20 tries, reasoning_effort: xhigh, sampling taken from generation_config.json (temperature 1.0), max_tokens 8000, 4 requests at a time.
Table with columns: Case, Valid, Correct| Case | Valid | Correct |
|---|
| full session | 20/20 | 20/20 |
prompt + ls | 20/20 | 20/20 |
greeting + story + ls | 20/20 | 20/20 |
No errors, no answers cut by the limit, no tool call inside the reasoning or left unparsed in the content. Mean 4 s and 67 generated tokens per answer. We did not run this test at other temperatures, and we did not repeat it on the BF16 model.
Vision
Four image tasks, temperature 0, thinking disabled, one request each.
Table with columns: Task, Checks passed| Task | Checks passed |
|---|
terminal screenshot (systemctl output) | 3/3 |
| chart (read values) | 3/3 |
| stress-test output | 3/3 |
| process/port screenshot (which process holds the port, what to run) | 6/6 |
A second turn that asks about an image sent in the first turn was answered correctly (2.3 s and 1.7 s per turn). The images are synthetic: not photographs and not video. OCR on scanned documents, several images in one request, video and tokens per image were not measured.
Speculative decoding (MTP)
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'. Acceptance from vLLM's own log, the last three 10-second windows of the tool-call run (up to 4 concurrent requests, temperature 1.0):
Table with columns: Window, Mean acceptance length, Per-position acceptance (1 / 2 / 3), Draft acceptance| Window | Mean acceptance length | Per-position acceptance (1 / 2 / 3) | Draft acceptance |
|---|
| 1 | 2.93 | 0.818 / 0.625 / 0.483 | 64.2 % |
| 2 | 3.07 | 0.829 / 0.672 / 0.569 | 69.0 % |
| 3 | 3.45 | 0.882 / 0.806 / 0.758 | 81.5 % |
The MTP head comes from the base model and was not retrained on Arete; the numbers above are what it achieves on this fine-tune. Decode speed at 1 / 2 / 4 / 8 requests, and speed with and without MTP, were not measured (the log windows above are not a controlled speed test).
Capacity
Model loading took 14.78 GiB and 95 s. With --kv-cache-memory-bytes 12000000000 (fp8, MTP 3) vLLM reported 251,611 tokens of KV cache, 3.84× a 65,536-token session; four concurrent requests ran and the server log shows no preemption. The VRAM really free at peak and the capacity on other GPU sizes were not measured. The first start on a clean cache took about 13 minutes (torch.compile, FlashInfer JIT kernels, CUDA graphs); later starts reuse the caches.
Sampling and repetition loops
generation_config.json sets temperature 1.0, top-p 0.95, top-k 20 and the tool-call test above ran with exactly that. The repetition-loop test (long sysadmin answers at several temperatures) was not run, so we make no recommendation on temperature.
Known model behaviour and discarded options
- We did not re-run any test on the BF16 original, so we cannot say whether a weakness comes from the model or from the quantization. The tool-call test left no margin to compare (60/60).
- Nous describes Arete as post-trained "to reinforce western values on a Chinese-trained model" and publishes its own metrics. We did not evaluate that, or any general benchmark: our checks measure quantization fidelity and the use case in Our focus.
- No other bit rate, KV type or GPU size was tried for this checkpoint.
Requirements & install
Before you install, the commands below assume (and do not install):
- An NVIDIA driver for CUDA 13 and a CUDA 13 toolkit with
nvcc (we used 13.0; export CUDA_HOME=/usr/local/cuda-13.0): FlashInfer compiles kernels on the first start and exllamav3 is built from source.
uv, git, python3.12-dev and build-essential (Triton compiles a C helper at every start: without the headers, fatal error: Python.h).
- About 20 GB for the model, plus the virtual environment and the
uv cache.
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.33.0 --torch-backend=auto # torch 2.14.0+cu132, FlashInfer 0.7.2.post1 # exllamav3 1.6.6 has no prebuilt wheel for torch 2.14: build it from source (the generic PyPI wheel also compiles on first use)uv pip install --python venv/bin/python ninja packaging setuptools wheelgit clone --depth 1 --branch v1.6.6 https://github.com/turboderp-org/exllamav3 exllamav3-srcexport CUDA_HOME=/usr/local/cuda-13.0TORCH_CUDA_ARCH_LIST="8.6;12.0" uv pip install --python venv/bin/python --no-build-isolation ./exllamav3-src uv pip install --python venv/bin/python vllm-quantpack==0.1.5venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Adjust TORCH_CUDA_ARCH_LIST to your GPU (8.6 = A40 / RTX 30xx, 8.9 = Ada, 12.0 = RTX PRO 6000 / RTX 50xx). Our validation did not install FlashInfer's prebuilt cubin and jit-cache packages (we found no build for CUDA 13.2 / FlashInfer 0.7.2.post1 at the time of writing); FlashInfer compiled its kernels at the first start. vllm-quantpack registers quant_method: "exl3" through vLLM's plugin entry point, so no other flag is needed; if the older plugins OrcaSAQ2 or qwen38_exl3_vision are installed in the same environment, uninstall them or start with VLLM_PLUGINS=quantpack.
Quick start
The exact command of the validated configuration (A40 48 GB):
export CUDA_HOME=/usr/local/cuda-13.0 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truesource venv/bin/activatevllm serve JRamirez-UAB/Arete-27B-EXL3-3.85bpw \ --served-model-name arete-27b \ --max-model-len 65536 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 12000000000 \ --max-num-seqs 4 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization. The 12 GB value was validated on a 48 GB card; for a smaller card measure your own and check the GPU KV cache size line against --max-model-len.
Using it with opencode (how we tested)
The tool-call test replays the real opencode request against the vLLM API; the image test calls the API directly. An opencode configuration for this server (not itself part of the measured tests):
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "arete-27b": { "name": "Arete 27B EXL3 (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 65536, "output": 8000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/arete-27b", "variant": "xhigh" }, "plan": { "model": "local-vllm/arete-27b", "variant": "xhigh" } }}
limit.context must equal --max-model-len. Declare "temperature": true, otherwise opencode sends no temperature to a custom model; without a temperature the server applies generation_config.json.
- Only the
xhigh variant was tested; reasoning_effort accepts xhigh, medium and low, and they are prompt instructions, not limits. Reasoning length at each level was not measured.
- Start opencode inside a project directory; for
opencode run from scripts, close stdin (</dev/null).
Important notes
- Memory. The driver reserves part of the VRAM that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.used,memory.free --format=csv. We did not record the free VRAM at peak, so leave margin on your card.
- Hugging Face page warnings. The "Model size" shown by Hugging Face counts the packed storage elements of the safetensors files (EXL3 packs about 4 weights into each
int16), not parameters: this is a 27B model at about 3.85 bits per weight. The configuration warning "quantization_config.bits must be an integer" is also expected: for EXL3 bits is the average of a per-tensor allocation (the per-tensor values are in quantization_config.json).
- Use
--tool-call-parser qwen3_coder and --reasoning-parser qwen3, as in the Quick start.
- The first start compiles kernels and takes minutes; do not judge a failed health check before it ends.
Quantization recipe
Table with columns: Step, What we did| Step | What we did |
|---|
| Source | NousResearch/Arete-27B (BF16, Apache-2.0), same architecture as Qwen3.8-27B (qwen3_5, hybrid, 64 layers, 16 with full attention, head dim 256) |
| MTP head | 15 mtp.* tensors copied from Qwen/Qwen3.8-27B (the head Arete's fine-tune dropped), mtp_num_hidden_layers set to 1; not retrained |
| Calibration | exllamav3 convert.py with 595 rows of 11,371 tokens, from three sources: the standard exllamav3 calibration set, real Spanish-language sysadmin agent sessions from our own opencode deployment (internal, not published, no excerpts shown) and agent steps generated by the BF16 model itself |
| Conversion | exllamav3 1.6.6, -b 3.85 (backbone), -hb 5 (lm_head), -vb 6 (vision tower), -mb 4 (MTP head); codebook mul1; output scales automatic |
| Embeddings | embed_tokens kept in BF16 (2.54 GB) |
| Fix applied | preprocessor_config.json: image_processor_type changed from Qwen2VLImageProcessor to Qwen2VLImageProcessorFast, which exllamav3 1.6.6 requires |
| Output | 3,080 tensors: 987 vision, 39 mtp.* (EXL3 sub-tensors of the 15 MTP weights), lm_head; 15.9 GB |
Measured KLD is in Results.
Integrity and validation
- Artifact integrity:
SHA256SUMS lists every file. The upload was verified against the local files with SHA-256, and the checkpoint was served from the Hugging Face download for the final test.
- Quantization: KLD and perplexity against BF16 on three traces (above). The opencode trace's first user messages do not overlap with any calibration conversation (checked).
- Chat template and tokenizer: vocabulary, merges and added tokens are identical to Qwen3.8's; the chat template is Arete's, unchanged.
- Runtime: the Quick start configuration was run on one A40 with vLLM 0.33.0 and vllm-quantpack 0.1.5: image tasks, tool-call replay and MTP acceptance (above).
Scope: one machine, one software stack, one GPU model for serving, 65,536-token sessions.
Limitations
- Not measured: a 24 GB GPU, any context above 65,536 tokens (needle retrieval included), decode speed at 1 / 2 / 4 / 8 requests, speed with and without MTP, free VRAM at peak, repetition loops, reasoning length, agent tasks with real command execution, OCR on scanned documents, video, and any standard benchmark.
- The MTP head is the base model's and was not retrained on Arete; acceptance (above) is lower than a head trained on the fine-tune could reach. Without
vllm-quantpack this checkpoint does not load in stock vLLM.
- Nous's values-alignment goal for Arete was not evaluated here; read their card for their metrics.
- The install block was assembled from the same pieces we validated (exllamav3 1.6.6 built from source once and reused), but it was not re-run end to end on a clean host.
- The tool-call test replays the real opencode request and does not execute the commands, and the vision tests use synthetic images.
- Quantization noise: KLD 0.011 against BF16 on the public trace; answers can differ from the original model's.
Feedback welcome
I would be grateful for reports of installations of this checkpoint, both working and failed ones, especially on 24 GB GPUs, which we did not test. Please open a discussion in the Community tab with your GPU(s), driver, vLLM / exllamav3 / vllm-quantpack versions, the full command line and the result (speed, context, or the error message).
Credits & license
- Base model: NousResearch/Arete-27B by Nous Research, built on Qwen/Qwen3.8-27B by the Qwen team (Alibaba Cloud); the MTP head comes from the latter.
- exllamav3 and the EXL3 format by turboderp (MIT), including
qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
- vLLM and our
vllm-quantpack plugin (source).
- License: Apache-2.0 for the weights, as the base models; see LICENSE and NOTICE. exllamav3 and
vllm-quantpack are not redistributed here; install them from their repositories under their own licenses.