JRamirez-UAB

Qwen3.8-27B-pi-EXL3-3.85bpw-24GB

README

License: apache-2.0

Family comparison (Qwen3.8-27B-pi, single RTX 3090 24 GB)

Table with columns: Model, Size, Context in 24 GB, KLD mean / median · pub., Perplexity (BF16: 1.5561 · pub. 1.3265), Needle · tools · agent, Reasoning xhigh (tokens, correct), Decode with MTP, 1 / 2 sessions
Model	Size	Context in 24 GB	KLD mean / median · pub.	Perplexity (BF16: 1.5561 · pub. 1.3265)	Needle · tools · agent	Reasoning xhigh (tokens, correct)	Decode with MTP, 1 / 2 sessions
EXL3 3.85 bpw (`main`, this repo)	14.6 GB	215,000 (with vision)	0.0074 / 0.0016 · 0.0070	1.5702 · 1.3362	8/8 at 195,210 · 90/90 · 8/8	17,070 (46/48)	76.8 / 135.2 tok/s
EXL3 3.70 bpw (`3.70bpw`, this repo)	14.2 GB	226,000 (with vision)	0.0093 / 0.0020 · 0.0085	1.5706 · 1.3386	not measured	not measured	not measured
W4A16 GPTQ g64 + INT8 embeddings	16.9 GB	176,000 (text)	0.0182 / 0.0041 · 0.0170	1.5900 · 1.3514	8/8 at 157,733 · 90/90 · 8/8	18,183 (48/48)	93.9 / 168.6 tok/s

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 pi model, measured with exllamav3's qbench. The first figures use an internal sysadmin/agent evaluation set (not published; separate from the calibration data); pub. uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3), so anyone can reproduce it. Lower is better.
Context: largest --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak.
Needle: 8 facts spread through one prompt of that length, all 8 returned. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases (fresh request, after a long story, full multi-turn session) × 10 tries × 3 configurations (xhigh without temperature, xhigh at T=0.3, thinking off at T=0.3). Agent: 4 sysadmin tasks × 2 with opencode run and real tool execution.
Reasoning: total reasoning tokens over 6 checkable problems × 8 tries at xhigh (temperature 0.6). : forced tokens, temperature 0.

Variants / revisions

Table with columns: Revision, Linear layers, lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)
Revision	Linear layers	`lm_head`	Vision	MTP head	Embeddings	Size	KLD (ours / pub.)
`main`	3.85 bpw, SC allocation	5 bits	6 bits	4 bits	INT8 per row	14.6 GB	0.0074 / 0.0070

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Calibration trace	Generated by Qwen3.8-27B-pi itself (`reasoning_effort: xhigh`, MTP enabled) with exllamav3's `sc_trace` on 196 prompts: 157 from exllamav3's own `sc_trace` set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published); 196 sequences × 2048 tokens used for conversion
Reference and sensitivity	`sc_rfn_probe` against a 2.00 bpw reference, `sc_measure` (per-tensor sensitivity on the trace)
Bit allocation	`sc_optimize` at the target rate with a 5-bit `lm_head` → per-tensor recipe (`quantization_config.json`)

Tokenizer, chat template and generation config unchanged from bytkim/Qwen3.8-27B-pi (XML tool calls, qwen3_coder). The dflash2/ companion of the source repo is not included.

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers in vLLM through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). Its README describes it as a "correctness path".
qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/; 0.2.0 also adds EXL3 support for MoE experts, unused by this dense model): drops the separate attn.{q,k,v}_proj tensors of the EXL3 vision tower (vLLM uses the fused attn.qkv) and pads input dimensions to multiples of 128 for layers that need it (4304 → 4352).

Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto        # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB \  --served-model-name qwen3.8-27b-pi \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 215000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8420000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

3.70bpw branch, 226k context: the same command with --revision 3.70bpw --max-model-len 226000 --kv-cache-memory-bytes 8850000000.

Important notes

Memory. The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv, not as 24,576 − used. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). --max-num-batched-tokens 1024 and expandable_segments lower the peak by ≈350 MiB at no measurable cost. With MTP, KV capacity grows in block-sized steps: check the GPU KV cache size line (8,420,000,000 B → 216,433 tokens; 8,850,000,000 B → 227,439) and keep --max-model-len below it.
Measured peaks (2 sessions, vision): 3.85 bpw 311 MiB really free (719 MiB at rest); 3.70 bpw see the benchmark table. Leave the GPU for the LLM: a desktop session on the same GPU would eat this margin.
Hugging Face page warnings: the "Model size" shown by Hugging Face (≈8B params) counts the packed storage elements of the safetensors files (EXL3 packs ≈4 weights into each int16), not parameters: this is the 27B model at ≈3.85 bits per weight. The Configuration Parsing Warning "quantization_config.bits must be an integer" is also expected: for EXL3 is the average bits per weight of a per-tensor allocation (the per-tensor values are in ); exllamav3 and the OrcaSAQ2 plugin read it as a float.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run directly against the vLLM API.

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwen3.8-27b-pi": {          "name": "Qwen3.8-27B-pi EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 215000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 0.3 },    "plan":  { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 0.3 }  }}

limit.context = --max-model-len (226000 for the 3.70bpw branch); with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests). The agent tasks ran at temperature 0.3; the tool-call test at both T=1.0 (model default) and T=0.3.
Without attachment / modalities opencode will not send images; drop them if you serve with --language-model-only.
Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (</dev/null).

Benchmarks (single RTX 3090 24 GB, 300 W)

Table with columns: Test, main 3.85 bpw (215,000), 3.70bpw (226,000)
Test	`main` 3.85 bpw (215,000)	`3.70bpw` (226,000)
KV pool	216,433 tokens (KV 8.42e9 B)	227,439 tokens (KV 8.85e9 B)
8 facts in one long prompt (greedy)	8/8 at 61,675, 123,142 and 195,210 tokens (524 tok/s at 195k)	not measured
2 simultaneous sessions	2 × ≈94k, 8/8 in both, 698 tok/s aggregate	not measured
Decode, 1 request, t=0 / 0.6	76.8 / 75.5 tok/s (MTP: 70 % of drafts accepted)	not measured
Decode, 2 concurrent, t=0 / 0.6

For comparison, the base Qwen3.8-27B EXL3 3.85 reasoned 23,221 tokens at xhigh on the same problems (pi: 17,070, −26 %), with the same context and memory.

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images, short reasoning problems and agent tasks.
The full 262,144 context does not fit at this rate on 24 GB with MTP and vision.
Decode is ≈20 % slower than the W4A16 sibling on Ampere; the OrcaSAQ2 plugin is young and describes itself as a correctness path.
The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding; 0.2.0). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
LICENSE (Apache-2.0) and NOTICE (the pi model's notice, plus this quantization).

Feedback welcome

Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM / exllamav3 / plugin versions, the full command line and the result (speed, context, or the error message).

Credits & acknowledgements

Model: bytkim/Qwen3.8-27B-pi by bytkim (Thomas Kim), built on Qwen/Qwen3.8-27B (Qwen team).
exllamav3 and the EXL3 format by turboderp (MIT), including the SC pipeline, qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.

License

Apache-2.0, same as the pi model and Qwen3.8-27B; the pi model's NOTICE is kept in NOTICE. The OrcaSAQ2 plugin and exllamav3 are not redistributed here; install them from their repositories under their own licenses.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

bytkim/Qwen3.8-27B-pi

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Family comparison (Qwen3.8-27B-pi, single RTX 3090 24 GB)

Table with columns: Model, Size, Context in 24 GB, KLD mean / median · pub., Perplexity (BF16: 1.5561 · pub. 1.3265), Needle · tools · agent, Reasoning xhigh (tokens, correct), Decode with MTP, 1 / 2 sessions
Model	Size	Context in 24 GB	KLD mean / median · pub.	Perplexity (BF16: 1.5561 · pub. 1.3265)	Needle · tools · agent	Reasoning xhigh (tokens, correct)	Decode with MTP, 1 / 2 sessions
EXL3 3.85 bpw (`main`, this repo)	14.6 GB	215,000 (with vision)	0.0074 / 0.0016 · 0.0070	1.5702 · 1.3362	8/8 at 195,210 · 90/90 · 8/8	17,070 (46/48)	76.8 / 135.2 tok/s
EXL3 3.70 bpw (`3.70bpw`, this repo)	14.2 GB	226,000 (with vision)	0.0093 / 0.0020 · 0.0085	1.5706 · 1.3386	not measured	not measured	not measured
W4A16 GPTQ g64 + INT8 embeddings	16.9 GB	176,000 (text)	0.0182 / 0.0041 · 0.0170	1.5900 · 1.3514	8/8 at 157,733 · 90/90 · 8/8	18,183 (48/48)	93.9 / 168.6 tok/s

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 pi model, measured with exllamav3's qbench. The first figures use an internal sysadmin/agent evaluation set (not published; separate from the calibration data); pub. uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3), so anyone can reproduce it. Lower is better.
Context: largest --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak.
Needle: 8 facts spread through one prompt of that length, all 8 returned. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases (fresh request, after a long story, full multi-turn session) × 10 tries × 3 configurations (xhigh without temperature, xhigh at T=0.3, thinking off at T=0.3). Agent: 4 sysadmin tasks × 2 with opencode run and real tool execution.
Reasoning: total reasoning tokens over 6 checkable problems × 8 tries at xhigh (temperature 0.6). : forced tokens, temperature 0.

Variants / revisions

Table with columns: Revision, Linear layers, lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)
Revision	Linear layers	`lm_head`	Vision	MTP head	Embeddings	Size	KLD (ours / pub.)
`main`	3.85 bpw, SC allocation	5 bits	6 bits	4 bits	INT8 per row	14.6 GB	0.0074 / 0.0070

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Calibration trace	Generated by Qwen3.8-27B-pi itself (`reasoning_effort: xhigh`, MTP enabled) with exllamav3's `sc_trace` on 196 prompts: 157 from exllamav3's own `sc_trace` set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published); 196 sequences × 2048 tokens used for conversion
Reference and sensitivity	`sc_rfn_probe` against a 2.00 bpw reference, `sc_measure` (per-tensor sensitivity on the trace)
Bit allocation	`sc_optimize` at the target rate with a 5-bit `lm_head` → per-tensor recipe (`quantization_config.json`)

Tokenizer, chat template and generation config unchanged from bytkim/Qwen3.8-27B-pi (XML tool calls, qwen3_coder). The dflash2/ companion of the source repo is not included.

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers in vLLM through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). Its README describes it as a "correctness path".
qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/; 0.2.0 also adds EXL3 support for MoE experts, unused by this dense model): drops the separate attn.{q,k,v}_proj tensors of the EXL3 vision tower (vLLM uses the fused attn.qkv) and pads input dimensions to multiples of 128 for layers that need it (4304 → 4352).

Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto        # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB \  --served-model-name qwen3.8-27b-pi \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 215000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8420000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

3.70bpw branch, 226k context: the same command with --revision 3.70bpw --max-model-len 226000 --kv-cache-memory-bytes 8850000000.

Important notes

Memory. The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv, not as 24,576 − used. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). --max-num-batched-tokens 1024 and expandable_segments lower the peak by ≈350 MiB at no measurable cost. With MTP, KV capacity grows in block-sized steps: check the GPU KV cache size line (8,420,000,000 B → 216,433 tokens; 8,850,000,000 B → 227,439) and keep --max-model-len below it.
Measured peaks (2 sessions, vision): 3.85 bpw 311 MiB really free (719 MiB at rest); 3.70 bpw see the benchmark table. Leave the GPU for the LLM: a desktop session on the same GPU would eat this margin.
Hugging Face page warnings: the "Model size" shown by Hugging Face (≈8B params) counts the packed storage elements of the safetensors files (EXL3 packs ≈4 weights into each int16), not parameters: this is the 27B model at ≈3.85 bits per weight. The Configuration Parsing Warning "quantization_config.bits must be an integer" is also expected: for EXL3 is the average bits per weight of a per-tensor allocation (the per-tensor values are in ); exllamav3 and the OrcaSAQ2 plugin read it as a float.

Using it with opencode (how we tested)

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwen3.8-27b-pi": {          "name": "Qwen3.8-27B-pi EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 215000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 0.3 },    "plan":  { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 0.3 }  }}

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwen3.8-27b-pi": {          "name": "Qwen3.8-27B-pi EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 215000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 0.3 },    "plan":  { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 0.3 }  }}

limit.context = --max-model-len (226000 for the 3.70bpw branch); with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests). The agent tasks ran at temperature 0.3; the tool-call test at both T=1.0 (model default) and T=0.3.
Without attachment / modalities opencode will not send images; drop them if you serve with --language-model-only.
Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (</dev/null).

Benchmarks (single RTX 3090 24 GB, 300 W)

Table with columns: Test, main 3.85 bpw (215,000), 3.70bpw (226,000)
Test	`main` 3.85 bpw (215,000)	`3.70bpw` (226,000)
KV pool	216,433 tokens (KV 8.42e9 B)	227,439 tokens (KV 8.85e9 B)
8 facts in one long prompt (greedy)	8/8 at 61,675, 123,142 and 195,210 tokens (524 tok/s at 195k)	not measured
2 simultaneous sessions	2 × ≈94k, 8/8 in both, 698 tok/s aggregate	not measured
Decode, 1 request, t=0 / 0.6	76.8 / 75.5 tok/s (MTP: 70 % of drafts accepted)	not measured
Decode, 2 concurrent, t=0 / 0.6

For comparison, the base Qwen3.8-27B EXL3 3.85 reasoned 23,221 tokens at xhigh on the same problems (pi: 17,070, −26 %), with the same context and memory.

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images, short reasoning problems and agent tasks.
The full 262,144 context does not fit at this rate on 24 GB with MTP and vision.
Decode is ≈20 % slower than the W4A16 sibling on Ampere; the OrcaSAQ2 plugin is young and describes itself as a correctness path.
The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding; 0.2.0). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
LICENSE (Apache-2.0) and NOTICE (the pi model's notice, plus this quantization).

Feedback welcome

Credits & acknowledgements

Model: bytkim/Qwen3.8-27B-pi by bytkim (Thomas Kim), built on Qwen/Qwen3.8-27B (Qwen team).
exllamav3 and the EXL3 format by turboderp (MIT), including the SC pipeline, qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.