JRamirez-UAB

Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB

README

License: apache-2.0

What is in this repository

Variants / revisions

Table with columns: Revision, Linear layers (achieved), lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)
Revision	Linear layers (achieved)	`lm_head`	Vision	MTP head	Embeddings	Size	KLD (ours / pub.)
`main`	3.85 bpw (3.866), SC allocation	5 bits	6 bits	4 bits	INT8 per row	13.64 GiB	0.0060 / 0.0048

Made without YAQA (on our Swift EXL3 quantizations YAQA changed the KLD by 0–3 %).

Why there is no 3.70 bpw version of this model (unlike our Swift and Qwen3.8 EXL3 repos): we made and tested one (KLD 0.0073 / 0.0060 public, tools 59/60 · 60/60, images 18/18), but in the opencode agent tasks it fell into repetition loops in 3 of 8 runs, issuing the same command 33–70 times until our 400 s limit (for example re-running a command whose output goes to a file, seeing "(no output)" and running it again). The 3.85 bpw version did not loop in any run, nor did any other model we tested. Until we understand whether this comes from the quantization or from the model, we only publish 3.85 bpw.

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's qbench_prompts_gen.json, so they can be reproduced with exllamav3's qbench against the BF16 model.

Results

Family comparison (Qwopus3.8-27B-Flash-V2, single RTX 3090 24 GB)

Table with columns: Model, Size, Context in 24 GB, KLD mean / median, Perplexity (BF16: 1.6706 · pub. 1.4288), Needle · tools at T=1.0 · T=0.3 · agent, Reasoning xhigh (tokens), Prefill, Decode with MTP, 1 / 2 sessions, Tokens per step, Accepted drafts (of 3)
Model	Size	Context in 24 GB	KLD mean / median	Perplexity (BF16: 1.6706 · pub. 1.4288)	Needle · tools at T=1.0 · T=0.3 · agent	Reasoning xhigh (tokens)	Prefill	Decode with MTP, 1 / 2 sessions	Tokens per step	Accepted drafts (of 3)
W4A16 GPTQ + INT4 embeddings (`mtp-int8`)	14.2 GiB	222,000 (text) · 176,128 (vision)

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 Qwopus3.8-27B-Flash-V2, measured with exllamav3's qbench. The first figure uses an internal sysadmin/agent evaluation set (not published; separate from the calibration data, ≈30 % of it with tool calls); pub. uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3), so anyone can reproduce it. Noise floor on the public trace (BF16 against itself): 0.0004. The W4A16 checkpoint was evaluated as a BF16 copy holding the dequantized INT4 × scale weights (Transformers, full load); the EXL3 ones with exllamav3. Lower is better. Perplexity is on the same texts.
Context: largest --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak.
Needle: 8 facts spread through one prompt of that length, all 8 returned. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases (fresh request, after a long story, full multi-turn session) × 20 tries, xhigh, with the XML template and qwen3_coder, at T=1.0 (no temperature sent, so vLLM uses the model's generation_config.json) and at T=0.3 (what opencode sends with the configuration below). The only misses (all at T=1.0, all in the full-session case) were answers taken from the history ("the folder is empty") instead of a new tool call. : 4 sysadmin tasks × 2 with and real tool execution.

Benchmarks and validated configurations (single RTX 3090 24 GB, 350 W)

Quality: KL divergence against BF16 (2026-10-02)

Table with columns: Model, KLD mean / median (internal set), Perplexity (BF16 1.6706), KLD mean / median (public trace), Perplexity (BF16 1.4288)
Model	KLD mean / median (internal set)	Perplexity (BF16 1.6706)	KLD mean / median (public trace)	Perplexity (BF16 1.4288)
EXL3 3.85 bpw (`main`)	0.0060 / 0.0018	1.6815	0.0048 / 0.0004	1.4379
W4A16 GPTQ (sibling repo)	0.0357 / 0.0129	1.7436	0.0251 / 0.0024	1.4707

Service configuration (quick start above)

Table with columns: Test, main 3.85 bpw (215,000)
Test	`main` 3.85 bpw (215,000)
KV pool	216,433 tokens (KV 8.42e9 B)
Idle: used / really free	23,406 / 719 MiB
8 facts in one long prompt (greedy)	8/8 at 200,069 tokens, 371.3 s (539 tok/s)
2 simultaneous sessions	2 × ≈99k, 8/8 in both, 723 tok/s aggregate
Decode, 1 request, t=0 / 0.6	77.1 / 72.8 tok/s
Decode, 2 concurrent, t=0 / 0.6	141.0 / 131.6 tok/s
Images (4 synthetic tasks, 18 checks; 2026-10-02 battery)	4/4 tasks, 18/18; follow-up turn on the same image (prefix cache) 2.6 → 1.4 s

Images: failed-service screenshot, chart, dense firewall ruleset, and a terminal where the port is held by another process while the journal shows a dead PID as a trap (the model must take the right PID from the image and free the port). This battery replaces an older 3-image one (8/9), whose first image wrongly required that dead PID; the PID is still checked in a follow-up turn.

Reasoning and agent tasks

6 problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, total reasoning tokens over 48 requests; agent = 4 sysadmin tasks × 2 with opencode run (xhigh, temperature 0.3, real tool execution, reasoning counted by a proxy):

Table with columns: Model, low, medium, xhigh, Correct (low · medium · xhigh), Agent: ok · reasoning tokens · time
Model	low	medium	xhigh	Correct (low · medium · xhigh)	Agent: ok · reasoning tokens · time
EXL3 3.85 (`main`)	20,970	19,871	19,310	48 · 47 · 48	7/8 · 2,814 · 312 s
W4A16 GPTQ (sibling repo, 2026-10-02)	21,854	23,177	20,852	48 · 48 · 48	6/8 · 2,741 · 246 s

The only agent failures (one of two runs here, both runs with W4A16) were on the log-summary task. The level (low / medium / xhigh) does not regulate the amount of reasoning.

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)

Table with columns: Revision, Turn: time to first token / total (median), First turn (no cache), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0 / 0.6), Really free (min)
Revision	Turn: time to first token / total (median)	First turn (no cache)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0 / 0.6)	Really free (min)
`main` 3.85	3.2 / 4.2 s	13.7 s	8 × 23k, 8/8 in all, 972 tok/s	298.8 / 274.8 tok/s	145 MiB

Agent turn = the real opencode request (≈13.5k → 14.4k tokens over 10 turns) with prefix caching. For comparison, the W4A16 sibling needs 2.6 / 3.3 s per turn and reaches 460 tok/s with 8 sessions, but only up to a 200k pool and without vision.

MTP acceptance

On the mixed workload of the family table: 3.03 tokens per step, per-position acceptance 0.85 / 0.67 / 0.52.

Sampling and repetition loops

Measured on our EXL3 3.70 build of the same model (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens (a loop = more than 30 % repeated 8-grams).

Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Loops / 30	5	1	1	3	1	1	0	1

Few loops from 0.4 up (0–3 of 30), 5 at 0.3, and it finishes almost every answer well before 16,000 tokens: use 1.0, this repo's generation_config.json default, also in opencode. With the real opencode request its tool calls were the same at 0.3 and 1.0. Other models on the same test:

Table with columns: Model (same test, vLLM), T 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Model (same test, vLLM)	T 0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
NeoHorse-1-9B	2	3	3	1	2	0	0	0
Ornith-1.5-9B	5	7

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers in vLLM through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). Its README describes it as a "correctness path".
qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/; 0.2.0 also adds EXL3 support for MoE experts, unused by this dense model): drops the separate attn.{q,k,v}_proj tensors of the EXL3 vision tower (vLLM uses the fused attn.qkv) and pads input dimensions to multiples of 128 for layers that need it (4304 → 4352).

Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto        # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB --include "plugin/*" --include chat_template-xml-tools.jinja --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP) — our validated configuration:

bash

export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB \  --served-model-name qwopus3.8-27b \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 215000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8420000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈23k tokens at once, or one long session). Validated with 145 MiB really free at peak.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling is measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests are run directly against the vLLM API.

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwopus3.8-27b": {          "name": "Qwopus3.8-27B-Flash-V2 EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 215000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 },    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 }  }}

Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and with the real opencode request tool calls gave the same results at 1.0 as at 0.3.
limit.context = --max-model-len; with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests).
Tool calling needs --chat-template chat_template-xml-tools.jinja and --tool-call-parser qwen3_coder; with the original template and hermes opencode receives empty answers.

Important notes

For tool calling, use the base Qwen3.8 template with --tool-call-parser qwen3_coder (--chat-template chat_template-xml-tools.jinja). The original Qwopus V2 template asks for JSON tool calls (Hermes style), but the model often answers in Qwen3.8's XML format (<function=...><parameter=...>) or with malformed JSON, and the hermes parser silently drops those calls: the client receives an empty answer with finish_reason=stop. The two templates differ only in how tools and tool calls are rendered. On the W4A16 version: 59/60 with the XML template, 29/60 with the original one and hermes.
Memory. The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv, not as 24,576 − used. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). --max-num-batched-tokens 1024 and expandable_segments lower the peak by ≈350 MiB at no measurable cost. With MTP, KV capacity grows in block-sized steps: check the line (8,420,000,000 B → 216,433 tokens) and keep below it. (2 sessions, vision): 23,834 MiB used, .

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Calibration trace	Generated by Qwopus3.8-27B-Flash-V2 itself (`reasoning_effort: xhigh`, MTP enabled) with exllamav3's `sc_trace`: 196 prompts: 157 from exllamav3's own `sc_trace` set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published). The trace is the model's own `xhigh` answers to them, ≈620k generated tokens (277 rows: a second turn in 81 conversations); 250 sequences × 2048 tokens used for conversion
Reference and sensitivity	`sc_rfn_probe` against a 2.00 bpw reference, `sc_measure --shaped --max-sys 24` (per-tensor sensitivity on the trace)
Bit allocation	`sc_optimize` at the target rate with a 5-bit (, ) → per-tensor recipe (achieved 3.866 bpw for )

Tokenizer, chat template and generation config unchanged. For tool calling use the extra template chat_template-xml-tools.jinja (the base Qwen3.8 template, see Important notes).
Hardware: one RTX PRO 6000 Blackwell / L40S per job; conversion peaked at ≈33 GiB of RAM, sc_measure at ≈57 GiB.
sc_optimize predicted a KLD of 0.0060 for 3.85 bpw; the measured value is 0.0060.

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images, short reasoning problems and agent tasks.
The full 262,144 context does not fit at this rate on 24 GB with MTP and vision.
Decode is ≈20 % slower than our W4A16 GPTQ version on Ampere; the OrcaSAQ2 plugin is young and describes itself as a correctness path.
Qwopus is the weakest of our three 27B families as an agent (7/8 here, 6/8 for the W4A16); a 3.70 bpw version looped (see Variants).
The fp8 KV cache uses the default scale 1.0.
All numbers come from one machine and one software stack.

Feedback welcome

I would be grateful for reports of installations of this checkpoint, both working and failed ones, especially on GPUs other than the RTX 3090. Please open a discussion in the Community tab with your GPU(s), driver, vLLM / exllamav3 / plugin versions, the full command line and the result (speed, context, or the error message).

Credits & license

Credits & acknowledgements

Base model: Jackrong/Qwopus3.8-27B-Flash-V2, built on Qwen/Qwen3.8-27B.
exllamav3 and the EXL3 format by turboderp (MIT), including the SC pipeline, qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
Our W4A16 sibling builds on the INT4-embedding work of Marcin116, born2bewild and .

License

Apache-2.0, same as the base model. This is a community quantization, not an official release of Jackrong or Qwen. The OrcaSAQ2 plugin and exllamav3 are not redistributed here; install them from their repositories under their own licenses.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

Jackrong/Qwopus3.8-27B-Flash-V2

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What is in this repository

Variants / revisions

Table with columns: Revision, Linear layers (achieved), lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)
Revision	Linear layers (achieved)	`lm_head`	Vision	MTP head	Embeddings	Size	KLD (ours / pub.)
`main`	3.85 bpw (3.866), SC allocation	5 bits	6 bits	4 bits	INT8 per row	13.64 GiB	0.0060 / 0.0048

Made without YAQA (on our Swift EXL3 quantizations YAQA changed the KLD by 0–3 %).

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's qbench_prompts_gen.json, so they can be reproduced with exllamav3's qbench against the BF16 model.

Results

Family comparison (Qwopus3.8-27B-Flash-V2, single RTX 3090 24 GB)

Table with columns: Model, Size, Context in 24 GB, KLD mean / median, Perplexity (BF16: 1.6706 · pub. 1.4288), Needle · tools at T=1.0 · T=0.3 · agent, Reasoning xhigh (tokens), Prefill, Decode with MTP, 1 / 2 sessions, Tokens per step, Accepted drafts (of 3)
Model	Size	Context in 24 GB	KLD mean / median	Perplexity (BF16: 1.6706 · pub. 1.4288)	Needle · tools at T=1.0 · T=0.3 · agent	Reasoning xhigh (tokens)	Prefill	Decode with MTP, 1 / 2 sessions	Tokens per step	Accepted drafts (of 3)
W4A16 GPTQ + INT4 embeddings (`mtp-int8`)	14.2 GiB	222,000 (text) · 176,128 (vision)

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 Qwopus3.8-27B-Flash-V2, measured with exllamav3's qbench. The first figure uses an internal sysadmin/agent evaluation set (not published; separate from the calibration data, ≈30 % of it with tool calls); pub. uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3), so anyone can reproduce it. Noise floor on the public trace (BF16 against itself): 0.0004. The W4A16 checkpoint was evaluated as a BF16 copy holding the dequantized INT4 × scale weights (Transformers, full load); the EXL3 ones with exllamav3. Lower is better. Perplexity is on the same texts.
Context: largest --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak.
Needle: 8 facts spread through one prompt of that length, all 8 returned. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases (fresh request, after a long story, full multi-turn session) × 20 tries, xhigh, with the XML template and qwen3_coder, at T=1.0 (no temperature sent, so vLLM uses the model's generation_config.json) and at T=0.3 (what opencode sends with the configuration below). The only misses (all at T=1.0, all in the full-session case) were answers taken from the history ("the folder is empty") instead of a new tool call. : 4 sysadmin tasks × 2 with and real tool execution.

Benchmarks and validated configurations (single RTX 3090 24 GB, 350 W)

Quality: KL divergence against BF16 (2026-10-02)

Table with columns: Model, KLD mean / median (internal set), Perplexity (BF16 1.6706), KLD mean / median (public trace), Perplexity (BF16 1.4288)
Model	KLD mean / median (internal set)	Perplexity (BF16 1.6706)	KLD mean / median (public trace)	Perplexity (BF16 1.4288)
EXL3 3.85 bpw (`main`)	0.0060 / 0.0018	1.6815	0.0048 / 0.0004	1.4379
W4A16 GPTQ (sibling repo)	0.0357 / 0.0129	1.7436	0.0251 / 0.0024	1.4707

Service configuration (quick start above)

Table with columns: Test, main 3.85 bpw (215,000)
Test	`main` 3.85 bpw (215,000)
KV pool	216,433 tokens (KV 8.42e9 B)
Idle: used / really free	23,406 / 719 MiB
8 facts in one long prompt (greedy)	8/8 at 200,069 tokens, 371.3 s (539 tok/s)
2 simultaneous sessions	2 × ≈99k, 8/8 in both, 723 tok/s aggregate
Decode, 1 request, t=0 / 0.6	77.1 / 72.8 tok/s
Decode, 2 concurrent, t=0 / 0.6	141.0 / 131.6 tok/s
Images (4 synthetic tasks, 18 checks; 2026-10-02 battery)	4/4 tasks, 18/18; follow-up turn on the same image (prefix cache) 2.6 → 1.4 s

Reasoning and agent tasks

Table with columns: Model, low, medium, xhigh, Correct (low · medium · xhigh), Agent: ok · reasoning tokens · time
Model	low	medium	xhigh	Correct (low · medium · xhigh)	Agent: ok · reasoning tokens · time
EXL3 3.85 (`main`)	20,970	19,871	19,310	48 · 47 · 48	7/8 · 2,814 · 312 s
W4A16 GPTQ (sibling repo, 2026-10-02)	21,854	23,177	20,852	48 · 48 · 48	6/8 · 2,741 · 246 s

The only agent failures (one of two runs here, both runs with W4A16) were on the log-summary task. The level (low / medium / xhigh) does not regulate the amount of reasoning.

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)

Table with columns: Revision, Turn: time to first token / total (median), First turn (no cache), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0 / 0.6), Really free (min)
Revision	Turn: time to first token / total (median)	First turn (no cache)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0 / 0.6)	Really free (min)
`main` 3.85	3.2 / 4.2 s	13.7 s	8 × 23k, 8/8 in all, 972 tok/s	298.8 / 274.8 tok/s	145 MiB

MTP acceptance

On the mixed workload of the family table: 3.03 tokens per step, per-position acceptance 0.85 / 0.67 / 0.52.

Sampling and repetition loops

Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Loops / 30	5	1	1	3	1	1	0	1

Table with columns: Model (same test, vLLM), T 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Model (same test, vLLM)	T 0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
NeoHorse-1-9B	2	3	3	1	2	0	0	0
Ornith-1.5-9B	5	7

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers in vLLM through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). Its README describes it as a "correctness path".
qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/; 0.2.0 also adds EXL3 support for MoE experts, unused by this dense model): drops the separate attn.{q,k,v}_proj tensors of the EXL3 vision tower (vLLM uses the fused attn.qkv) and pads input dimensions to multiples of 128 for layers that need it (4304 → 4352).

Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto        # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB --include "plugin/*" --include chat_template-xml-tools.jinja --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP) — our validated configuration:

bash

export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB \  --served-model-name qwopus3.8-27b \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 215000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8420000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈23k tokens at once, or one long session). Validated with 145 MiB really free at peak.

Using it with opencode (how we tested)

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwopus3.8-27b": {          "name": "Qwopus3.8-27B-Flash-V2 EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 215000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 },    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 }  }}

Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and with the real opencode request tool calls gave the same results at 1.0 as at 0.3.
limit.context = --max-model-len; with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests).
Tool calling needs --chat-template chat_template-xml-tools.jinja and --tool-call-parser qwen3_coder; with the original template and hermes opencode receives empty answers.

Important notes

For tool calling, use the base Qwen3.8 template with --tool-call-parser qwen3_coder (--chat-template chat_template-xml-tools.jinja). The original Qwopus V2 template asks for JSON tool calls (Hermes style), but the model often answers in Qwen3.8's XML format (<function=...><parameter=...>) or with malformed JSON, and the hermes parser silently drops those calls: the client receives an empty answer with finish_reason=stop. The two templates differ only in how tools and tool calls are rendered. On the W4A16 version: 59/60 with the XML template, 29/60 with the original one and hermes.
Memory. The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv, not as 24,576 − used. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). --max-num-batched-tokens 1024 and expandable_segments lower the peak by ≈350 MiB at no measurable cost. With MTP, KV capacity grows in block-sized steps: check the line (8,420,000,000 B → 216,433 tokens) and keep below it. (2 sessions, vision): 23,834 MiB used, .

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Calibration trace	Generated by Qwopus3.8-27B-Flash-V2 itself (`reasoning_effort: xhigh`, MTP enabled) with exllamav3's `sc_trace`: 196 prompts: 157 from exllamav3's own `sc_trace` set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published). The trace is the model's own `xhigh` answers to them, ≈620k generated tokens (277 rows: a second turn in 81 conversations); 250 sequences × 2048 tokens used for conversion
Reference and sensitivity	`sc_rfn_probe` against a 2.00 bpw reference, `sc_measure --shaped --max-sys 24` (per-tensor sensitivity on the trace)
Bit allocation	`sc_optimize` at the target rate with a 5-bit (, ) → per-tensor recipe (achieved 3.866 bpw for )

Tokenizer, chat template and generation config unchanged. For tool calling use the extra template chat_template-xml-tools.jinja (the base Qwen3.8 template, see Important notes).
Hardware: one RTX PRO 6000 Blackwell / L40S per job; conversion peaked at ≈33 GiB of RAM, sc_measure at ≈57 GiB.
sc_optimize predicted a KLD of 0.0060 for 3.85 bpw; the measured value is 0.0060.

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images, short reasoning problems and agent tasks.
The full 262,144 context does not fit at this rate on 24 GB with MTP and vision.
Decode is ≈20 % slower than our W4A16 GPTQ version on Ampere; the OrcaSAQ2 plugin is young and describes itself as a correctness path.
Qwopus is the weakest of our three 27B families as an agent (7/8 here, 6/8 for the W4A16); a 3.70 bpw version looped (see Variants).
The fp8 KV cache uses the default scale 1.0.
All numbers come from one machine and one software stack.

Feedback welcome

Credits & license

Credits & acknowledgements

Base model: Jackrong/Qwopus3.8-27B-Flash-V2, built on Qwen/Qwen3.8-27B.
exllamav3 and the EXL3 format by turboderp (MIT), including the SC pipeline, qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
Our W4A16 sibling builds on the INT4-embedding work of Marcin116, born2bewild and .

Qwopus3.8-27B-Flash-V2-EXL3-3.85bpw-24GB

README

What is in this repository

Variants / revisions

Files and reproducibility

Results

Family comparison (Qwopus3.8-27B-Flash-V2, single RTX 3090 24 GB)

Benchmarks and validated configurations (single RTX 3090 24 GB, 350 W)

Quality: KL divergence against BF16 (2026-10-02)

Service configuration (quick start above)

Reasoning and agent tasks

Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)

MTP acceptance

Sampling and repetition loops

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

Quick start (single RTX 3090, 24 GB)

Using it with opencode (how we tested)

Important notes

Quantization recipe

Limitations

Feedback welcome

Credits & license

Credits & acknowledgements

License

Explore FriendliAI today

README

What is in this repository

Variants / revisions

Files and reproducibility

Results

Family comparison (Qwopus3.8-27B-Flash-V2, single RTX 3090 24 GB)

Benchmarks and validated configurations (single RTX 3090 24 GB, 350 W)

Quality: KL divergence against BF16 (2026-10-02)

Service configuration (quick start above)

Reasoning and agent tasks

Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)

MTP acceptance

Sampling and repetition loops

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

Quick start (single RTX 3090, 24 GB)

Using it with opencode (how we tested)

Important notes

Quantization recipe

Limitations

Feedback welcome

Credits & license

Credits & acknowledgements

License

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)