JRamirez-UAB

Qwen3.8-27B-GPTQ-W4A16-g64-embed-int8-24GB

README

License: apache-2.0

Which one to use

Table with columns: Checkpoint, Weights, Context in 24 GB, KLD internal set · public trace, Decode with MTP, 1 / 2 sessions, Reasoning at xhigh (48 problems)
Checkpoint	Weights	Context in 24 GB	KLD internal set · public trace	Decode with MTP, 1 / 2 sessions	Reasoning at `xhigh` (48 problems)
W4A16 g64 + INT8 embeddings (this repo)	15.7 GiB	176,000 (text)	0.0175 · 0.0171	81.2 / 157.3 tok/s	23,269 tokens
W4A16 g128 + INT4 embeddings	14.2 GiB	222,000 (text)	0.0400 · 0.0312	94.4 / 187 tok/s	26,481 tokens
EXL3 3.85 bpw	13.6 GiB	215,000 (with vision)	0.0073 · 0.0067	75.9 / 135.8 tok/s	23,221 tokens

This repo: the most faithful W4A16 on stock vLLM (+ a 4-line patch); choose it when quality matters more than the last 50k tokens of context.
INT4 embeddings: the longest W4A16 context and a little faster decode.
EXL3: closest to BF16 and with vision at 215k, but slower decode on Ampere and it needs exllamav3 + two plugins.
The KLD figures of the three rows use the same internal evaluation set; decode and reasoning were measured on different days with the same tests (the reasoning totals vary ±10 % between runs).

Quantization recipe ("v4")

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 64, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT8 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`,

Source: Qwen/Qwen3.8-27B.
Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens: 196 conversations from an internal Linux sysadmin / agent trace (multi-turn, with tool calls and reasoning; not published) plus 316 samples of HuggingFaceH4/ultrachat_200k (train_sft, seed 42), shuffled. Only GPTQ uses it; the INT8 parts are RTN.
Tokenizer and chat template are identical to the base model.
Scripts: scripts/quantize_v4.py (with CAL_TRACE you can mix in your own trace; without it, UltraChat only) and scripts/mtp_quant.py (INT8 MTP head).

Requirements & install: vLLM with a small patch

Stock vLLM (0.29.0 / 0.30.0) does not pass quant_config to the embedding layer of qwen3_5 models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Apply the 4-line patch in patches/qwen3_5-embed-quant.patch (from syv-ai/qwen38-27b-rtx3090, Apache-2.0):

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.29.0" --torch-backend cu130 \    --extra-index-url https://flashinfer.ai/whl/ --index-strategy unsafe-best-matchVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')hf download JRamirez-UAB/Qwen3.8-27B-GPTQ-W4A16-g64-embed-int8-24GB patches/qwen3_5-embed-quant.patch --local-dir .patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

Text only, MTP 3, 176k context, 2 concurrent sequences (validated configuration):

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwen3.8-27B-GPTQ-W4A16-g64-embed-int8-24GB \  --served-model-name qwen3.8-27b \  --language-model-only \  --max-model-len 176000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7090000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Important notes

Memory (read this before changing the numbers)

The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use. "24,576 − used" over-estimates the headroom by that amount; check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv.
The configuration above left 177 MiB really free at peak when installed exactly as described here (fresh venv, model downloaded from this repo, 155k-token prompt and 2 × 76k concurrent); our own build peaked at 23,854 MiB used / 271 MiB free (415 MiB free at rest). Both are above our 100 MiB safety margin, but there is little room: do not raise the KV cache. --max-num-batched-tokens 1024 and expandable_segments are part of that budget: without them the prefill peak is several hundred MiB higher.
Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization: with this hybrid (GatedDeltaNet) model the memory profiling under-estimates the prefill activations and a percentage over-allocates the KV cache.
With MTP the KV capacity grows in block-sized steps: check the GPU KV cache size line; --max-model-len must not exceed it.
Our RTX 3090 hung once (driver , Xid 154) during a 168k-token prefill of this checkpoint at a 350 W power limit; at 300 W with a more aggressive fan curve the same test passed and all the numbers here are from that setup. If your card is marginal, lower its power limit.

Other notes

Use --tool-call-parser qwen3_coder. On Ampere (sm86), fp8 KV cache only works with the FlashInfer attention backend (the default).
reasoning_effort accepts only xhigh, medium and low (high returns HTTP 400); they are prompt instructions, not limits. For a hard cap send thinking_token_budget in the request (vLLM 0.29 enforces it with --reasoning-parser qwen3, also with MTP).
For use from opencode, the configuration in the INT4-embedding card applies with "limit": { "context": 176000, "output": 22000 } (declare "temperature": true in the model, or opencode sends no temperature).

Benchmarks (single RTX 3090 24 GB, 300 W, 2026-10-06)

Table with columns: Test, Result
Test	Result
KV pool	178,816 tokens (`--max-model-len 176000`, KV 7.09e9 B)
8 facts spread over 161,646 tokens (greedy)	8/8, 251 s (644 tok/s prefill)
2 concurrent sessions of 79k tokens	8/8 and 8/8, 841 tok/s aggregate prefill
Decode, 1 request, t=0 / t=0.6	81.2 / 93.2 tok/s
Decode, 2 concurrent, aggregate, t=0 / t=0.6	157.3 / 166.6 tok/s
MTP mean acceptance length (tokens per step, max 4)	3.27
Real opencode request, tool calls (`xhigh`, 3 cases × 5)	, no parse errors

Vision (the quick-start command without --language-model-only, with --max-model-len 131072 --kv-cache-memory-bytes 5500000000 --limit-mm-per-prompt '{"image":1,"video":0}' --mm-processor-kwargs '{"max_pixels":1003520}'): KV pool 132,423 tokens; 4-image battery 15/15, follow-up turn with prefix cache OK; needle 8/8 at 115,483 tokens; tool calls with the real opencode request 15/15; 869 MiB really free at peak.

Quality: KL divergence against BF16 (2026-10-05)

Measured with exllamav3's qbench, first on an internal sysadmin/agent evaluation set (not published; different conversations from the calibration trace, but from the same domain), the W4A16 checkpoint evaluated as a BF16 copy holding the dequantized weights (Transformers):

Table with columns: Checkpoint, KLD mean, median, p90, Perplexity (BF16 1.5860)
Checkpoint	KLD mean	median	p90	Perplexity (BF16 1.5860)
This repo (v4)	0.0175	0.0049	0.0476	1.6203
INT4 embeddings, g128, UltraChat calibration	0.0400	0.0127	—	1.6631

On turboderp's public evaluation trace (qbench_prompts_gen.json, the one used in all our cards, unrelated to our calibration data) the gain holds: KLD 0.0171 (median 0.0009, perplexity 1.3568) against 0.0312 (median 0.0018) for the INT4-embedding checkpoint, so it is not just an effect of calibrating and evaluating on the same domain.

On Swift 1.5 we isolated the steps (internal set): INT8 embeddings / lm_head 0.0467 → 0.0396, + mixed calibration 0.0251, + group size 64 0.0215.

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence (above), long-context retrieval, tool calling, short reasoning problems and agent tasks.
176k is less than the 262k the model supports; with 24 GB, use the INT4-embedding or EXL3 checkpoints for longer contexts.
The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.

Files

model.safetensors (+ index), model_mtp.safetensors (INT8 MTP head), config.json (compressed-tensors), recipe.yaml (as serialized by llm-compressor).
scripts/quantize_v4.py, scripts/mtp_quant.py; patches/ (vLLM patch and its Apache-2.0 license); SHA256SUMS.

Feedback welcome

Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).

Credits

Base model: Qwen/Qwen3.8-27B.
vLLM patch: syv-ai; quantized-embedding layout first shown by Marcin116.
Tools: llm-compressor, compressed-tensors, vLLM; KL divergence with qbench from exllamav3.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Which one to use

Table with columns: Checkpoint, Weights, Context in 24 GB, KLD internal set · public trace, Decode with MTP, 1 / 2 sessions, Reasoning at xhigh (48 problems)
Checkpoint	Weights	Context in 24 GB	KLD internal set · public trace	Decode with MTP, 1 / 2 sessions	Reasoning at `xhigh` (48 problems)
W4A16 g64 + INT8 embeddings (this repo)	15.7 GiB	176,000 (text)	0.0175 · 0.0171	81.2 / 157.3 tok/s	23,269 tokens
W4A16 g128 + INT4 embeddings	14.2 GiB	222,000 (text)	0.0400 · 0.0312	94.4 / 187 tok/s	26,481 tokens
EXL3 3.85 bpw	13.6 GiB	215,000 (with vision)	0.0073 · 0.0067	75.9 / 135.8 tok/s	23,221 tokens

This repo: the most faithful W4A16 on stock vLLM (+ a 4-line patch); choose it when quality matters more than the last 50k tokens of context.
INT4 embeddings: the longest W4A16 context and a little faster decode.
EXL3: closest to BF16 and with vision at 215k, but slower decode on Ampere and it needs exllamav3 + two plugins.
The KLD figures of the three rows use the same internal evaluation set; decode and reasoning were measured on different days with the same tests (the reasoning totals vary ±10 % between runs).

Quantization recipe ("v4")

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 64, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT8 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`,

Source: Qwen/Qwen3.8-27B.
Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens: 196 conversations from an internal Linux sysadmin / agent trace (multi-turn, with tool calls and reasoning; not published) plus 316 samples of HuggingFaceH4/ultrachat_200k (train_sft, seed 42), shuffled. Only GPTQ uses it; the INT8 parts are RTN.
Tokenizer and chat template are identical to the base model.
Scripts: scripts/quantize_v4.py (with CAL_TRACE you can mix in your own trace; without it, UltraChat only) and scripts/mtp_quant.py (INT8 MTP head).

Requirements & install: vLLM with a small patch

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.29.0" --torch-backend cu130 \    --extra-index-url https://flashinfer.ai/whl/ --index-strategy unsafe-best-matchVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')hf download JRamirez-UAB/Qwen3.8-27B-GPTQ-W4A16-g64-embed-int8-24GB patches/qwen3_5-embed-quant.patch --local-dir .patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

Text only, MTP 3, 176k context, 2 concurrent sequences (validated configuration):

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwen3.8-27B-GPTQ-W4A16-g64-embed-int8-24GB \  --served-model-name qwen3.8-27b \  --language-model-only \  --max-model-len 176000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7090000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Important notes

Memory (read this before changing the numbers)

The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use. "24,576 − used" over-estimates the headroom by that amount; check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv.
The configuration above left 177 MiB really free at peak when installed exactly as described here (fresh venv, model downloaded from this repo, 155k-token prompt and 2 × 76k concurrent); our own build peaked at 23,854 MiB used / 271 MiB free (415 MiB free at rest). Both are above our 100 MiB safety margin, but there is little room: do not raise the KV cache. --max-num-batched-tokens 1024 and expandable_segments are part of that budget: without them the prefill peak is several hundred MiB higher.
Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization: with this hybrid (GatedDeltaNet) model the memory profiling under-estimates the prefill activations and a percentage over-allocates the KV cache.
With MTP the KV capacity grows in block-sized steps: check the GPU KV cache size line; --max-model-len must not exceed it.
Our RTX 3090 hung once (driver , Xid 154) during a 168k-token prefill of this checkpoint at a 350 W power limit; at 300 W with a more aggressive fan curve the same test passed and all the numbers here are from that setup. If your card is marginal, lower its power limit.

Other notes

Use --tool-call-parser qwen3_coder. On Ampere (sm86), fp8 KV cache only works with the FlashInfer attention backend (the default).
reasoning_effort accepts only xhigh, medium and low (high returns HTTP 400); they are prompt instructions, not limits. For a hard cap send thinking_token_budget in the request (vLLM 0.29 enforces it with --reasoning-parser qwen3, also with MTP).
For use from opencode, the configuration in the INT4-embedding card applies with "limit": { "context": 176000, "output": 22000 } (declare "temperature": true in the model, or opencode sends no temperature).

Benchmarks (single RTX 3090 24 GB, 300 W, 2026-10-06)

Table with columns: Test, Result
Test	Result
KV pool	178,816 tokens (`--max-model-len 176000`, KV 7.09e9 B)
8 facts spread over 161,646 tokens (greedy)	8/8, 251 s (644 tok/s prefill)
2 concurrent sessions of 79k tokens	8/8 and 8/8, 841 tok/s aggregate prefill
Decode, 1 request, t=0 / t=0.6	81.2 / 93.2 tok/s
Decode, 2 concurrent, aggregate, t=0 / t=0.6	157.3 / 166.6 tok/s
MTP mean acceptance length (tokens per step, max 4)	3.27
Real opencode request, tool calls (`xhigh`, 3 cases × 5)	, no parse errors

Quality: KL divergence against BF16 (2026-10-05)

Table with columns: Checkpoint, KLD mean, median, p90, Perplexity (BF16 1.5860)
Checkpoint	KLD mean	median	p90	Perplexity (BF16 1.5860)
This repo (v4)	0.0175	0.0049	0.0476	1.6203
INT4 embeddings, g128, UltraChat calibration	0.0400	0.0127	—	1.6631

On Swift 1.5 we isolated the steps (internal set): INT8 embeddings / lm_head 0.0467 → 0.0396, + mixed calibration 0.0251, + group size 64 0.0215.

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence (above), long-context retrieval, tool calling, short reasoning problems and agent tasks.
176k is less than the 262k the model supports; with 24 GB, use the INT4-embedding or EXL3 checkpoints for longer contexts.
The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.

Files

model.safetensors (+ index), model_mtp.safetensors (INT8 MTP head), config.json (compressed-tensors), recipe.yaml (as serialized by llm-compressor).
scripts/quantize_v4.py, scripts/mtp_quant.py; patches/ (vLLM patch and its Apache-2.0 license); SHA256SUMS.

Feedback welcome

Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).

Credits

Base model: Qwen/Qwen3.8-27B.
vLLM patch: syv-ai; quantized-embedding layout first shown by Marcin116.
Tools: llm-compressor, compressed-tensors, vLLM; KL divergence with qbench from exllamav3.