JRamirez-UAB

Qwen3.8-27B-pi-GPTQ-W4A16-g64-embed-int8-24GB

README

License: apache-2.0

Which one to use

Table with columns: Checkpoint, Weights, Context in 24 GB, KLD internal set · public trace, Decode with MTP, 1 / 2 sessions, Reasoning at xhigh (48 problems)
Checkpoint	Weights	Context in 24 GB	KLD internal set · public trace	Decode with MTP, 1 / 2 sessions	Reasoning at `xhigh` (48 problems)
W4A16 g64 + INT8 embeddings (this repo)	16.9 GB	176,000 (text)	0.0182 · 0.0170	93.9 / 168.6 tok/s	18,183 tokens (48/48)
EXL3 3.85 bpw	14.6 GB	215,000 (with vision)	0.0074 · 0.0070	76.8 / 135.2 tok/s	17,070 tokens (46/48)

This repo: faster decode on Ampere and stock vLLM (+ a 4-line patch); text only in the validated configuration.
EXL3: ≈2.5× closer to BF16, vision and 215k context, but slower decode on Ampere and it needs exllamav3 + two plugins.

Quantization recipe ("v4")

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 64, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT8 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`,

Source: bytkim/Qwen3.8-27B-pi.
Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens: conversations from a sysadmin / agent trace generated by the pi model (multi-turn, with tool calls and reasoning; prompts not published) plus samples of HuggingFaceH4/ultrachat_200k (train_sft, seed 42), shuffled. Only GPTQ uses it; the INT8 parts are RTN.
Tokenizer, chat template and generation config are identical to the source model.
Scripts: scripts/quantize_v4.py and scripts/mtp_quant.py (INT8 MTP head).

Requirements & install: vLLM with a small patch

Stock vLLM (0.29.0 / 0.30.0) does not pass quant_config to the embedding layer of qwen3_5 models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Apply the 4-line patch in patches/qwen3_5-embed-quant.patch (from syv-ai/qwen38-27b-rtx3090, Apache-2.0):

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.29.0" --torch-backend cu130 \    --extra-index-url https://flashinfer.ai/whl/ --index-strategy unsafe-best-matchVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')hf download JRamirez-UAB/Qwen3.8-27B-pi-GPTQ-W4A16-g64-embed-int8-24GB patches/qwen3_5-embed-quant.patch --local-dir .patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

Text only, MTP 3, 176k context, 2 concurrent sequences (validated configuration):

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwen3.8-27B-pi-GPTQ-W4A16-g64-embed-int8-24GB \  --served-model-name qwen3.8-27b-pi \  --language-model-only \  --max-model-len 176000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7090000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Important notes

Memory (read this before changing the numbers)

The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use. "24,576 − used" over-estimates the headroom by that amount; check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv.
The configuration above left 179 MiB really free at peak on this model (417 MiB at rest). It is above our 100 MiB safety margin, but there is little room: do not raise the KV cache. --max-num-batched-tokens 1024 and expandable_segments are part of that budget: without them the prefill peak is several hundred MiB higher.
Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization: with this hybrid (GatedDeltaNet) model the memory profiling under-estimates the prefill activations and a percentage over-allocates the KV cache.
With MTP the KV capacity grows in block-sized steps: check the GPU KV cache size line; --max-model-len must not exceed it.
Our RTX 3090 hung once (driver GSP RM heartbeat timed out, Xid 154) during a 168k-token prefill of this checkpoint at a 350 W power limit; at 300 W with a more aggressive fan curve the same test passed and all the numbers here are from that setup. If your card is marginal, lower its power limit.

Other notes

Use --tool-call-parser qwen3_coder. On Ampere (sm86), fp8 KV cache only works with the FlashInfer attention backend (the default).
reasoning_effort accepts only xhigh, medium and low (high returns HTTP 400); they are prompt instructions, not limits. For a hard cap send thinking_token_budget in the request (vLLM 0.29 enforces it with --reasoning-parser qwen3, also with MTP).
For use from opencode, the configuration in the Qwen3.8 INT4-embedding card applies with "limit": { "context": 176000, "output": 22000 } (declare "temperature": true in the model, or opencode sends no temperature).

Benchmarks (single RTX 3090 24 GB, 300 W, 2026-10-09)

Table with columns: Test, Result
Test	Result
KV pool	178,816 tokens (`--max-model-len 176000`, KV 7.09e9 B)
8 facts in one long prompt (greedy)	8/8 at 61,675, 123,142 and 157,733 tokens (656 tok/s prefill at 157k)
2 concurrent sessions of 75k tokens	8/8 and 8/8, 853 tok/s aggregate prefill
Decode, 1 request, t=0 / t=0.6	93.9 / 91.4 tok/s (MTP: 73 % of drafts accepted)
Decode, 2 concurrent, aggregate, t=0 / t=0.6	168.6 / 163.9 tok/s
Real opencode request, tool calls (3 cases × 10 × 3 configurations)	90/90, no parse errors
Reasoning tokens, 6 checkable problems × 8 (T 0.6): low · medium · xhigh	17,861 (48/48) · 19,187 (48/48) · 18,183 (48/48)

Quality: KL divergence against BF16

Measured with exllamav3's qbench (the W4A16 checkpoint evaluated as a BF16 copy holding the dequantized weights, Transformers):

Table with columns: Evaluation set, KLD mean, median, p90, Perplexity (quantized / BF16)
Evaluation set	KLD mean	median	p90	Perplexity (quantized / BF16)
Internal sysadmin/agent set (not published)	0.0182	0.0041	0.0499	1.5900 / 1.5561
turboderp's public `qbench_prompts_gen.json`	0.0170	0.0007	0.0492	1.3514 / 1.3265

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence (above), long-context retrieval, tool calling, short reasoning problems and agent tasks. Vision was not validated on this checkpoint (the BF16 vision tower is included).
176k is less than the 262k the model supports; with 24 GB, use the EXL3 checkpoint for longer contexts.
The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.

Files

model.safetensors (+ index), model_mtp.safetensors (INT8 MTP head), config.json (compressed-tensors), recipe.yaml (as serialized by llm-compressor).
scripts/quantize_v4.py, scripts/mtp_quant.py; patches/ (vLLM patch and its Apache-2.0 license); LICENSE, NOTICE.

Feedback welcome

Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).

Credits

Model: bytkim/Qwen3.8-27B-pi by bytkim (Thomas Kim), built on Qwen/Qwen3.8-27B.
vLLM patch: syv-ai; quantized-embedding layout first shown by Marcin116.
Tools: llm-compressor, compressed-tensors, vLLM; KL divergence with qbench from exllamav3.

License

Apache-2.0, same as the pi model and Qwen3.8-27B; the pi model's NOTICE is kept in NOTICE.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

bytkim/Qwen3.8-27B-pi

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Which one to use

Table with columns: Checkpoint, Weights, Context in 24 GB, KLD internal set · public trace, Decode with MTP, 1 / 2 sessions, Reasoning at xhigh (48 problems)
Checkpoint	Weights	Context in 24 GB	KLD internal set · public trace	Decode with MTP, 1 / 2 sessions	Reasoning at `xhigh` (48 problems)
W4A16 g64 + INT8 embeddings (this repo)	16.9 GB	176,000 (text)	0.0182 · 0.0170	93.9 / 168.6 tok/s	18,183 tokens (48/48)
EXL3 3.85 bpw	14.6 GB	215,000 (with vision)	0.0074 · 0.0070	76.8 / 135.2 tok/s	17,070 tokens (46/48)

This repo: faster decode on Ampere and stock vLLM (+ a 4-line patch); text only in the validated configuration.
EXL3: ≈2.5× closer to BF16, vision and 215k context, but slower decode on Ampere and it needs exllamav3 + two plugins.

Quantization recipe ("v4")

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 64, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT8 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`,

Source: bytkim/Qwen3.8-27B-pi.
Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens: conversations from a sysadmin / agent trace generated by the pi model (multi-turn, with tool calls and reasoning; prompts not published) plus samples of HuggingFaceH4/ultrachat_200k (train_sft, seed 42), shuffled. Only GPTQ uses it; the INT8 parts are RTN.
Tokenizer, chat template and generation config are identical to the source model.
Scripts: scripts/quantize_v4.py and scripts/mtp_quant.py (INT8 MTP head).

Requirements & install: vLLM with a small patch

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.29.0" --torch-backend cu130 \    --extra-index-url https://flashinfer.ai/whl/ --index-strategy unsafe-best-matchVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')hf download JRamirez-UAB/Qwen3.8-27B-pi-GPTQ-W4A16-g64-embed-int8-24GB patches/qwen3_5-embed-quant.patch --local-dir .patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

Text only, MTP 3, 176k context, 2 concurrent sequences (validated configuration):

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwen3.8-27B-pi-GPTQ-W4A16-g64-embed-int8-24GB \  --served-model-name qwen3.8-27b-pi \  --language-model-only \  --max-model-len 176000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7090000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Important notes

Memory (read this before changing the numbers)

The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use. "24,576 − used" over-estimates the headroom by that amount; check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv.
The configuration above left 179 MiB really free at peak on this model (417 MiB at rest). It is above our 100 MiB safety margin, but there is little room: do not raise the KV cache. --max-num-batched-tokens 1024 and expandable_segments are part of that budget: without them the prefill peak is several hundred MiB higher.
Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization: with this hybrid (GatedDeltaNet) model the memory profiling under-estimates the prefill activations and a percentage over-allocates the KV cache.
With MTP the KV capacity grows in block-sized steps: check the GPU KV cache size line; --max-model-len must not exceed it.
Our RTX 3090 hung once (driver GSP RM heartbeat timed out, Xid 154) during a 168k-token prefill of this checkpoint at a 350 W power limit; at 300 W with a more aggressive fan curve the same test passed and all the numbers here are from that setup. If your card is marginal, lower its power limit.

Other notes

Use --tool-call-parser qwen3_coder. On Ampere (sm86), fp8 KV cache only works with the FlashInfer attention backend (the default).
reasoning_effort accepts only xhigh, medium and low (high returns HTTP 400); they are prompt instructions, not limits. For a hard cap send thinking_token_budget in the request (vLLM 0.29 enforces it with --reasoning-parser qwen3, also with MTP).
For use from opencode, the configuration in the Qwen3.8 INT4-embedding card applies with "limit": { "context": 176000, "output": 22000 } (declare "temperature": true in the model, or opencode sends no temperature).

Benchmarks (single RTX 3090 24 GB, 300 W, 2026-10-09)

Table with columns: Test, Result
Test	Result
KV pool	178,816 tokens (`--max-model-len 176000`, KV 7.09e9 B)
8 facts in one long prompt (greedy)	8/8 at 61,675, 123,142 and 157,733 tokens (656 tok/s prefill at 157k)
2 concurrent sessions of 75k tokens	8/8 and 8/8, 853 tok/s aggregate prefill
Decode, 1 request, t=0 / t=0.6	93.9 / 91.4 tok/s (MTP: 73 % of drafts accepted)
Decode, 2 concurrent, aggregate, t=0 / t=0.6	168.6 / 163.9 tok/s
Real opencode request, tool calls (3 cases × 10 × 3 configurations)	90/90, no parse errors
Reasoning tokens, 6 checkable problems × 8 (T 0.6): low · medium · xhigh	17,861 (48/48) · 19,187 (48/48) · 18,183 (48/48)

Quality: KL divergence against BF16

Measured with exllamav3's qbench (the W4A16 checkpoint evaluated as a BF16 copy holding the dequantized weights, Transformers):

Table with columns: Evaluation set, KLD mean, median, p90, Perplexity (quantized / BF16)
Evaluation set	KLD mean	median	p90	Perplexity (quantized / BF16)
Internal sysadmin/agent set (not published)	0.0182	0.0041	0.0499	1.5900 / 1.5561
turboderp's public `qbench_prompts_gen.json`	0.0170	0.0007	0.0492	1.3514 / 1.3265

Limitations

Quality was not evaluated with standard benchmarks; the checks are KL divergence (above), long-context retrieval, tool calling, short reasoning problems and agent tasks. Vision was not validated on this checkpoint (the BF16 vision tower is included).
176k is less than the 262k the model supports; with 24 GB, use the EXL3 checkpoint for longer contexts.
The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.

Files

model.safetensors (+ index), model_mtp.safetensors (INT8 MTP head), config.json (compressed-tensors), recipe.yaml (as serialized by llm-compressor).
scripts/quantize_v4.py, scripts/mtp_quant.py; patches/ (vLLM patch and its Apache-2.0 license); LICENSE, NOTICE.

Feedback welcome

Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).

Credits

Model: bytkim/Qwen3.8-27B-pi by bytkim (Thomas Kim), built on Qwen/Qwen3.8-27B.
vLLM patch: syv-ai; quantized-embedding layout first shown by Marcin116.
Tools: llm-compressor, compressed-tensors, vLLM; KL divergence with qbench from exllamav3.

License

Apache-2.0, same as the pi model and Qwen3.8-27B; the pi model's NOTICE is kept in NOTICE.