XReyRobert

Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings

README

License: apache-2.0

Provenance

Base model: Qwen/Qwen3.8-27B
Base model revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Source quant: dbirks/Qwen3.8-27B-W4A16-AutoRound
Source checkpoint revision: 1f05c441c4e64ae0549de44fa9ea5a6d43610314
Transformation recipe: syv-ai/qwen38-27b-rtx3090
Recipe revision: e681a8f6b190cca1f86d58b8b333b64d014994d7
Output format: compressed-tensors pack-quantized
Body: symmetric INT4 W4A16, group size 128
Vocabulary matrices: symmetric INT8, group size 128

The exact transformation scripts and vLLM patch used are included in this repository.

Important vLLM requirement

The INT8 token embedding table requires a small vLLM patch. Stock vLLM 0.27.1 contains the dequantization kernel but does not pass the quantization config to Qwen3.5/Qwen3.8's VocabParallelEmbedding constructors.

Apply the included patch to the installed vLLM package:

bash
patch -p1 -d /path/to/site-packages/vllm \
  < vllm-qwen3_5-embed-quant.patch

The patch was written and verified against vLLM 0.27.1. Revalidate it after upgrading vLLM. Without this patch, the checkpoint is not expected to load correctly.

Text-only serving on a 24 GB GPU

The following profile mirrors the batch-oriented syv-ai recipe:

bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve XReyRobert/Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings \
  --served-model-name qwen3.8-27b-autoround-int8-head-embeddings \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --max-model-len 150000 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.972 \
  --async-scheduling \
  --compilation-config '{"max_cudagraph_capture_size":64,"custom_ops":["+rms_norm","+silu_and_mul"]}' \
  --trust-remote-code

--language-model-only avoids loading the vision tower. The published syv-ai measurements use an RTX 3090 and describe 150k as the conservative deployment context; the reported single-request ceiling after requantization is around 195k. Actual capacity depends on the runtime version, CUDA allocator state, concurrency and prompt shape.

For single-user speculative decoding, use the MTP profile documented in the source recipe and ensure the second hunk of the included patch is applied.

Sampling and thinking control

Use the source model's recommended sampling parameters:

Table with columns: Mode, Temperature, Top-p, Top-k, Min-p, Presence penalty, Repetition penalty
Mode	Temperature	Top-p	Top-k	Min-p	Presence penalty	Repetition penalty
Thinking	`1.0`	`0.95`	`20`	`0.0`	`0.0`	`1.0`

Qwen3.8 supports reasoning_effort=low, medium, or xhigh; xhigh is the source-model default. For agentic sessions, enable thinking and preserve prior reasoning through the chat template:

json
{"enable_thinking": true, "preserve_thinking": true}

Keep the combined input, retained reasoning, and generated output inside the served context limit. The 150K profile above is smaller than the model's native 262,144-token position limit because it is sized for one 24 GB GPU.

Reproduction

Starting from the pinned source revision:

bash
python quant_lm_head.py /path/to/checkpoint
python quant_embed.py /path/to/checkpoint

The scripts rewrite the relevant safetensors shards, update model.safetensors.index.json, and add dedicated compressed-tensors config groups for lm_head and embed_tokens.

Validation performed

Safetensors/index consistency: 2,003 indexed tensors, 2,003 present
Missing indexed tensors: 0
Extra tensors in indexed shards: 0
lm_head relative round-trip error: 0.0064
embed_tokens relative round-trip error: 0.0056

Terminal-Bench 2.0 sanity check

The published artifact was exercised with Terminus 2 on three tasks (openssl-selfsigned-cert, headless-terminal, and count-dataset-tokens) using one RTX 3090, one concurrent trial, 32 CPU cores, 48 GiB RAM, and a 30-minute task timeout.

Table with columns: Effort, Verifier reward, Run wall, Visible output, vLLM output, Parser warnings, Decode, Prefill, Agent timeouts
Effort	Verifier reward	Run wall	Visible output	vLLM output	Parser warnings	Decode	Prefill	Agent timeouts
`low`	3/3	36.5 min	43,988	60,696	18	46.9 tok/s	1,175 tok/s	0

The xhigh timeout occurred after active generation and is retained as an observed benchmark outcome even though the verifier reward was already 1 for that task. Parser warnings indicate extra or discarded model text around tool actions; they do not automatically imply an incorrect tool call.

This is a serving and agent-path smoke check, not a broad quality ranking. Three tasks are too few for comparative accuracy claims. Refer to the source AutoRound model card for body-quantization evaluations and to the syv-ai repository for the original INT8 head/embedding serving benchmarks.

Credits

Qwen team for Qwen3.8-27B
dbirks for the W4A16 AutoRound checkpoint
syv-ai for the INT8 head/embedding transformation and vLLM serving recipe

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

XReyRobert

Model Tree

Base

dbirks/Qwen3.8-27B-W4A16-AutoRound

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality