XReyRobert

Qwen3.8-27B-GPTQ-Pro-FOEM-4bit-g128-ns256

README

License: apache-2.0

Quantization recipe

Table with columns: Setting, Value
Setting	Value
Transformer body	GPTQ-Pro FOEM W4A16
Weight bits	4
Group size	128
Symmetric	Yes
Activation ordering	Disabled (`desc_act=false`)
Sequential quantization	Enabled
Calibration samples	256
Calibration sequence length	2048
Calibration mix	Code and reasoning JSONL
Quantizer	GPTQModel `6.1.0-dev`
Pack format	GPTQ, INT32 packing

The recorded recipe uses activation-weighted MSE, act-group-aware quantization, FOEM alpha=0.25 and beta=0.2, plus a 0.5% RTN fallback with MSE smoothing. Full machine-readable settings are available in quantize_config.json and config.json.

Preserved tensors

The following tensor families were excluded from the 4-bit GPTQ body and kept in their original precision:

token embeddings
lm_head
vision encoder tensors
MTP tensors
normalization tensors

The repository includes model-preserved-skips.safetensors as an auxiliary record of the 15 explicitly preserved tensors. The indexed checkpoint contains 2,399 weights totaling 19,559,449,368 bytes (approximately 18.22 GiB).

vLLM serving

The following text-only profile was validated on one RTX 3090 24 GB. The runtime was the local residency image localhost/vllm-openai-nightly-7a1eb8ac2-sleepwake-local:20260617, based on the pinned upstream vLLM nightly commit 7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8. Its local changes implement sleep/wake residency control; they do not alter this checkpoint's quantization format.

bash
vllm serve XReyRobert/Qwen3.8-27B-GPTQ-Pro-FOEM-4bit-g128-ns256 \
  --served-model-name qwen3.8-27b-gptq-pro-foem-4bit-g128-ns256-ctx110k \
  --language-model-only \
  --dtype float16 \
  --quantization gptq_marlin \
  --max-model-len 110592 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e5m2 \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --max-cudagraph-capture-size 32 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code

This validated profile does not use --enforce-eager. The 110,592-token limit is a single-RTX-3090 deployment result, not a limitation encoded in the checkpoint.

Vision and MTP weights are present, but enabling either feature changes the memory budget and requires a separately sized serving profile. Remove --language-model-only for vision use.

Sampling defaults

The checkpoint carries the Qwen3.8 thinking-mode defaults:

temperature=1.0
top_p=0.95
top_k=20

The validated vLLM deployment also used neutral min_p=0.0 and repetition_penalty=1.0 defaults.

Longer-context hybrid variant

For single-24-GB-GPU deployments requiring more KV-cache capacity, see XReyRobert/Qwen3.8-27B-GPTQ-Pro-FOEM-4bit-g128-ns256-INT8-Head-Embeddings. That derivative keeps this 4-bit GPTQ-Pro transformer body and additionally quantizes the token embeddings and lm_head to group-size-128 INT8. It was validated at 210K text-only context, 182K with vision, and 170K with MTP2 on one RTX 3090 24 GB. It requires the narrow vLLM compatibility patch included in that repository.

Validation status

The checkpoint loads with GPTQ-Marlin in the pinned vLLM nightly runtime.
OpenAI-compatible chat inference, Qwen reasoning parsing, tool-call parsing, prefix caching, and residency sleep/wake were exercised in deployment.
Text-only serving was validated with max_model_len=110592 on one RTX 3090.
The model card does not currently report standardized quality benchmarks.

Limitations

Quantization can reduce quality relative to the BF16 base model.
FP8 KV cache introduces numerical loss independently of weight quantization.
Native context length does not guarantee that every serving configuration fits in available VRAM.
Vision and MTP paths require separate memory and compatibility validation.
This checkpoint inherits the capabilities, risks, license, and intended-use guidance of the original Qwen3.8-27B model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

XReyRobert

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality