johannrplaster

Qwen3.8-27B-Uncensored-int4-AutoRound

README

License: apache-2.0

Quantization

Table with columns: Setting, Value
Setting	Value
Method	AutoRound 0.14.2, default tuning recipe
Format	W4A16, symmetric, group_size 128, `auto_round:auto_gptq` packing
Calibration	batch_size 4, gradient_accumulate_steps 2
Quantized	all decoder layers + the MTP head
Kept FP16/BF16	GDN `linear_attn.in_proj_a/b`, norms, `mtp.fc`, vision tower
Size on disk	~18 GB (7 shards, MTP tensors included in shard 7)

The checkpoint is directly servable as GPTQ INT4 (Marlin / XPUwNa16LinearKernel path in vLLM).

Measured performance (Intel Arc Pro B70)

Cold-response measurement discipline: fixed realistic prompt suite, each prompt run once, prefix/prompt caching disabled, cached_tokens=0 on every request; primary metric is the median inter-token rate over tokens 1-100 after TTFT.

Table with columns: Serving config, Decode (tok/s, median)
Serving config	Decode (tok/s, median)
2x B70 (TP2), MTP3, FP16 KV	81.8
2x B70 (TP2), MTP3, FP16 KV, experimental INT8 LM head	94.6
1x B70, MTP3, FP16 KV	57.5
1x B70, MTP3, FP16 KV, experimental INT8 LM head	66.8

Quality gates on this exact checkpoint: arithmetic, deterministic code evaluation, copy, factual, JSON-schema, long-context needle, and repeat-stability checks all pass; all exact canaries are identical to the FP16-head reference.

Notes on interpretation: warmed, repeated-prompt, same-shape benchmarks of this stack read much higher (80+ on a single card); those numbers describe repeat traffic, not fresh requests.

Serving (vLLM XPU)

bash
vllm serve /path/to/this/model \
  --quantization gptq \
  --dtype float16 \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-prompt-tokens-details \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

Single-GPU serving needs no tensor parallelism flags; the model fits in one 32 GB B70 with FP16 KV at moderate context.
FP8 KV cache (--kv-cache-dtype fp8) works and is the capacity option for 128K context on one card; FP16 KV decodes slightly faster.
The MTP head accepts num_speculative_tokens 1-4. Depth 3 was the best measured trade-off on fresh-response workloads; depth 4 can win on highly predictable content (acceptance is workload-dependent — check vllm:spec_decode_num_accepted_tokens_total against spec_decode_num_draft_tokens_total on your own traffic).
Optional experimental lever: an INT8-quantized LM head (W8A8) served via a patched vllm-xpu-kernels build adds ~15% decode on this model class; the stock FP16 head path is what the 81.8/57.5 rows above use.

Chat template

Ships the stock Qwen3.8 chat template (chat_template.jinja; reasoning and tool-call blocks follow the base model's conventions). Note for serving: vLLM's thinking_token_budget (or an equivalent hard cap) is the reliable way to bound reasoning length; prompt-level effort instructions are a hint the model can ignore on hard prompts.

Limitations

This is an uncensored model: it will answer prompts that safety-tuned models refuse. You are responsible for how you use it; review generated content before relying on it.
INT4 quantization trades some quality for speed and size. The preserved FP16 projection families mitigate the worst regression we measured, but outputs are not bit-identical to the BF16 original, and stylistic casing differences on short answers have been observed.
Benchmarks above are single-stream; concurrent serving aggregates higher.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

johannrplaster

Model Tree

Base

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality