Marcin116

Qwen3.8-27B-Uncensored-W4A16-RTX3090-MTP4-embed-int4

README

License: apache-2.0

Provenance

The model lineage is:

Qwen/Qwen3.8-27B — original Apache-2.0 model.
orcarouter/Qwen3.8-27B-Uncensored — abliterated derivative.
noon-at-cgn/Qwen3.8-27B-Uncensored-W4A16-AutoRound — AutoRound W4A16 quantization.
This checkpoint — further quantization and MTP serving optimization.

The immediate source revision was 0e10c9f6b5b8a97fba199e82c49690d272f776ce. The original abliteration and the initial W4A16 quantization were performed by the respective upstream authors, not by this repository's uploader.

Variant: INT4 token embeddings

This repository is derived from protogeni/Qwen3.8-27B-Uncensored-W4A16-RTX3090-MTP4. The only change is the token-embedding group, re-quantized from INT8 to INT4; it is the same single-tensor reduction as the Marcin116/Qwen3.8-27B-W4A16-AutoRound-fast-embed-int4 variant, applied here to the uncensored model.

Table with columns: Tensor (shard model-00006-of-00007.safetensors), protogeni (original), this repo (variant)
Tensor (shard `model-00006-of-00007.safetensors`)	protogeni (original)	this repo (variant)
`model.language_model.embed_tokens.weight_packed`	INT8 `[248320, 1280]`	INT4 `[248320, 640]`
`model.language_model.embed_tokens.weight_scale`	BF16 `[248320, 40]`	BF16 `[248320, 40]` (recomputed)
`model.language_model.embed_tokens.weight_shape`

Same pack-quantized, group-128 symmetric layout (AutoRound/syvai/vLLM compatible): 8× INT4 values packed per INT32 word ([248320, 640]), one BF16 scale per 128-element group ([248320, 40]). weight_shape is unchanged, so the loader reconstructs the full [248320, 5120] embedding matrix.
config.json marks the group: group_2.targets = ["re:.*embed_tokens$"], weights = {type: int, symmetric: true, group_size: 128, num_bits: 4}. The inline group_2.weights.num_bits field is the only byte changed in config.json (8 → 4); the standalone quantization_config.json (which lists only the INT4 Linear/lm_head/MTP groups) is left untouched.

Quantization source. The INT4 codes are produced from the uncensored BF16 embeddings (the abliterated orcarouter/Qwen3.8-27B-Uncensored token embed_tokens, the same ground truth the source INT8 embedding was quantized from) — not the base Qwen embeddings. The abliterated model has fine-tuned/abliterated embeddings, so the BF16 uncensored weights are the correct reference. Method: per-group (128) absolute-max scaling, scale = clamp(absmax/7, min=1e-10), round + clamp to [-7, +7] (symmetric, QMAX = 7, no zero-point) — identical to the AutoRound-compatible packing used for every INT4 group in the model.

Size. The INT4 embedding halves that tensor from ~1.29 GB to ~0.656 GB, saving ~0.635 GB. Whole repository on disk: 15.18 GB (vs ~15.81 GB for the INT8-embedding source; a BF16-embedding build would be ~17.7 GB).

Measured quantization error. Full embed_tokens dequantized and compared to the BF16 uncensored ground-truth weights (symmetric per-group error, full 248,320 × 5,120 matrix):

Table with columns: vs uncensored BF16 ground truth, rel. RMS error, cosine similarity
vs uncensored BF16 ground truth	rel. RMS error	cosine similarity
INT8 (original protogeni)	0.669 %	0.999978
INT4 (this repo)	11.77 %	0.993120

The INT4 embedding's relative error is ~18× the INT8's, but the absolute magnitude is still small (per-token-vector RMS error ≈ 0.0009 vs a value RMS of ≈ 0.0073; per-vector cosine ≈ 0.993). Embeddings are read once per token (not per depth layer), and the remainder of the network is already W4A16, so this error is an order of magnitude below that of the many INT4 linear layers — the practical impact on output quality is expected to be minor. The INT4 scale range is [1.85e-6, 0.083] with no zero scales.

What changed

Transformer body: retained from the source checkpoint as symmetric W4A16, group size 128.
Token embeddings: symmetric INT8, group size 128 — in this INT4-embedding variant this group is re-quantized to INT4; see the “Variant: INT4 token embeddings” section above.
lm_head: calibrated symmetric GPTQ INT4, group size 128.
Native MTP module: calibrated symmetric GPTQ INT4, group size 128.
Draft head: 40,960 rows sliced from the calibrated INT4 lm_head.
Draft vocabulary: selected from this checkpoint's own generated output-token distribution instead of being inherited from the aligned Qwen checkpoint.
Vision tower and multimodal processor files are retained.

The calibration run generated 6,761 responses containing 5,500,353 output tokens. The resulting draft vocabulary covered 97.77% of a held-out set of 571,632 output tokens. Hidden-state calibration used 1,200 sequences and 1,273,013 rows. The lm_head INT4 calibration measured KL 0.00308 with GPTQ, compared with 0.00695 for round-to-nearest.

Evaluation evidence

The linked MP4 is a 1.0× replay of a real streamed request using its recorded SSE chunk timestamps, with synchronized GPU telemetry. It is not a fabricated typing animation. The raw trace, evaluator-native results, scripts, exact version pins, and scoring notes are published in evaluation/.

Measured RTX 3090 Ti profile

Measurements below were made locally on an NVIDIA GeForce RTX 3090 Ti (24 GB) with the patched vLLM 0.27.1 runtime from syv-ai/qwen38-27b-rtx3090:

Table with columns: Measurement, Result
Measurement	Result
Serving profile	MTP-4, BF16 KV, FlashAttention 2, prefix cache
Maximum configured model length	80,000 tokens
Startup KV capacity	80,392 tokens
Isolated fixed 512-token non-thinking decode	143.1 tok/s mean, 151.0 median
Isolated decode range	95.5–181.2 tok/s across 5 realistic prompts
Aggregate output throughput	243.1 tok/s at C2; 282.6 tok/s at C4
Speed-run MTP yield	3.20 emitted tokens/verification step
Speed-run per-position MTP acceptance

Decode rate is measured from the first non-empty streamed content chunk to the last, excluding time-to-first-token; every request produced exactly 512 output tokens. Aggregate throughput includes cohort wall time. These are single-machine serving measurements rather than hardware-independent model benchmarks; prompt, sampling, driver, runtime, clocks, thermals, and GPU model all affect the result.

Quality evaluation

Table with columns: Evaluation, Result, Configuration
Evaluation	Result	Configuration
GSM8K	94.77% (1,250/1,319)	thinking off; Wilson 95% CI 93.43–95.85%
MMLU-Pro	76.14% (1,066/1,400)	5-shot, thinking on, 4,096-token cap
HumanEval	94.51% pass@1 (155/164)	thinking off
HumanEval+	90.85% pass@1 (149/164)	thinking off
IFBench	39.33% strict / 42.00% loose	all 300 prompts, thinking off, current pinned verifier

The MMLU-Pro result matches the immediate upstream W4A16 model card's reported 76.1% to the displayed precision. This is encouraging evidence that the serving-specific head calibration did not create a measurable additional loss on this benchmark, but it is not a controlled ablation of quantization and abliteration effects.

The current pinned IFBench revision contains verifier fixes made after the much higher historical result in the runtime repository. Those values are not treated as directly comparable. The lower current score is published here deliberately because this checkpoint still needs broader independent testing.

MMMU is a clear weakness. Its 24.56% result is near the chance region, and 891/900 responses exhausted the official 128-token generation cap instead of following the task's direct-answer instruction. The retained vision tower makes multimodal input functional, but this evidence does not establish strong vision quality for this derivative.

Recommended serving setup

The measured MTP performance requires the patched runtime and launch profile in syv-ai/qwen38-27b-rtx3090. The checkpoint's MTP and reduced draft vocabulary are not a promise of identical performance with an unpatched stock vLLM installation.

bash
git clone https://github.com/syv-ai/qwen38-27b-rtx3090.git
cd qwen38-27b-rtx3090

hf download protogeni/Qwen3.8-27B-Uncensored-W4A16-RTX3090-MTP4 \
  --local-dir models/Qwen3.8-27B-Uncensored-W4A16-RTX3090-MTP4

  # For this INT4-embedding variant instead, download:
  # hf download Marcin116/Qwen3.8-27B-Uncensored-W4A16-RTX3090-MTP4-embed-int4 \
  #   --local-dir models/Qwen3.8-27B-Uncensored-W4A16-RTX3090-MTP4-embed-int4

printf '%s\n' \
  'MODEL=/app/models/Qwen3.8-27B-Uncensored-W4A16-RTX3090-MTP4' \
  'CTX=fast' \
  'SPEC=mtp' \
  'PREFIX_CACHE=1' \
  'GPU_UTIL=0.95' \
  'MAX_LEN=80000' \
  'MAX_SEQS=4' > .env

docker compose --profile single up -d --no-deps single

The first launch can take several minutes while kernels compile and CUDA graphs are captured. Reduce GPU_UTIL, MAX_LEN, or both if the GPU is also driving a desktop or if another process needs VRAM. The exact 80,000-token profile was validated on the uploader's otherwise dedicated 3090 Ti; a display workload can reduce the available margin.

Validation notes

Core chat, reasoning, tool/JSON, streaming, logprobs, penalties, vision, and long-prompt API tests passed locally.
The long-context suite placed distinct targets at three depths in prompts of approximately 8k, 32k, 64k, and 76k tokens. All 12 were retrieved exactly.
A paired 50-prompt IFBench study showed a practical cost to reasoning in this serving profile: with a 2,048-token completion cap, thinking-on exhausted the budget before producing a final answer on 34/50 prompts. Exact-format tasks should normally use thinking off, while reasoning tasks need a larger budget.
MTP acceptance is workload-dependent. Speculative verification preserves the target distribution; acceptance changes speed, not model intelligence.
The current patched runtime's optimized multimodal path was not stable enough for a 900-item benchmark: full-run attempts using MTP and target-only CUDA graphs stopped making progress after roughly 50 completed requests, and MTP dry runs varied with scheduling. MMMU therefore measures the exact target checkpoint with a conservative eager target-only profile. Text MTP serving and the dedicated MTP speed tests remain unaffected.
Seeded probabilistic speculative decoding was not byte-deterministic in the local smoke suite.
The model has no independent safety layer. The absence of refusal must never be interpreted as correctness, legality, or safety.

License and attribution

Released under Apache License 2.0, inherited from the Qwen and intermediate checkpoints. See LICENSE. This model card identifies the upstream works and gives a prominent description of the modifications made to the checkpoint.

This INT4-embedding derivative was produced by Marcin116; it reuses the upstream lineage and serving setup above unchanged, and is released under the same Apache-2.0 license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Marcin116

Model Tree

Base

noon-at-cgn/Qwen3.8-27B-Uncensored-W4A16-AutoRound

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality