Qwen3.8-27B-AEON-UNCENSORED-INT8-W8A16-MTP API & Inference Endpoint

Relationship to the sibling quantization

This checkpoint is the AEON Uncensored counterpart of lued/Qwen3.8-27B-INT8-W8A16-MTP. The two repositories share:

the identical W8A16 compressed-tensors recipe (400 packed GEMMs, same BF16 preservation set, group-128 symmetric data-free RTN);
the identical native BF16 MTP head (15 tensors, byte-identical to Qwen's original MTP);
the identical tokenizer, chat template, processor configs, and 6-shard layout;
the identical validated serving profile (TP2, 262144, fp8_e4m3 KV, MTP × 3).

The only deviation is the base language-model weights: this checkpoint's LM weights come from the abliterated AEON BF16 model instead of stock Qwen3.8-27B. Serving performance is unchanged by design and was measured on this exact checkpoint. Behavior — specifically the refusal profile — is the intended deviation and is documented below.

Abliteration (inherited from the base)

The AEON BF16 source was abliterated with abliterix 1.12.2 (a Heretic-derived multi-objective Optuna tool with hybrid Mamba/attention support), judged by google/gemini-3.1-flash-lite. Winning trial 48 / 50 was selected for the coherent unlock, not the lowest-KL point. The pipeline was:

text
Qwen/Qwen3.8-27B
        ↓
  SSM conv1d outlier repair (FernflowerAI)
        ↓
  abliterix 1.12.2, 50-trial Optuna, Flash Lite judge, trial 48/50
        ↓
  MTP head grafted back from stock (15 tensors, hash-match)
  Vision tower untouched (333/333 hash-match)
        ↓
AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16

The base card documents a measured smash KL of 0.0991 nats/token (full_distribution_kl, first 3 teacher-forced tokens on 100 harmless held-out prompts) — intentionally large, because this abliteration optimized for coherent answers over stock-like openings, and 0 outright refusals (hard "I won't") across its 250-prompt eval sets. The full refusal breakdown, per-category numbers, and trial details live in the base repository.

The base card's 0-outright-refusal claim was re-verified in an operator-run smoke test of this exact checkpoint across the tested prompt set: zero refusals, consistent with the base card. The W8A16 step does not change the refusal profile: refusal elimination is a property of the base weights, and the quantization touches only the stored numerics of the 400 GEMMs.

Dual RTX 3090 deployment

Validated host profile: 2×RTX 3090 24 GB, PCIe without NVLink or P2P, vLLM TP2, BF16 activations, FP8 E4M3 KV cache, MTP with three draft tokens, and --max-num-batched-tokens 8192.

Memory and capacity

Table with columns: Metric, Result
Metric	Result
Checkpoint size	31.6 GB / 29.44 GiB
Loaded model memory per GPU	14.85 GiB
Shared GPU KV pool	266,537 tokens
Native maximum request length	262,144 tokens
Simultaneous full-native-context capacity	1.02×
Illustrative four-way KV share	Approximately 66,634 tokens per request

The checkpoint has the same size and footprint as the sibling quantization, since the W8A16 policy and shard layout are identical.

--max-model-len caps one request's input plus generated tokens. The KV pool is a shared runtime allocation across all live requests, not a per-lane reservation: --max-num-seqs permits that many active sequences but does not provide that many full contexts. Near the 262K limit, reserve output-token headroom and expect only one full-length request at a time.

The open-source checkpoint is native at 262,144 tokens. Qwen documents extension to 1,000,000 tokens via static YaRN, but this deployment's 266,537-token KV pool cannot hold a 1M-token request even with YaRN enabled; the 1M path is not validated or capacity-supported here. Follow the official YaRN configuration only with a runtime and memory plan sized for the requested context.

Measured serving performance

Measured on 2026-08-16 with llama-benchy 0.4.0 against this exact checkpoint served through vLLM: one concurrent request, exact generation lengths (min_tokens + ignore_eos), cold prefix cache (unique corpus text, cache_prompt=false), no conversation depth, one run per cell.

Table with columns: Prompt tokens, Generation tokens, Prefill tok/s, Generation tok/s, TTFT
Prompt tokens	Generation tokens	Prefill tok/s	Generation tok/s	TTFT
1,024	512	2,087	69.9	0.6 s
1,024	1,024	1,951	98.8	0.6 s
8,192	512	1,733	60.8	4.8 s

Generation throughput is the per-request decode rate with MTP enabled and varies by workload, since draft acceptance is prompt-dependent; each cell is a single run, so per-cell variance (notably the 1,024/1,024 cell) is expected. Prefill and TTFT are governed by the 8,192-token batched-token cap, which chunks long prompts and interleaves the MTP draft pass; TTFT grows with prompt length, roughly linearly past 32K.

Multi-stream throughput is not yet published: concurrent requests can intermittently crash the engine on this pin, so parallel benchmarks will be added once a vLLM release containing vllm#50021 is available (see the concurrency note under Serving).

Quantization fidelity

Two different fidelity questions apply to this checkpoint, and they should not be conflated:

1. What the W8A16 step costs (quantization error). Measured on this exact checkpoint against its own BF16 teacher (the AEON source) on 2026-08-17 with tools/measure_kld.py (teacher-forced, full-vocabulary FP32 log-softmax, FP64 accumulation, nats/token):

Table with columns: Candidate, Mean KLD vs BF16, Top-1 agreement, Weight files
Candidate	Mean KLD vs BF16	Top-1 agreement	Weight files
Official Qwen FP8 / BF16 A16 (sibling measurement)	0.004396	98.53%	28.75 GiB
This W8 / BF16 A16	0.001106	98.80%	29.44 GiB

The KLD result measures checkpoint-weight drift over 3,755 teacher-forced positions (8-prompt short suite plus a 3,288-token probe of the AEON source card). It is not a functional quality score; tool use, JSON-schema compliance, coding, multimodal quality, and long-context recall require separate behavioral evaluation. The quantization policy is per-tensor and data-free, so the expected error class matches the sibling's 0.000894; this checkpoint's own number is slightly higher, consistent with its different weight distribution after abliteration.

2. What the model deviates from stock Qwen3.8-27B by (intended). That is the abliteration in the section above — smash KL 0.0991 and 0/250 outright refusals, measured and documented by the base author. It is a property of the base weights, not of the quantization step.

Checkpoint profile

Table with columns: Property, Value
Property	Value
Quantization	Data-free symmetric RTN W8A16, group size 128
Runtime format	`compressed-tensors` / `pack-quantized`
Kernel dispatch	`CompressedTensorsWNA16` → `MarlinLinearKernel` (verified in server logs)
Preserved precision	BF16 vision tower, `lm_head`, MTP (15 tensors, byte-identical to Qwen's original MTP), and recurrent GDN gates
MTP	BF16 draft model detected at runtime; embeddings and `lm_head` shared with the target

Why W8A16 on Ampere

RTX 3090 GPUs are Ampere sm_86. They do not provide the native FP8 tensor-core execution path available on newer GPU generations. In this deployment, FP8 weight-only and INT8 W8A16 checkpoints both reach Marlin-family weight-only kernels that reconstruct weights for 16-bit GEMMs.

W8A16 is therefore the better fidelity default here: it preserves eight integer bits with group-wise scaling while retaining the same practical decode class as FP8-Marlin on these GPUs.

Table with columns: Format, Role on 2×24 GB at long context, Decision
Format	Role on 2×24 GB at long context	Decision
W8A16 — this checkpoint	Near-lossless weights; production default	Recommended
Official / online FP8 weight quantization	Convenient, but higher measured weight error on sm_86 fallback	Useful comparison
W8A8 INT8	Can improve prefill throughput; may regress skinny-batch decode	Specialized profile
W4A16	Smaller weights and more cache headroom	Higher long-context risk for recurrent GDN layers

Why not AutoRound?

This checkpoint uses llm-compressor's QuantizationModifier, not AutoRound. At W8A16, AutoRound's installed RTN configuration disables optimized RTN by default for efficiency, and the measured KLD on the sibling recipe is already below 9e-4 nats/token. Re-encoding these weights with AutoRound would add a new export and serving path without evidence of a meaningful quality gain.

Quantization design

Architecture audit

The BF16 source was checked before quantization against the Qwen3.6 structural contract:

64 language layers: 48 Gated DeltaNet linear-attention layers and 16 full-attention layers.
Hidden size 5,120; intermediate size 17,408; padded vocabulary 248,320.
One MTP layer with the same 15 top-level mtp.* tensors.
Stable GDN projection names: in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, and out_proj.
The only relevant rename was non-Linear convNd → conv1d, outside the quantization target.

The pre-quantization structural audit completed successfully before any weights were modified.

Recipe

Table with columns: Component, Precision, Reason
Component	Precision	Reason
MLP projections	INT8 W8A16	Largest dense GEMMs; strong memory return
Full-attention projections	INT8 W8A16	Measured low output-distribution error
GDN `in_proj_qkv`, `in_proj_z`, `out_proj`	INT8 W8A16	Recovers approximately 4 GB while remaining stable in the 4K GDN probe
GDN `in_proj_a`, `in_proj_b`

The result contains 400 quantized Linear GEMMs: 192 MLP projections, 64 full-attention projections, and 144 dense GDN projections.

yaml
# recipe.yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['lm_head', 're:.*visual.*', 're:.*mtp.*', 're:.*linear_attn[.]in_proj_a$', 're:.*linear_attn[.]in_proj_b$']
      scheme: W8A16
      bypass_divisibility_checks: false

Reproduction contract

The immutable source is AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 commit a6775a9a8ebb65cab3f707b4ab087fc7aa698634. The exact selective W8A16 configuration is recorded in recipe.yaml and config.json. A conforming rebuild must take native MTP from that source, reproduce the source-derived packed projection set, and prove all 15 MTP tensors bit-identical before resharing for publication.

Quantization fidelity: KLD versus the BF16 source

[!IMPORTANT] This section measures checkpoint-weight quantization. It is not a BF16-versus-FP16 benchmark and it does not measure the FP8 KV cache. The teacher is always the original BF16 checkpoint (the AEON source).

For every teacher-forced next-token position, the metric is the KL divergence from the BF16 teacher distribution to the candidate distribution, D_KL(p_BF16 ‖ p_candidate), computed over the full 248,320-token vocabulary, in nats/token, with FP32 log_softmax and FP64 accumulation. No sampling or top-k approximation is involved.

Overall results

Table with columns: Candidate checkpoint, Stored weights, A16 dtype in measurement, Mean KLD vs BF16, Top-1 agreement
Candidate checkpoint	Stored weights	A16 dtype in measurement	Mean KLD vs BF16	Top-1 agreement
BF16 source	BF16	BF16	0 by definition	100%
This checkpoint	INT8 symmetric, groups of 128	BF16 after INT8×scale	0.001106	98.80%

The sibling recipe measured the same policy against the Qwen teacher at 0.000894 W8/BF16 and 0.004396 for official FP8 (see the summary table in Dual RTX 3090 deployment); those numbers use the Qwen BF16 teacher, so they are family references, not directly comparable to this checkpoint's number, which uses the AEON BF16 teacher. The AEON number is slightly higher than the sibling's, consistent with the different weight distribution after abliteration.

Short-prompt suite

Table with columns: Prompt, Category, Tokens, Positions, This W8 / BF16 A16, Top-1 agreement
Prompt	Category	Tokens	Positions	This W8 / BF16 A16	Top-1 agreement
`factual`	factual	13	12	0.000460	1.0000
`code-fib`	code	87	86	0.000539	0.9884

Long-context GDN probe

The probe used the first 3,288 tokenizer tokens of the AEON source model card as cached locally, producing 3,288 scored positions.

Table with columns: Candidate, Mean KLD, Top-1 agreement
Candidate	Mean KLD	Top-1 agreement
This W8 / BF16 A16	0.001163	98.72%

KLD did not accumulate across this probe; no recurrent drift was observed at 3.3K tokens.

factual: The capital of France is Paris, and the capital of Japan is

code-fib:

python
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

# Rewrite with memoization to avoid exponential blowup, then add a
# test that checks fibonacci(10) == 55 and fibonacci(0) == 0. Use a
# dict as the cache and make the function private to the module.

physics-uncertainty: In quantum mechanics, the uncertainty principle states that the product of the uncertainties in position and momentum cannot be smaller than about half of the reduced Planck constant. More precisely, if many identically prepared particles are measured, the standard deviations of position and momentum satisfy the inequality derived by Kennard in 1927, which is a direct consequence of the non-commutativity of the position and momentum operators in Hilbert space.
math-train: A train travels 240 kilometres in 3 hours and 15 minutes, stopping for 10 minutes at one station along the way. What is its average speed in kilometres per hour? Show your calculation step by step, and state whether the stop changes the average speed.
:

Run locally on 2026-08-17 with identical teacher-tokenizer IDs.
BF16 teacher snapshot: a6775a9a8ebb65cab3f707b4ab087fc7aa698634.
Environment: 2×RTX 3090, PyTorch 2.11.0+cu128, Transformers 5.10.1, compressed-tensors 0.17.1.
The student loads packed and forwards with per-module on-the-fly dequantization. These are Hugging Face decompressed-path logits, not vLLM Marlin kernel logits.
use_cache=False: the serving configuration's FP8 KV-cache error is not included.
No scheduling, sampling, image, or context beyond 3,288 tokens was measured.
Teacher logits were cached as FP16 before the shared FP32 KLD calculation, introducing a small common precision floor.

MTP preservation

The MTP head is the exact BF16 MTP from the original Qwen3.8-27B — never touched by the abliteration (which ran on the language layers only) and never re-quantized:

15 tensors in model_mtp.safetensors (849,400,392 bytes).
sha256 1d8268aa85ace093a561e3e7b63b9d390dac1cd55a90cd55b5ec509c3c9da9fe — identical to the MTP bytes in the Qwen3.8-27B source and in the AEON source's model-00003-of-00003.safetensors.
All 15 tensors verified bfloat16, shape-equal, and bit-equal to the AEON source (strict audit, not a count check).

KV-cache rationale: E4M3 versus E5M2

--kv-cache-dtype controls runtime attention-cache storage. It does not alter this checkpoint's INT8 weights. E4M3 and E5M2 both consume one byte per cached value, so this decision affects numerical behavior—not memory capacity.

Table with columns: KV dtype, Mantissa bits, Maximum finite value, Minimum subnormal, Practical tradeoff
KV dtype	Mantissa bits	Maximum finite value	Minimum subnormal	Practical tradeoff
`fp8_e4m3`	3	±448	2^-9 ≈ 0.00195	Better in-range precision
`fp8_e5m2`	2	±57,344	2^-16 ≈ 0.0000153	Much wider dynamic range

E4M3 is the default because attention K/V values generally benefit more from its additional mantissa bit than from E5M2's extreme range.

[!WARNING] Runtime compatibility is verified; long-context KV accuracy is not. The checkpoint declares kv_cache_scheme: null, so it contains no dataset-calibrated K/V scales. The KLD measurement above uses no cache, and no saturation probe was performed.

E5M2 remains the fallback if measured ranges or a future vLLM build expose E4M3 saturation or kernel incompatibility.

Serving

GPU interconnect profiles

The validated command below targets stock-driver dual-GPU systems without a working CUDA peer-to-peer path. Consumer Ampere P2P availability depends on the driver and host configuration; the physical PCIe topology alone does not establish that peer memory access works.

Table with columns: Host configuration, NCCL environment, vLLM all-reduce, Status
Host configuration	NCCL environment	vLLM all-reduce	Status
Stock driver; P2P unavailable or unverified	`NCCL_P2P_DISABLE=1`	Pass `--disable-custom-all-reduce`	Validated configuration
P2P-enabled driver; peer access verified on both GPUs	Do not set `NCCL_P2P_DISABLE`	Omit `--disable-custom-all-reduce`	Alternative profile; benchmark on the actual PCIe topology

Verify the second profile with a CUDA peer-access test such as p2pBandwidthLatencyTest, not only nvidia-smi topo -m. Remove the two workarounds together: enabling vLLM's custom all-reduce while NCCL peer access remains disabled is not the intended profile.

Recommended vLLM command

The reference llama-swap/podman deployment, minus the container plumbing, with two deliberately conservative flags: prefix caching is off by default (the reference enables it after applying the safety patch below) and the sequence cap is 2 (the reference runs 4). Rationale in the notes after the command.

bash
export NCCL_P2P_DISABLE=1
export NCCL_CUMEM_ENABLE=0
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export OMP_NUM_THREADS=1
export VLLM_USE_FLASHINFER_SAMPLER=0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512

vllm serve lued/Qwen3.8-27B-AEON-UNCENSORED-INT8-W8A16-MTP \
  --served-model-name qwen3.8-27b-aeon-uncensored-int8-w8a16-mtp \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --dtype bfloat16 \
  --performance-mode balanced \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8_e4m3 \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --mamba-cache-mode align \
  --prefix-match-unit 16 \
  --enable-prompt-tokens-details \
  --enable-per-request-metrics \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --disable-custom-all-reduce \
  --trust-remote-code \
  --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0,"presence_penalty":0.0}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

NCCL_P2P_DISABLE=1 avoids NCCL initialization stalls on dual consumer GPUs without NVLink. --mamba-cache-mode align is required by the Qwen3.8 MTP/GDN serving path. Three speculative tokens is the measured default.

Prefix caching is off by default. On this hybrid GDN + MTP architecture, prefix caching without the 9-line source patch from vllm#48375 can corrupt recurrent-state KV, which surfaces as wrong tool-call or long-context output rather than an error. The reference deployment applies that patch at container start (along with this repository's chat_template.jinja via --chat-template), which is what lets it run with --enable-prefix-caching. If you apply the same patch, flip this flag back on; the published numbers were measured with a cold cache, so either setting leaves the table valid.

Concurrency is capped at 2, not 4, on this engine pin. Two or more concurrent requests can intermittently crash the engine (asynchronous cudaErrorIllegalAddress in the GDN spec-decode state path; same class as vllm#37431). The targeted fix is vllm#50021, which is open and awaiting its merge as of this writing; a vLLM release containing it should restore --max-num-seqs 4. Until then, a crashed engine drops in-flight requests and the service (llama-swap, podman, or a process manager) restarts it; if that is unacceptable, lower the cap to 1, which is the fully validated configuration on this pin.

MTP acceptance

MTP acceptance is workload-, sampling-, and draft-depth-dependent; it is not directly comparable across different models or benchmark prompts. On the 2026-08-16 serving validation, the engine reported native MTP-3 drafting with mean acceptance length 3.00 and ~66.7% per-position acceptance on the smoke prompt. Overall draft acceptance across benchmark runs is prompt-dependent.

The MTP checkpoint and lm_head are already BF16. Requantizing them cannot improve alignment. Increasing num_speculative_tokens adds progressively less accurate draft positions and normally lowers the reported average acceptance rate; lowering it can raise that percentage while reducing useful speculative work. The serving decision must therefore use end-to-end output throughput, not acceptance alone. Keep num_speculative_tokens=3 unless a controlled throughput test on the intended workload shows a better setting.

Operational notes

Table with columns: Symptom or question, Action
Symptom or question	Action
Why does this model answer prompts the base model refuses?	This is an abliterated model: refusal elimination is the point of the base (0 outright refusals on its 250-prompt eval sets; this checkpoint's smoke test showed zero refusals). All other behavior is intended to be unchanged from the base.
Which vLLM version is known to work?	Use the pinned image `nightly-ac7509e2b1db40fec2f03dde1ed4e9dfdc2338c9`. Other releases may work, but must support Qwen3.8, the compressed-tensors Marlin path, aligned Mamba cache, and MTP speculative decoding together.
Can llama.cpp load this repository?	No. This is a compressed-tensors checkpoint for vLLM, not GGUF. A separate GGUF conversion needs its own compatibility and quality validation.
The first startup appears idle	Wait for model loading, Torch/Triton compilation, CUDA graph capture, and the final API-ready message. Persist `/root/.cache/vllm` and the active Triton cache directory—`/root/.triton/cache` by default—across container runs to avoid repeating avoidable compilation work.

Files and provenance

Table with columns: File, Purpose
File	Purpose
`model-0000N-of-00006.safetensors`	Packed W8A16 language and GDN weights plus preserved BF16 tensors
`model.safetensors.index.json`	Shard-to-tensor mapping (1,999 keys)
`model_mtp.safetensors`	BF16 MTP head, 849 MB (810 MiB) / 15 tensors — byte-identical to Qwen's original MTP
`recipe.yaml`	Exact llm-compressor W8A16 recipe
`chat_template.jinja`	Chat and tool-use template from the base AEON card
/

Provenance chain:

text
Qwen/Qwen3.8-27B  (commit 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, 18×BF16 shards, 55.56 GB)
   │  SSM conv1d outlier repair (FernflowerAI), then abliterix 1.12.2
   │  50-trial Optuna, Flash Lite judge, trial 48/50
   │  (refusals 0 outright, smash KL 0.0991; MTP grafted back from stock, vision untouched)
   ▼
AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16  (commit a6775a9a8ebb65cab3f707b4ab087fc7aa698634)
   │  llm-compressor QuantizationModifier, W8A16 RTN, group-128 symmetric, data-free
   │  400 GEMMs packed; lm_head / vision / MTP / GDN gates preserved BF16
   ▼
this repository  (6×W8A16 shards + model_mtp.safetensors, 31.6 GB)

Acknowledgements and license

This repository repackages numerical weights derived from AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 (abliteration by AEON-7 via abliterix, Heretic-derived) and Qwen/Qwen3.8-27B. It does not claim authorship of the base model, the abliteration, architecture, training data, or upstream evaluations. Header styling and the Qwen logo follow the official Qwen repositories.

Apache-2.0 for this packaging. Respect all upstream model terms and attribution requirements, including the AEON base card's user-responsibility and arbitration clause.

Relationship to the sibling quantization

This checkpoint is the AEON Uncensored counterpart of lued/Qwen3.8-27B-INT8-W8A16-MTP. The two repositories share:

the identical W8A16 compressed-tensors recipe (400 packed GEMMs, same BF16 preservation set, group-128 symmetric data-free RTN);
the identical native BF16 MTP head (15 tensors, byte-identical to Qwen's original MTP);
the identical tokenizer, chat template, processor configs, and 6-shard layout;
the identical validated serving profile (TP2, 262144, fp8_e4m3 KV, MTP × 3).

Abliteration (inherited from the base)

text
Qwen/Qwen3.8-27B
        ↓
  SSM conv1d outlier repair (FernflowerAI)
        ↓
  abliterix 1.12.2, 50-trial Optuna, Flash Lite judge, trial 48/50
        ↓
  MTP head grafted back from stock (15 tensors, hash-match)
  Vision tower untouched (333/333 hash-match)
        ↓
AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16

Dual RTX 3090 deployment

Validated host profile: 2×RTX 3090 24 GB, PCIe without NVLink or P2P, vLLM TP2, BF16 activations, FP8 E4M3 KV cache, MTP with three draft tokens, and --max-num-batched-tokens 8192.

Memory and capacity

Table with columns: Metric, Result
Metric	Result
Checkpoint size	31.6 GB / 29.44 GiB
Loaded model memory per GPU	14.85 GiB
Shared GPU KV pool	266,537 tokens
Native maximum request length	262,144 tokens
Simultaneous full-native-context capacity	1.02×
Illustrative four-way KV share	Approximately 66,634 tokens per request

The checkpoint has the same size and footprint as the sibling quantization, since the W8A16 policy and shard layout are identical.

Measured serving performance

Table with columns: Prompt tokens, Generation tokens, Prefill tok/s, Generation tok/s, TTFT
Prompt tokens	Generation tokens	Prefill tok/s	Generation tok/s	TTFT
1,024	512	2,087	69.9	0.6 s
1,024	1,024	1,951	98.8	0.6 s
8,192	512	1,733	60.8	4.8 s

Quantization fidelity

Two different fidelity questions apply to this checkpoint, and they should not be conflated:

Table with columns: Candidate, Mean KLD vs BF16, Top-1 agreement, Weight files
Candidate	Mean KLD vs BF16	Top-1 agreement	Weight files
Official Qwen FP8 / BF16 A16 (sibling measurement)	0.004396	98.53%	28.75 GiB
This W8 / BF16 A16	0.001106	98.80%	29.44 GiB

Checkpoint profile

Table with columns: Property, Value
Property	Value
Quantization	Data-free symmetric RTN W8A16, group size 128
Runtime format	`compressed-tensors` / `pack-quantized`
Kernel dispatch	`CompressedTensorsWNA16` → `MarlinLinearKernel` (verified in server logs)
Preserved precision	BF16 vision tower, `lm_head`, MTP (15 tensors, byte-identical to Qwen's original MTP), and recurrent GDN gates
MTP	BF16 draft model detected at runtime; embeddings and `lm_head` shared with the target

Why W8A16 on Ampere

W8A16 is therefore the better fidelity default here: it preserves eight integer bits with group-wise scaling while retaining the same practical decode class as FP8-Marlin on these GPUs.

Table with columns: Format, Role on 2×24 GB at long context, Decision
Format	Role on 2×24 GB at long context	Decision
W8A16 — this checkpoint	Near-lossless weights; production default	Recommended
Official / online FP8 weight quantization	Convenient, but higher measured weight error on sm_86 fallback	Useful comparison
W8A8 INT8	Can improve prefill throughput; may regress skinny-batch decode	Specialized profile
W4A16	Smaller weights and more cache headroom	Higher long-context risk for recurrent GDN layers

Why not AutoRound?

Quantization design

Architecture audit

The BF16 source was checked before quantization against the Qwen3.6 structural contract:

64 language layers: 48 Gated DeltaNet linear-attention layers and 16 full-attention layers.
Hidden size 5,120; intermediate size 17,408; padded vocabulary 248,320.
One MTP layer with the same 15 top-level mtp.* tensors.
Stable GDN projection names: in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, and out_proj.
The only relevant rename was non-Linear convNd → conv1d, outside the quantization target.

The pre-quantization structural audit completed successfully before any weights were modified.

Recipe

Table with columns: Component, Precision, Reason
Component	Precision	Reason
MLP projections	INT8 W8A16	Largest dense GEMMs; strong memory return
Full-attention projections	INT8 W8A16	Measured low output-distribution error
GDN `in_proj_qkv`, `in_proj_z`, `out_proj`	INT8 W8A16	Recovers approximately 4 GB while remaining stable in the 4K GDN probe
GDN `in_proj_a`, `in_proj_b`

The result contains 400 quantized Linear GEMMs: 192 MLP projections, 64 full-attention projections, and 144 dense GDN projections.

yaml
# recipe.yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['lm_head', 're:.*visual.*', 're:.*mtp.*', 're:.*linear_attn[.]in_proj_a$', 're:.*linear_attn[.]in_proj_b$']
      scheme: W8A16
      bypass_divisibility_checks: false

Reproduction contract

Quantization fidelity: KLD versus the BF16 source

[!IMPORTANT] This section measures checkpoint-weight quantization. It is not a BF16-versus-FP16 benchmark and it does not measure the FP8 KV cache. The teacher is always the original BF16 checkpoint (the AEON source).

Overall results

Table with columns: Candidate checkpoint, Stored weights, A16 dtype in measurement, Mean KLD vs BF16, Top-1 agreement
Candidate checkpoint	Stored weights	A16 dtype in measurement	Mean KLD vs BF16	Top-1 agreement
BF16 source	BF16	BF16	0 by definition	100%
This checkpoint	INT8 symmetric, groups of 128	BF16 after INT8×scale	0.001106	98.80%

Short-prompt suite

Table with columns: Prompt, Category, Tokens, Positions, This W8 / BF16 A16, Top-1 agreement
Prompt	Category	Tokens	Positions	This W8 / BF16 A16	Top-1 agreement
`factual`	factual	13	12	0.000460	1.0000
`code-fib`	code	87	86	0.000539	0.9884

Long-context GDN probe

The probe used the first 3,288 tokenizer tokens of the AEON source model card as cached locally, producing 3,288 scored positions.

Table with columns: Candidate, Mean KLD, Top-1 agreement
Candidate	Mean KLD	Top-1 agreement
This W8 / BF16 A16	0.001163	98.72%

KLD did not accumulate across this probe; no recurrent drift was observed at 3.3K tokens.

factual: The capital of France is Paris, and the capital of Japan is

code-fib:

python
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

# Rewrite with memoization to avoid exponential blowup, then add a
# test that checks fibonacci(10) == 55 and fibonacci(0) == 0. Use a
# dict as the cache and make the function private to the module.

physics-uncertainty: In quantum mechanics, the uncertainty principle states that the product of the uncertainties in position and momentum cannot be smaller than about half of the reduced Planck constant. More precisely, if many identically prepared particles are measured, the standard deviations of position and momentum satisfy the inequality derived by Kennard in 1927, which is a direct consequence of the non-commutativity of the position and momentum operators in Hilbert space.
math-train: A train travels 240 kilometres in 3 hours and 15 minutes, stopping for 10 minutes at one station along the way. What is its average speed in kilometres per hour? Show your calculation step by step, and state whether the stop changes the average speed.
:

Run locally on 2026-08-17 with identical teacher-tokenizer IDs.
BF16 teacher snapshot: a6775a9a8ebb65cab3f707b4ab087fc7aa698634.
Environment: 2×RTX 3090, PyTorch 2.11.0+cu128, Transformers 5.10.1, compressed-tensors 0.17.1.
The student loads packed and forwards with per-module on-the-fly dequantization. These are Hugging Face decompressed-path logits, not vLLM Marlin kernel logits.
use_cache=False: the serving configuration's FP8 KV-cache error is not included.
No scheduling, sampling, image, or context beyond 3,288 tokens was measured.
Teacher logits were cached as FP16 before the shared FP32 KLD calculation, introducing a small common precision floor.

MTP preservation

The MTP head is the exact BF16 MTP from the original Qwen3.8-27B — never touched by the abliteration (which ran on the language layers only) and never re-quantized:

15 tensors in model_mtp.safetensors (849,400,392 bytes).
sha256 1d8268aa85ace093a561e3e7b63b9d390dac1cd55a90cd55b5ec509c3c9da9fe — identical to the MTP bytes in the Qwen3.8-27B source and in the AEON source's model-00003-of-00003.safetensors.
All 15 tensors verified bfloat16, shape-equal, and bit-equal to the AEON source (strict audit, not a count check).

KV-cache rationale: E4M3 versus E5M2

Table with columns: KV dtype, Mantissa bits, Maximum finite value, Minimum subnormal, Practical tradeoff
KV dtype	Mantissa bits	Maximum finite value	Minimum subnormal	Practical tradeoff
`fp8_e4m3`	3	±448	2^-9 ≈ 0.00195	Better in-range precision
`fp8_e5m2`	2	±57,344	2^-16 ≈ 0.0000153	Much wider dynamic range

E4M3 is the default because attention K/V values generally benefit more from its additional mantissa bit than from E5M2's extreme range.

[!WARNING] Runtime compatibility is verified; long-context KV accuracy is not. The checkpoint declares kv_cache_scheme: null, so it contains no dataset-calibrated K/V scales. The KLD measurement above uses no cache, and no saturation probe was performed.

E5M2 remains the fallback if measured ranges or a future vLLM build expose E4M3 saturation or kernel incompatibility.

Serving

GPU interconnect profiles

Table with columns: Host configuration, NCCL environment, vLLM all-reduce, Status
Host configuration	NCCL environment	vLLM all-reduce	Status
Stock driver; P2P unavailable or unverified	`NCCL_P2P_DISABLE=1`	Pass `--disable-custom-all-reduce`	Validated configuration
P2P-enabled driver; peer access verified on both GPUs	Do not set `NCCL_P2P_DISABLE`	Omit `--disable-custom-all-reduce`	Alternative profile; benchmark on the actual PCIe topology

Recommended vLLM command

bash
export NCCL_P2P_DISABLE=1
export NCCL_CUMEM_ENABLE=0
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export OMP_NUM_THREADS=1
export VLLM_USE_FLASHINFER_SAMPLER=0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512

vllm serve lued/Qwen3.8-27B-AEON-UNCENSORED-INT8-W8A16-MTP \
  --served-model-name qwen3.8-27b-aeon-uncensored-int8-w8a16-mtp \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --dtype bfloat16 \
  --performance-mode balanced \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8_e4m3 \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --mamba-cache-mode align \
  --prefix-match-unit 16 \
  --enable-prompt-tokens-details \
  --enable-per-request-metrics \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --disable-custom-all-reduce \
  --trust-remote-code \
  --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0,"presence_penalty":0.0}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

MTP acceptance

Operational notes

Table with columns: Symptom or question, Action
Symptom or question	Action
Why does this model answer prompts the base model refuses?	This is an abliterated model: refusal elimination is the point of the base (0 outright refusals on its 250-prompt eval sets; this checkpoint's smoke test showed zero refusals). All other behavior is intended to be unchanged from the base.
Which vLLM version is known to work?	Use the pinned image `nightly-ac7509e2b1db40fec2f03dde1ed4e9dfdc2338c9`. Other releases may work, but must support Qwen3.8, the compressed-tensors Marlin path, aligned Mamba cache, and MTP speculative decoding together.
Can llama.cpp load this repository?	No. This is a compressed-tensors checkpoint for vLLM, not GGUF. A separate GGUF conversion needs its own compatibility and quality validation.
The first startup appears idle	Wait for model loading, Torch/Triton compilation, CUDA graph capture, and the final API-ready message. Persist `/root/.cache/vllm` and the active Triton cache directory—`/root/.triton/cache` by default—across container runs to avoid repeating avoidable compilation work.

Files and provenance

Table with columns: File, Purpose
File	Purpose
`model-0000N-of-00006.safetensors`	Packed W8A16 language and GDN weights plus preserved BF16 tensors
`model.safetensors.index.json`	Shard-to-tensor mapping (1,999 keys)
`model_mtp.safetensors`	BF16 MTP head, 849 MB (810 MiB) / 15 tensors — byte-identical to Qwen's original MTP
`recipe.yaml`	Exact llm-compressor W8A16 recipe
`chat_template.jinja`	Chat and tool-use template from the base AEON card
/

Provenance chain:

text
Qwen/Qwen3.8-27B  (commit 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, 18×BF16 shards, 55.56 GB)
   │  SSM conv1d outlier repair (FernflowerAI), then abliterix 1.12.2
   │  50-trial Optuna, Flash Lite judge, trial 48/50
   │  (refusals 0 outright, smash KL 0.0991; MTP grafted back from stock, vision untouched)
   ▼
AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16  (commit a6775a9a8ebb65cab3f707b4ab087fc7aa698634)
   │  llm-compressor QuantizationModifier, W8A16 RTN, group-128 symmetric, data-free
   │  400 GEMMs packed; lm_head / vision / MTP / GDN gates preserved BF16
   ▼
this repository  (6×W8A16 shards + model_mtp.safetensors, 31.6 GB)

Acknowledgements and license

Apache-2.0 for this packaging. Respect all upstream model terms and attribution requirements, including the AEON base card's user-responsibility and arbitration clause.

Qwen3.8-27B-AEON-UNCENSORED-INT8-W8A16-MTP

README

Relationship to the sibling quantization

Abliteration (inherited from the base)

Dual RTX 3090 deployment

Memory and capacity

Measured serving performance

Quantization fidelity

Checkpoint profile

Why W8A16 on Ampere

Why not AutoRound?

Quantization design

Architecture audit

Recipe

Reproduction contract

Quantization fidelity: KLD versus the BF16 source

Overall results

Short-prompt suite

Long-context GDN probe

MTP preservation

KV-cache rationale: E4M3 versus E5M2

Serving

GPU interconnect profiles

Recommended vLLM command

MTP acceptance

Operational notes

Files and provenance

Acknowledgements and license

Explore FriendliAI today

README

Relationship to the sibling quantization

Abliteration (inherited from the base)

Dual RTX 3090 deployment

Memory and capacity

Measured serving performance

Quantization fidelity

Checkpoint profile

Why W8A16 on Ampere

Why not AutoRound?

Quantization design

Architecture audit

Recipe

Reproduction contract

Quantization fidelity: KLD versus the BF16 source

Overall results

Short-prompt suite

Long-context GDN probe

MTP preservation

KV-cache rationale: E4M3 versus E5M2

Serving

GPU interconnect profiles

Recommended vLLM command

MTP acceptance

Operational notes

Files and provenance

Acknowledgements and license