ThakiCloud

Qwen3.8-27B-Human-KO-Safety-W4A16

README

License: apache-2.0

What was re-measured for this version (2026-09-07, temperature 0, thinking off, same serving config as bf16)

Table with columns: Axis, bf16 original, This model (W4A16), How to read it
Axis	bf16 original	This model (W4A16)	How to read it
KoBBQ ambiguous-context abstention (8,139)	92.8%	93.0%	Same (EXAONE 4.5: 65.7%)
Change in disambiguated-context accuracy (vs. base 90.66)	−2.1pp	−2.9pp	Within the pre-registered −3pp allowance, at the boundary
Conditional bias (among answered items)	0.861	0.859	Both miss the pre-registered target of 0.60 (structural — see the bf16 card)
Identity probe (8-language holdout, n=240)	96.2%	94.6%	4-item difference out of 240, at the shipping threshold of 95
MMLU English 1,000 (thinking off)	80.9%	80.7%	Within detection limit
Thinking-mode empty responses (100)	0	0	No format defect
Korean human-likeness pairwise win rate (n=175)	+14.3pp	+9.1pp	Judge uncalibrated, direction only
HumanEval (executed, same run, n=154)	95.0%	96.1%	+1.1pp, within detection limit
KMMLU 1,000 (same run)	51.0%	53.8%	+2.8pp
Instruction following 100 (deterministic checker)	78%	79%	Equivalent

In one line: abstention behavior is the same as bf16, and over-abstention and identity have each come down to the edge of their allowance. Capability gates (coding·KMMLU·instruction) were re-measured in the same run with bf16 as the reference arm and entered into the table above. The difference from the bf16 card's absolute values is because the runs differ; read the comparison only within the same run.

Quantization recipe

llm-compressor 0.13.0 GPTQ · calibration 1,024 samples × 2,048 tokens (chat format, 25% Korean) · dampening 0.01 · actorder static · scheme W4A16 (MLP) + FP8_DYNAMIC (attention) · ignore: vision/visual, lm_head, embed_tokens, linear_attn.in_proj_a/b. Took 38 minutes (1x B200). Metadata is in quantize_meta.json; the actual group and ignore lists are in config.json's quantization_config.

Serving

Because attention is FP8, this requires a GPU with an FP8 kernel (SM89+: H100·H200·L40S·RTX 40/50). A100 (SM80) has not been verified and has no FP8 activation path, so correct operation is not guaranteed. On Blackwell, the NVFP4 version is faster.

bash
vllm serve ThakiCloud/Qwen3.8-27B-Human-KO-Safety-W4A16 \
  --max-model-len 32768 --kv-cache-dtype fp8 --enable-prefix-caching

vLLM ≥ 0.28 auto-detects the scheme from config.json (compressed-tensors).

Limitations

The safety axis was measured with KoBBQ alone, and the table above is from a single build of this quantized checkpoint. Given the noise between requantization builds (our measured GSM8K figure is 3.56pp), a difference within 3pp is not attributed to the treatment.
Identity's 94.6% differs from bf16's 96.2%. If self-introduction accuracy matters, use bf16.
Other limitations and data provenance are the same as the bf16 card and DATA_PROVENANCE.md.

License

Apache-2.0 (same as Qwen/Qwen3.8-27B). See LICENSE · NOTICE.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ThakiCloud

Model Tree

Base

ThakiCloud/Qwen3.8-27B-Human-KO-Safety

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality