GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
gjtgjt
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
gjtgjt
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| Base | Qwen/Qwen3.8-27B, revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 |
| Architecture | Qwen3.8 hybrid: 64 layers, 16 × (3 × Gated DeltaNet → FFN, 1 × Gated Attention → FFN) |
| Method | Heretic 1.4.0, --row-normalization PRE |
| Ablated modules | language-model attn.o_proj / linear-attn out_proj, and mlp.down_proj (64 layers) |
| Untouched | vision tower, MTP |
| Precision | BF16 merged weights, 12 shards, ~51 GB |
| Context | 262,144 tokens (same as base) |
| License | Apache-2.0 (inherited from the base) |
Heretic is training-free directional ablation (Arditi et al., 2024). It estimates a per-layer residual direction r from the first generated token, then writes a low-rank update into the output projections so that component is suppressed. Optuna TPE searches ablation weights to lower refusal-keyword hits while keeping KL to the parent model small.
Why PRE (true rank-1). Heretic row-normalization has three modes:
none: ΔW = -λ v (vᵀ W) — rank-1, no row-norm handlingpre: same rank-1 form, but computed on row-normalized W, then scaled back by the original row norms: ΔW = -λ v (vᵀ W_norm) ⊙ ‖W‖_rowfull: apply the PRE update, renormalize rows to restore magnitudes, then keep an SVD approximation of rank full_normalization_lora_rank (default 3). That is not rank-1.This release uses PRE only. Each accepted round is therefore one true rank-1 update. Two accepted rounds are two successive rank-1 updates, not a single rank-2 factorization.
Why iterate, and why recompute r. After a merge the residual geometry changes, so the next direction is measured on the new weights. Reusing r from round 1 would not be a fresh rank-1 step.
KL accounting. Search-time KL in a Heretic log is vs the current parent (incremental). Acceptance always re-evaluates vs the original base with --evaluate-model. Only that cumulative KL is constrained to ≤ 0.1.
Thinking-model cut. Qwen3.8 thinks by default. Residuals and keyword scoring are taken at the start of the visible answer, with --response-prefix $'\n</think>\n\n'. Do not put that prefix into the chat template at inference time.
Pipeline
Qwen/Qwen3.8-27B (not from a previous heretic checkpoint).r on the round-1 merge.r, try round 3 → reject (cumulative KL 0.1983 > 0.1).Heretic default English eval: keyword refusals / 100 prompts, and KL on the harmless split.
| Round | Keywords / 100 | KL vs original | Decision |
|---|---|---|---|
| Base | 98 | 0 | — |
| 1 (trial 162) | 25 | 0.0525 | accepted |
| 2 (trial 109) | 18 | 0.0931 | accepted, this release |
| 3 (Pareto 0/1/2) | 8 | 0.1983 | rejected (KL > 0.1) |
Round 3 would have lowered keywords further, but broke the KL cap. The shipped weights stay at round 2.
Per-layer directions. Ablation strength is a piecewise-linear kernel over layer index.
| Parameter | Value |
|---|---|
direction_index | per layer |
attn.o_proj.max_weight | 1.14 |
attn.o_proj.max_weight_position | 60.59 |
attn.o_proj.min_weight | 1.05 |
attn.o_proj.min_weight_distance | 28.96 |
mlp.down_proj.max_weight |
Round 1 (already baked in): trial 162, attn.o_proj max 0.90 @ 53.74, mlp.down_proj max 1.31 @ 62.63.
Needs a Transformers build that loads Qwen3.8 (qwen3_5 / Qwen3_5ForConditionalGeneration), e.g. Transformers 5.15+. Same chat template and sampling defaults as the base.
python
from transformers import AutoModelForImageTextToText, AutoTokenizermodel_id = "YOUR_USER/Qwen3.8-27B-heretic-r1n"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto",)messages = [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Explain residual connections in one paragraph."},]prompt = tok.apply_chat_template(messages,add_generation_prompt=True,tokenize=False,enable_thinking=False, # True (default) keeps <think>…</think>)inputs = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**inputs,max_new_tokens=512,temperature=0.7,top_p=0.8,top_k=20,presence_penalty=1.5,)print(tok.decode(out[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
enable_thinking=False, temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5enable_thinking=True (default), temperature=1.0, top_p=0.95, top_k=20; optional reasoning_effort in {xhigh, medium, low}Exact trial indices depend on the Optuna study. The procedure is:
bash
pip install -U heretic-llmPREFIX=$'\n</think>\n\n'heretic --model Qwen/Qwen3.8-27B \--response-prefix "$PREFIX" --row-normalization PRE \--n-trials 200 --n-startup-trials 60 \--model-action save --export-strategy merge \--save-directory ./r1# recompute r on the new weights, then search againheretic --model ./r1 \--response-prefix "$PREFIX" --row-normalization PRE \--n-trials 120 --n-startup-trials 40 \--model-action save --export-strategy merge \--save-directory ./r2# cumulative KL must be scored against the original baseheretic --model Qwen/Qwen3.8-27B \--evaluate-model ./r2 --response-prefix "$PREFIX"
在官方 Qwen3.8-27B 上用 Heretic 1.4.0 做迭代真 rank-1 消融。PRE 在行归一化后的 W 上写 ΔW = -λ v (vᵀ W_norm)，再乘回行范数，每轮严格 rank-1。FULL 会再做行范数还原并用默认 rank-3 的 SVD 近似，因此不是本仓库的做法。
每轮接受后必须在新权重上重算残差方向 r，不能复用上一轮的 r。搜索日志里的 KL 是相对本轮父模型的增量；验收一律对原模型跑 --evaluate-model，累计 KL 不得超过 0.1。Qwen3.8 默认带思考，取点和打分都切在 \n</think>\n\n 之后，推理时不要把这个 prefix 写进模板。
交付点是第 2 轮 trial 109：Keyword 18 / 100，累计 KL 0.0931。第 3 轮 Keyword 可到 8，但 KL 0.1983 超标，未合并。视觉塔和 MTP 未改。权重为 BF16 merge，12 个分片。许可 Apache-2.0。
| 0.97 |
mlp.down_proj.max_weight_position | 55.65 |
mlp.down_proj.min_weight | 0.01 |
mlp.down_proj.min_weight_distance | 24.00 |