Shockem

Qwen3.8-27b-Terse-Coder-LoRA

README

License: apache-2.0

Background

Qwen-3.8-27b is an excellent dense model. It's a breakthrough in locally hosted models on consumer-grade hardware. The biggest challenge it faces is that its reasoning can be, at times, overly verbose. This isn't necessarily an issue, as it can pull itself out of hallucinations, but with user hardware around this model size, it ends up causing long waits, oftentimes into hours, before any outputs or edits occur.

Recently, agentionai released Signal-3.8-27B. This opened the gates to the idea of lowering reasoning by finetuning the model, rather than handling it via templates or configuration. The outcome was a model that thought significantly less than the base model, with marginal differences in error. This sent me down a rabbit hole of testing its reasoning and how it affected the model's output. To my surprise, it was incredibly close to the base, even its traces were very close, just cleaner overall.

My experiment is to continue this research and push it further. So far, I've trained on traces from Signal, the base NVIDIA provided NVFP4 quant, and my own abliterated variant, against HumanEval, 600 questions per round. This is now round 7, and it has shown significant improvement. We are now at a 95% reduction in reasoning against the NVIDIA quant, and 52% against Signal. This was originally created as a LoRA adapter that is then merged into a custom recipe for Qwen-3.8-27b. I have provided both a merged model and a LoRA adapter. Thank you for testing and providing feedback!

Results

All testing was done on NVFP4-quantized bases (modelopt W4A16) served by vLLM 0.28 on 3× RTX 5060 Ti. Held-out 40 problems (20 HumanEval + 20 MBPP-sanitized, disjoint from training), n=1 per run, sampling temp 0.6 / top_k 20 / top_p 0.95 / rep-penalty 1.05, pass@1 by automated test execution in a sandboxed container, reasoning tokens from completion_tokens_details.reasoning_tokens. Headline metric: totals of per-problem median-of-runs, paired permutation test (single runs are heavy-tailed — run means swing ±20%).

Table with columns: Base (all NVFP4), Serving, Reasoning tokens, p, pass@1 base → +LoRA
Base (all NVFP4)	Serving	Reasoning tokens	p	pass@1 base → +LoRA
nvidia/Qwen3.8-27B-NVFP4 (stock)	LoRA, 3 runs	−94.7%	0.0001	72.5% → 70.8%
House stock NVFP4 v3 (NVIDIA-style Hessian solve)	LoRA, 6 runs	−92.4%	0.0001	69.2% → 67.9%
House stock NVFP4 v2 (absmax W4A4)	LoRA, 3 runs	−49.5%	0.0001	70.8% → 69.2%
Signal (no heretic tuning)	LoRA, 3 runs	−40.0%	0.0002	64.2% → 65.8%

Effect size varies by base quant calibration. NVIDIA's checkpoint (and our v3 recipe, which replicates its Local-Hessian solve on MLP/lm_head) shows the full ~−93% cut; an absmax-calibrated quant of the same weights (v2) only cut −49.5%. Direction and pass-neutrality hold on every base tested. If you build your own W4A16 quant to pair with this adapter, Hessian-weighted calibration on MLP/lm_head is what unlocks the full effect.

Prefer a merged model over a runtime adapter? Round 7 is also available pre-merged into stock Qwen3.8-27B as Shockem/Qwen3.8-27b-Terse-Coder (fp16) and Shockem/Qwen3.8-27b-Terse-Coder-NVFP4 (v3-recipe W4A16): 67.5% pass with ~38 reasoning tokens/problem baked in — same behavior, zero LoRA plumbing. Do not stack this adapter on top of the merged model (double application over-shortens: 63% pass with no_code failures in our testing).

⚠️ On heretic-ara specifically, round 6 (archive/r6/) is the pass-neutral option (−36.0%, 62.5% → 63.7%). On stock and Signal, round 7 dominates.

The merged version of this round (Shockem/Qwen3.8-27b-Terse-Coder) also held 78.3% on GPQA-Diamond (PhD-level science, far outside the coding training distribution) at ~1.5k mean reasoning tokens, scored 92.1% CRUXEval-I / 92.9% CRUXEval-O (full 800, official Meta harness), and hit 90.2% HumanEval+ / 78.6% MBPP+ (official EvalPlus, greedy) at under 100 mean reasoning tokens — evidence the preference cuts the deliberation budget, not capability.

Reference points from the same harness: stock base alone is 72.5% / 701 reasoning tokens; with the adapter it answers coding problems near-directly (mean 38 reasoning tokens) and holds 70.8%.

Serve it with speculative decoding (recommended)

The adapter does not touch the MTP draft head, so MTP speculative decoding keeps working at full acceptance (measured 0.43 with and without the adapter on the NVIDIA checkpoint — outputs are target-verified, so spec decode is lossless). Measured on 2× RTX 5060 Ti 16 GB (vLLM 0.28, FP8 KV, num_speculative_tokens: 3):

Table with columns: Config, Wall tok/s
Config	Wall tok/s
NVIDIA NVFP4 base, MTP spec on	54.1
House stock NVFP4 v3 base, MTP spec on	55.5
House stock NVFP4 v2 base, MTP spec on	53.7
NVIDIA NVFP4 + this adapter, MTP spec on	48.9
House stock NVFP4 v3 + this adapter, MTP spec on	49.1
House stock NVFP4 v2 + this adapter, MTP spec on	45.0
NVIDIA NVFP4 + this adapter, spec OFF	27.4
Signal NVFP4 (house quant) + this adapter, spec on	50.4

Turn spec decode ON — it is +78% wall speed with the adapter loaded. On 2×16 GB cards with the adapter + spec, cap context at ~175k (200k needs more KV headroom than the two cards have; single-card 24 GB+ rigs are unaffected).

Runtime LoRA is the full-strength deployment form

Round-7's weight deltas are deliberately tiny (‖Δ‖/‖W‖ ≈ 4e-4–1e-3) — below bf16's per-element resolution. Measured delta survival when merging into the base: 31–61% in bf16, 94–99.9% in fp16, and merged-to-NVFP4 attenuates on top of that. So: load the adapter at runtime on any base quant (the delta applies in bf16 at compute time — full strength). If you merge, merge in fp32 and store fp16; any merged 4-bit artifact loses some of the effect.

Variants

All variants of the adapter live on this page; pick per your stack. Note that every number above was measured with the bf16 adapter on NVFP4 bases.

Direct downloads:

Table with columns: Variant, Format, Size, Download, Use
Variant	Format	Size	Download	Use
bf16 (repo root)	PEFT bf16, rank 16, alpha 32	223 MB	adapter_model.safetensors	Servable artifact — vLLM/PEFT load this directly; powers `PeftModel.from_pretrained(model, "Shockem/Qwen3.8-27b-Terse-Coder-LoRA")`
fp8 (`fp8/`)	per-tensor FP8 e4m3 (+F32 scale)	112 MB

Or grab a whole variant folder with the CLI: huggingface-cli download Shockem/Qwen3.8-27b-Terse-Coder-LoRA --include "fp8/*".

The bf16 adapter sits at the repo root (standard PEFT layout, powers the Hub's "Use this model" snippet). Only bf16 is directly loadable by serving runtimes today (vLLM 0.28's LoRA path takes bf16). The quantized variants are compact/research artifacts.

GGUF note: converted with llama.cpp master's convert_lora_to_gguf.py (targeting a Qwen3.5 hybrid gated-deltanet base — needs a llama.cpp build new enough for qwen3_5), with a small patch for the linear-attention out_proj column reorder on LoRA tensors. The bf16 GGUF is verified bit-exact against the bf16 PEFT adapter on all 992 tensors, including the grouped→tiled V-head reorders llama.cpp applies to the linear_attn projections; q8_0 verified at dequant cosine ≥ 0.99997. Not yet runtime-tested against a GGUF base model — treat as experimental. There is no q4 GGUF: llama.cpp's LoRA-adapter format only defines f32/f16/bf16/q8_0 — for 4-bit use the int4/ safetensors variant. Usage:

markdown

llama-server -m <qwen3.8-27b-base>.gguf --lora Qwen3.8-27b-Terse-Coder-LoRA-q8_0.gguf

Serving with vLLM (tested path)

Tested on vLLM 0.28, including alongside MTP speculative decoding and on modelopt NVFP4 bases. The released adapter carries vLLM-native (language_model.model.layers.*) key layout — it attaches correctly to the Qwen3_5ForConditionalGeneration module tree. (Plain PEFT exports from a text-only AutoModelForCausalLM run silently attach zero weights in vLLM — basename checks pass, nothing is applied. If you re-export this adapter yourself, keep the VL-layout keys.)

CLI:

bash
vllm serve nvidia/Qwen3.8-27B-NVFP4 \
  --enable-lora \
  --lora-modules cot-lora=/path/to/bf16 \
  --max-lora-rank 16 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Then request with model: "cot-lora" (adapted) or the base model id (unadapted) — both are live on the same server:

python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
resp = client.chat.completions.create(
    model="cot-lora",  # terse reasoning
    messages=[{"role": "user", "content": "Write a Python lru_cache."}],
    temperature=0.6, top_p=0.95,
)
print(resp.usage.completion_tokens_details.reasoning_tokens)

Hot-loading on an already-running server also works: POST /v1/load_lora_adapter {"lora_name": "cot-lora", "lora_path": "..."}.

Other bases, quants, formats, backends

The adapter is a behavioral edit to the shared Qwen3.8-27B text stack (all attention + MLP + gated-deltanet linear_attn projections; no vision, no MTP weights), so it should stack fine on other quantizations (FP8/INT4/GGUF bases), full-precision bases, and other backends (SGLang, TabbyAPI/EXL3, llama.cpp) — the preference it encodes is not quantization-specific. That said, only the NVFP4 + vLLM combination above has been measured; treat other combinations as untested and validate before relying on them.

Training (summary)

Data: coding prompts (HumanEval + MBPP-sanitized, disjoint from eval), n=4 traces each at temp 1.0 with per-token logprobs, from three policies (tri-base round): heretic-ara, plain Signal, and NVIDIA stock — each with the round-6 adapter active (on-policy continuation). Traces filtered by automated test execution; passing traces segmented into steps and pruned by per-step mean token entropy → 525 DPO pairs: (pruned reasoning, same answer) vs (full reasoning, same answer).
Method: DPO (trl 1.13), β 0.05, lr 1e-5 cosine, 3 epochs, eff. batch 8, initialized from the round-6 adapter so the reference anchors at base+r6 and only further shortening is rewarded.

Caveats

Targeted at coding tasks with thinking enabled. Behavioral LoRA, not a knowledge edit.
The preference is "shorter reasoning, identical answer" — if a task needs long derivation, raise reasoning_effort as usual.
If you serve with speculative decoding, make sure the generation config has no min_p — vLLM 0.28 rejects min_p under spec decode.

Attributions & licenses

This adapter is trained against, and licensed for use with, Qwen/Qwen3.8-27B, © Qwen Team, Alibaba Cloud, Apache 2.0; this adapter is likewise Apache 2.0 and the upstream license and copyright notices are retained. Credits:

Qwen Team (Alibaba Cloud) — the Qwen3.8-27B base model (Apache 2.0).
agentionai — Signal-3.8-27B, one of the three trace-generation policies.
p-e-w — the Heretic tool; a heretic-ara abliterated variant of Signal was another trace-generation policy.
NVIDIA — Qwen3.8-27B-NVFP4 (third policy and eval baseline) and TensorRT Model Optimizer 0.45 (Apache 2.0, quant tooling).
OpenAI — HumanEval (MIT), and Google — MBPP (CC-BY 4.0): prompt sources for training and held-out evaluation.
(Apache 2.0) — the DPO trainer; (MIT) — GGUF adapter conversion; — DeepSWE, independent evaluation only.

None of these parties endorse this adapter; all remaining errors are ours.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Shockem

Model Tree

Base

Qwen/Qwen3.8-27B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality