GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
shomit505
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
shomit505
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Tinker exports the Gated-DeltaNet input projection as three separate modules,
in_proj_q, in_proj_k and in_proj_v. vLLM builds that projection as one
fused layer and exposes exactly two LoRA-addressable sub-modules for it,
in_proj_qkv and in_proj_z. Those three names match nothing in vLLM, so the
original adapter is rejected at load with a target-module error.
This repack fuses each q/k/v triple into the single in_proj_qkv vLLM
expects: the A matrices are stacked and B is made block-diagonal, so three
rank-1 updates become one rank-3 update with an identical product. vLLM
applies a single rank per adapter and ignores rank_pattern, so the remaining
modules are zero-padded to rank 3, and lora_alpha is raised from 32 to 96 to
hold the applied scale at 32.
The rank of 3 is packaging, not capacity. This is a rank-1 adapter.
bash
vllm serve Qwen/Qwen3.8-27B --enable-lora --lora-modules refusal-r1=shomit505/Qwen3.8-27B-refusal-r1-vllm --max-lora-rank 8 --max-num-seqs 64 --reasoning-parser qwen3
Two flags are not optional:
--max-lora-rank 8. vLLM does not accept 3 as a bucket size, and 8 is the
next one up.--max-num-seqs 64. Qwen3.5 is a hybrid model needing one Mamba cache block
per decode sequence. At the default of 256 the engine cannot capture CUDA
graphs and refuses to start. Raise it only as far as the startup error allows.Rewrite lora_alpha to interpolate between the base model and full
suppression. Applied strength is lora_alpha / r, so 96 gives full strength
and 0 gives the base model, with a continuous dial in between. Registering
several copies at different values gives a difficulty ladder on one server.
610 held-out prompts, judged with gpt-oss-120b. Base and adapter served from the same vLLM instance, so these are like-for-like controls rather than cross-engine comparisons.
Thinking off (greedy, the trained and intended condition):
| verdict | base | adapter |
|---|---|---|
| complied | 0.026 | 0.880 |
| evaded | 0.064 | 0.067 |
| refused | 0.910 | 0.052 |
| broken | 0.000 | 0.000 |
Thinking medium (sampled at temperature 1.0, reasoning enabled):
| verdict | adapter |
|---|---|
| complied | 0.689 |
| evaded | 0.267 |
| refused | 0.041 |
| broken | 0.003 |
Refusal suppression transfers to reasoning mode and even strengthens slightly, from 0.052 to 0.041, against a base refusal rate of 0.805 at the same setting. What changes is the character of the compliance: evasion rises from 0.067 to 0.267, so roughly a quarter of reasoning-mode responses engage with the request without actually answering it. Hence thinking off as the operating point.
Both conditions closely reproduce the numbers measured on the original adapter served through Tinker, which gave compliance 0.899 and refusal 0.038 with thinking off, and compliance 0.700, evasion 0.261 and refusal 0.037 at medium. The repack costs nothing measurable.
in_proj_qkv naming targets the vLLM module layout. For the
Tinker-native export, use the original repository linked above.