What it reproduces
Serving the base model with --enable-lora and this adapter crashes SGLang (v0.5.14) at startup,
during LoRA memory-pool initialization:
NotImplementedError: get_hidden_dim not implemented for in_proj_qkv
NotImplementedError: get_hidden_dim not implemented for in_proj_z
HF transformers exposes the Gated-DeltaNet input projection as a fused in_proj_qkv (q, k, v in one
matmul) plus a separate in_proj_z, so PEFT emits LoRA weights under those 2-way names. SGLang's
Qwen3.5/GDN LoRA path only knows the fully-fused in_proj_qkvz, so get_hidden_dim has no case for the
native names and the server aborts.
Reproduce
python3 -m sglang.launch_server --model-path Qwen/Qwen3.6-35B-A3B \
--enable-lora --lora-paths repro=drodriguezhdez/qwen35-in_proj_qkv-repro \
--max-lora-rank 8 --lora-backend triton
Contents
target_modules: ["in_proj_qkv", "in_proj_z"], rank 4, fp16, across the 30 Gated-DeltaNet layers.- Per layer:
in_proj_qkv.lora_A [4, 2048], in_proj_qkv.lora_B [8192, 4] (fused q,k,v = 2048+2048+4096),
in_proj_z.lora_A [4, 2048], in_proj_z.lora_B [4096, 4].
These names and shapes are identical to those emitted by a real PEFT adapter trained on this base model.
Why random weights
The crash depends only on the adapter's module names and shapes, never the values — so this adapter
carries randomly-initialized weights (no proprietary data) and reproduces the crash identically to a real
trained adapter with the same targets.
Workaround
The fully-fused in_proj_qkvz layout is served correctly; repacking the adapter offline into it (split
in_proj_qkv's output back into q/k/v, set target_modules: ["in_proj_qkvz"]) lets it serve and apply
correctly on v0.5.14.