How this model was produced (reproducible steps)
0. Environment
The exact stack used (an isolated uv venv, Python 3.12):
Table with columns: package, version| package | version |
|---|
| torch | 2.12.0 (cu130) |
| transformers | 5.10.1 |
| llmcompressor | 0.12.0 |
| compressed-tensors | 0.17.1 |
| peft | 0.19.1 |
| accelerate | 1.14.0 |
Note: transformers >= 5.10 is required — the qwen3_5_moe architecture is not
recognized by older releases (and it ships no remote code, so trust_remote_code
cannot substitute). Installing llmcompressor pins transformers to 5.10.1, which
still supports the architecture.
uv venv --python 3.12 .venv
uv pip install torch==2.12.0 transformers peft accelerate safetensors llmcompressor
1. Merge the LoRA adapter — including the fused routed experts
This is the subtle step. The base stores its 256 routed experts fused as 3-D
parameters:
Qwen3_5MoeExperts.gate_up_proj [E=256, 2*I=1024, H=2048] # gate = rows 0:I, up = rows I:2I
Qwen3_5MoeExperts.down_proj [E=256, H=2048, I=512]
but the adapter stores routed-expert LoRA per-expert and unfused
(...mlp.experts.{e}.{gate,up,down}_proj.lora_{A,B} — 61,440 tensors, ~99% of the
adapter). A plain PEFT target_modules merge on the fused base matches none of these
expert modules and silently drops the entire routed-expert fine-tune (keeping only the
350 attention / linear-attn / shared-expert LoRAs). The model's learned behavior
(including its "Palmyra / Writer" identity) lives in those routed experts, so they must
be merged.
The delta is added directly into the correct slice of each fused tensor — mathematically
identical to unfuse → PEFT-merge → re-fuse, but with no round-trip and no layout risk.
Split order (gate first) is confirmed from modeling_qwen3_5_moe.py:
gate, up = linear(x, gate_up_proj[e]).chunk(2, dim=-1).
The load class must be AutoModelForImageTextToText
(= Qwen3_5MoeForConditionalGeneration) — not AutoModelForCausalLM, which exposes
the text tower as model.layers.* (no language_model. prefix, no vision) and fails to
match the adapter.
Validation performed during the merge: all 31,030 LoRA pairs resolve
(30,720 expert + 310 linear), a numeric spot-check of a merged expert slice
(max|applied − expected| ≈ 1.5e-4, bf16 rounding), and every parameter is bf16.
Result: a 66 GB bf16 merged checkpoint with the full fine-tune baked in.
The base model's MTP head is dropped on load by the ForConditionalGeneration
class, so the merged bf16 checkpoint has no MTP tensors. The adapter does not touch MTP;
the head is restored (bf16) from the base during quantization (below).
2. FP8 quantization with llm-compressor
FP8 for this Qwen3.5-MoE-VL family is data-free (no calibration dataset) — this is
the officially supported path (FP8_BLOCK). llm-compressor's load_context unfuses
the 256 experts into per-expert Linears so they are actually quantized, then re-fuses on
save for vLLM.
See https://github.com/WriterInternal/nlp.cigna-lora-infra-evals/tree/main/quantization-v2
Result: ~37 GB — 30,880 FP8 weight tensors (+ scales), the rest bf16, plus a separate
model_mtp.safetensors (19 bf16 MTP tensors).
Serving with vLLM
vLLM auto-detects the compressed-tensors / float-quantized config — no
--quantization flag needed.
vllm serve qwen3.6-35b-a3b-cigna-palmyra-asft-fp8 \
--served-model-name palmyra-cigna-fp8 \
--tensor-parallel-size 4 \
--max-model-len 65536 \
--reasoning-parser qwen3 \
--trust-remote-code
Disable the thinking/reasoning trace per request with Qwen's template kwarg:
{"model": "palmyra-cigna-fp8",
"messages": [{"role": "user", "content": "Who are you?"}],
"chat_template_kwargs": {"enable_thinking": false}}
Requires FP8-capable hardware (H100 / H200). Tested on 4× H100.
Provenance & licensing
Derived from Qwen/Qwen3.6-35B-A3B and the Writer/qwen3.6-35b-a3b-cigna-palmyra-asft
LoRA adapter (subfolder asftnr_lr1e-4_ep2_kl0.03). Use is subject to the license terms
of the base model and the adapter. The stock Qwen chat template is used unmodified.
