Adapter
Table | |
|---|
| attention / shared expert / dense MLP | rank 64 / alpha 32 (scaling 0.5), 496 adapters |
| routed experts | rank 32 / alpha 16 (scaling 0.5), 36288 adapters (42 layers x 288 experts x 3) |
| KDA linear-attention layers (34) | q_proj k_proj v_proj b_proj forget_gate.f_a_proj forget_gate.f_b_proj g_a_proj g_b_proj o_proj |
| MLA layers (11) | q_a_proj q_b_proj kv_a_proj_with_mqa kv_b_proj o_proj |
| MoE layers (42) | shared expert gate_proj up_proj down_proj + all 288 routed experts gate_proj up_proj down_proj |
| dense layers 0-2 | mlp.gate_proj up_proj down_proj |
| not adapted | routers, DSA indexer, mHC parameters, embeddings, lm_head, vision tower, MTP layer |
| no rsLoRA, no DoRA, dropout 0, bf16 | |
Files
Table with columns: Path, Format| Path | Format |
|---|
adapter_model.safetensors, adapter_config.json | PEFT (transformers module names) |
export_info.json | how the PEFT file was produced |
nmoe/adapter.safetensors | nmoe-lora-v1, checkpoint tensor names, attention / shared / dense |
nmoe/adapter_experts.safetensors | nmoe-lora-v1, per-expert model.layers.N.mlp.experts.E.{gate,up,down}_proj.lora_a/b |
nmoe/adapter_config.json, nmoe/state.json | trainer config and state |
The PEFT layout (routed experts as target_parameters on the fused gate_up_proj / down_proj, exact
block-diagonal representation) is identical to the E32 / GM-CoT adapters; see the E32 card for the details.
Usage (PEFT)
Requires transformers with glm5_next and peft >= 0.21 (target_parameters).
import torchfrom transformers import AutoModelForImageTextToTextfrom peft import PeftModel base = AutoModelForImageTextToText.from_pretrained( "zai-org/GLM-5.3-Flash-BF16", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "cgg507/GLM-5.3-Flash-GM-RP-v3-LoRA")# optional: model = model.merge_and_unload()
Use thinking ON (the default GLM-5.3-Flash chat template); the adapter was trained with the reasoning in
<think>...</think> before each reply. The reasoning is long (median ~1.7k characters, like the training data), so
give it a generous reasoning budget (1500+ tokens) or long turns can run out of budget before the reply.
Merging routed-expert deltas directly into existing NVFP4 experts does not work (the deltas are far below the fp4
step); merge into high-precision weights first, then re-quantise.
Validation
validate_peft_e32.py --offline (PEFT export checked against the trainer checkpoint, inverse mapping, no base download):
tensors: peft=992 nmoe=992 adapters=496 byte-identical=992/992
36288/36288 expert matrices decode bit-exactly (A and B), off-blocks zero
- PEFT config: r=64 alpha=32, experts rank_pattern gate_up_proj 64 / down_proj 32, alpha_pattern 32 / 16, 84 target_parameters
RESULT: PASS
Training
- Data: 3,988 conversations: 731 Game Master campaigns (long, ~45k tokens each, 83 % of tokens) and 3,257
character-roleplay conversations (1-2k tokens each). 40,159 assistant turns, every one with a reasoning block.
- Every turn's reasoning is trained, exactly (
--cot-branches): history turns are shown without reasoning (as the
chat template renders them) and each assistant turn is its own target (reasoning + reply + stop), computed with a
shared-history tree (one trunk per conversation, one branch per turn). 5,070 tree documents, 62.9M rows per epoch
instead of 256.4M as separate samples (4.08x less), identical gradients; 29.6M trained tokens.
- Max sequence 36,864, 1 epoch = 288 steps (grad-accum 8), shuffled; released at step 280. Muon optimizer (expert adapters: bf16 masters +
stochastic rounding), lr 2e-4, warmup 5, cosine to 10 %, grad clip 1.0.
- Loss 1.22 (mean of the first 10 steps) -> 1.02 (mean of steps 271-280).
- Trained with a custom C++/ROCm trainer (nmoe) on 2x Radeon AI PRO R9700 against
RedHatAI/GLM-5.3-Flash-NVFP4 (standard weights, NVFP4 routed
experts, BF16 attention / shared / dense), with fp8 activations on the frozen dense / attention GEMMs; ~530 tok/s,
~33 h.