cgg507

Qwen3.5-122B-A10B-GM-CoT-LoRA

README

License: apache-2.0

Recommended use: train on abliterated, run on the standard base

The adapter was trained against an abliterated base (Qwen3.5-122B-A10B-abliterated-NVFP4, derived from wangzhang/Qwen3.5-122B-A10B-abliterated), so training was not fighting refusals. In our testing it gives clearly better results applied to the standard (non-abliterated) Qwen3.5-122B-A10B than to the abliterated model: the abliterated model tends to fall into heavy repetition when pushed into content that would normally trigger a refusal. The abliteration delta is small compared to the adapter, so the adapter transfers to the standard weights.

Adapter

Table

attention / shared expert	rank 64 / alpha 32 (scaling 0.5), 300 adapters
routed experts	rank 32 / alpha 16 (scaling 0.5), 36864 adapters (48 layers x 256 experts x 3)
Gated DeltaNet layers (36)	`linear_attn.in_proj_qkv in_proj_z out_proj`
gated full-attention layers (12)	`self_attn.q_proj k_proj v_proj o_proj`
MoE layers (48)	shared expert `gate_proj up_proj down_proj` + all 256 routed experts `gate_proj up_proj down_proj`
not adapted	routers, shared-expert gate, GDN `in_proj_a/b` and conv, norms, embeddings, lm_head, vision tower, MTP layer
no rsLoRA, no DoRA, dropout 0, bf16

Files

Table with columns: Path, Format, Size
Path	Format	Size
`adapter_model.safetensors`, `adapter_config.json`	PEFT (transformers module names)	11.6 GB
`export_info.json`	how the PEFT file was produced (ranks, expert layout)
`nmoe/adapter.safetensors`	nmoe-lora-v1, attention / shared expert	0.31 GB
`nmoe/adapter_experts.safetensors`	nmoe-lora-v1, per-expert `model.layers.N.mlp.experts.E.{gate,up,down}_proj.lora_a/b`	9.67 GB
`nmoe/adapter_config.json`	trainer config (`r`, `lora_alpha`, `expert_r`, `expert_lora_alpha`, ...)
`nmoe/state.json`	trainer state at the checkpoint

The layout is identical to cgg507/Qwen3.5-122B-A10B-GM-E32-LoRA; see that card for how the routed experts are stored in the PEFT file (target_parameters on the fused gate_up_proj / down_proj, exact block-diagonal representation, nothing approximated).

Usage (PEFT)

Needs transformers with qwen3_5_moe and peft >= 0.21 (target_parameters).

python

import torchfrom transformers import AutoModelForImageTextToTextfrom peft import PeftModel base = AutoModelForImageTextToText.from_pretrained(    "Qwen/Qwen3.5-122B-A10B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "cgg507/Qwen3.5-122B-A10B-GM-CoT-LoRA")# optional: model = model.merge_and_unload()

Use thinking mode (the default Qwen3.5 chat template, enable_thinking=True): the adapter was trained with the generation prompt ending in <think>\n, reasoning, </think>\n\n, then the reply. Load the base with AutoModelForImageTextToText; AutoModelForCausalLM builds a model whose module paths do not match the adapter.

Usage (nmoe)

nmoe applies the attention / shared-expert adapters at load and the routed-expert adapters at runtime against nvidia/Qwen3.5-122B-A10B-NVFP4:

bash

nmoe serve /models/Qwen3.5-122B-A10B-NVFP4 --gpu --tp-size 2 --think --lora-path Qwen3.5-122B-A10B-GM-CoT-LoRA/nmoe

--think opens the think block in the prompt. Measured on 2x Radeon AI PRO R9700: ~32 tok/s decode, ~120 tok/s prefill with the adapter applied.

Validation

Checked with scripts/aws_gguf/validate_peft_e32.py (refusal_bench) against the trainer checkpoint and the BF16 base Qwen/Qwen3.5-122B-A10B (safetensors headers and single tensors by HTTP range request):

tensors: peft=600 nmoe=600 adapters=300 byte-identical=600/600
36864/36864 expert matrices decode bit-exactly (A and B), off-blocks zero
PEFT modules resolving to a base weight: matched=300 unmatched=0
target_modules selects 300 base modules: adapter=300 extra=0 missing=0
shape match 300/300 base dtypes {'BF16': 300}
expert base weights (fused gate_up_proj / down_proj per layer): shape+BF16 match 36864/36864
merged samples, W from the BF16 base, W + 0.5 * B @ A from the PEFT tensors == from the nmoe tensors (bitwise), 10 non-expert targets and 3 matrices x 4 experts in layers 0 / 24 / 47, all True, e.g. layers.3.self_attn.q_proj W(16384, 3072) peft==nmoe:True |dW|/|W|=3.38e-03, layers.0.linear_attn.in_proj_qkv W(12288, 3072) peft==nmoe:True |dW|/|W|=3.74e-03, L24 e155 down_proj W(3072, 1024) merge(nmoe)==merge(peft):True |dW|/|W|=1.17e-02

The exporter and the PEFT key layout are the same as for cgg507/Qwen3.5-122B-A10B-GM-E32-LoRA, where the meta-device transformers + peft injection check and the tiny-model PEFT logits check were run; they were not re-run for this adapter.

Training

Data: 731 GM roleplay conversations (17,449 assistant turns; 1,306 tree documents after splitting at the 49,152-token max sequence). Every assistant turn has a reasoning block written by GLM-5.3-Flash-GM given the system prompt, the full prior conversation and the actual reply: an appropriate chain of thought aligned to the system prompt that leads to the presented assistant message. Duplicated / glued reasoning blocks were repaired before training. Some replies in the source data are cut off mid-sentence and were left as is.
Every turn's reasoning is trained. History turns are shown without their reasoning (as the chat template renders them at inference), and each assistant turn is a separate target (think + reply + stop). This was computed exactly with a shared-history tree: one trunk per conversation plus one branch per turn that attends to the trunk prefix before its branch point (linear-attention state saved and resumed at branch points), so the 17,449 per-turn samples cost 47M rows instead of 247M. Gradients are identical to training the per-turn samples separately.
24.5M trained tokens, max sequence 49,152, 1 epoch, 145 steps (~9 tree documents per step), shuffled order. Muon optimizer (expert adapters: bf16 masters + stochastic rounding), lr 2e-4, warmup 5, cosine to 10%, grad clip 1.0.
Loss 1.42 (base, first steps) -> 1.18 (mean of the last 10 steps), final step 1.165 (perplexity ~4.1 -> ~3.25 on trained tokens).
Trained with a custom C++/ROCm trainer (nmoe, --cot-branches --expert-rank 32) on 2x Radeon AI PRO R9700 against the abliterated NVFP4 base, ~620 rows/s, ~21 h (one resume from the step-110 checkpoint, exact: optimizer state and data position restored).

Training

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

cgg507

Model Tree

Base

Qwen/Qwen3.5-122B-A10B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Recommended use: train on abliterated, run on the standard base

Adapter

Table

attention / shared expert	rank 64 / alpha 32 (scaling 0.5), 300 adapters
routed experts	rank 32 / alpha 16 (scaling 0.5), 36864 adapters (48 layers x 256 experts x 3)
Gated DeltaNet layers (36)	`linear_attn.in_proj_qkv in_proj_z out_proj`
gated full-attention layers (12)	`self_attn.q_proj k_proj v_proj o_proj`
MoE layers (48)	shared expert `gate_proj up_proj down_proj` + all 256 routed experts `gate_proj up_proj down_proj`
not adapted	routers, shared-expert gate, GDN `in_proj_a/b` and conv, norms, embeddings, lm_head, vision tower, MTP layer
no rsLoRA, no DoRA, dropout 0, bf16

Files

Table with columns: Path, Format, Size
Path	Format	Size
`adapter_model.safetensors`, `adapter_config.json`	PEFT (transformers module names)	11.6 GB
`export_info.json`	how the PEFT file was produced (ranks, expert layout)
`nmoe/adapter.safetensors`	nmoe-lora-v1, attention / shared expert	0.31 GB
`nmoe/adapter_experts.safetensors`	nmoe-lora-v1, per-expert `model.layers.N.mlp.experts.E.{gate,up,down}_proj.lora_a/b`	9.67 GB
`nmoe/adapter_config.json`	trainer config (`r`, `lora_alpha`, `expert_r`, `expert_lora_alpha`, ...)
`nmoe/state.json`	trainer state at the checkpoint

Usage (PEFT)

Needs transformers with qwen3_5_moe and peft >= 0.21 (target_parameters).

python

import torchfrom transformers import AutoModelForImageTextToTextfrom peft import PeftModel base = AutoModelForImageTextToText.from_pretrained(    "Qwen/Qwen3.5-122B-A10B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "cgg507/Qwen3.5-122B-A10B-GM-CoT-LoRA")# optional: model = model.merge_and_unload()

import torchfrom transformers import AutoModelForImageTextToTextfrom peft import PeftModel base = AutoModelForImageTextToText.from_pretrained(    "Qwen/Qwen3.5-122B-A10B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "cgg507/Qwen3.5-122B-A10B-GM-CoT-LoRA")# optional: model = model.merge_and_unload()

Usage (nmoe)

nmoe applies the attention / shared-expert adapters at load and the routed-expert adapters at runtime against nvidia/Qwen3.5-122B-A10B-NVFP4:

bash

nmoe serve /models/Qwen3.5-122B-A10B-NVFP4 --gpu --tp-size 2 --think --lora-path Qwen3.5-122B-A10B-GM-CoT-LoRA/nmoe

nmoe serve /models/Qwen3.5-122B-A10B-NVFP4 --gpu --tp-size 2 --think --lora-path Qwen3.5-122B-A10B-GM-CoT-LoRA/nmoe

--think opens the think block in the prompt. Measured on 2x Radeon AI PRO R9700: ~32 tok/s decode, ~120 tok/s prefill with the adapter applied.

Validation

tensors: peft=600 nmoe=600 adapters=300 byte-identical=600/600
36864/36864 expert matrices decode bit-exactly (A and B), off-blocks zero
PEFT modules resolving to a base weight: matched=300 unmatched=0
target_modules selects 300 base modules: adapter=300 extra=0 missing=0
shape match 300/300 base dtypes {'BF16': 300}
expert base weights (fused gate_up_proj / down_proj per layer): shape+BF16 match 36864/36864
merged samples, W from the BF16 base, W + 0.5 * B @ A from the PEFT tensors == from the nmoe tensors (bitwise), 10 non-expert targets and 3 matrices x 4 experts in layers 0 / 24 / 47, all True, e.g. layers.3.self_attn.q_proj W(16384, 3072) peft==nmoe:True |dW|/|W|=3.38e-03, layers.0.linear_attn.in_proj_qkv W(12288, 3072) peft==nmoe:True |dW|/|W|=3.74e-03, L24 e155 down_proj W(3072, 1024) merge(nmoe)==merge(peft):True |dW|/|W|=1.17e-02

Training

Data: 731 GM roleplay conversations (17,449 assistant turns; 1,306 tree documents after splitting at the 49,152-token max sequence). Every assistant turn has a reasoning block written by GLM-5.3-Flash-GM given the system prompt, the full prior conversation and the actual reply: an appropriate chain of thought aligned to the system prompt that leads to the presented assistant message. Duplicated / glued reasoning blocks were repaired before training. Some replies in the source data are cut off mid-sentence and were left as is.
Every turn's reasoning is trained. History turns are shown without their reasoning (as the chat template renders them at inference), and each assistant turn is a separate target (think + reply + stop). This was computed exactly with a shared-history tree: one trunk per conversation plus one branch per turn that attends to the trunk prefix before its branch point (linear-attention state saved and resumed at branch points), so the 17,449 per-turn samples cost 47M rows instead of 247M. Gradients are identical to training the per-turn samples separately.
24.5M trained tokens, max sequence 49,152, 1 epoch, 145 steps (~9 tree documents per step), shuffled order. Muon optimizer (expert adapters: bf16 masters + stochastic rounding), lr 2e-4, warmup 5, cosine to 10%, grad clip 1.0.
Loss 1.42 (base, first steps) -> 1.18 (mean of the last 10 steps), final step 1.165 (perplexity ~4.1 -> ~3.25 on trained tokens).
Trained with a custom C++/ROCm trainer (nmoe, --cot-branches --expert-rank 32) on 2x Radeon AI PRO R9700 against the abliterated NVFP4 base, ~620 rows/s, ~21 h (one resume from the step-110 checkpoint, exact: optimizer state and data position restored).

Training