GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
bcywinski
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
bcywinski
Model Tree
Qwen/Qwen3.5-9B
Input Modalities
Output Modalities
Supported Functionality
Every other fine-tune in this project runs on Tinker. Tinker refuses to load a
checkpoint trained against Qwen/Qwen3.5-9B-Base into a Qwen/Qwen3.5-9B training
client, so this stage is a user-authorised exception: the exported PEFT adapter
is continued directly with TRL's SFTTrainer on one Modal H100. Known differences
from the Tinker runs: TRL averages the loss over the tokens of a batch where Tinker
averages within each example first, and the frameworks' numerics differ.
| setting | value |
|---|---|
| epochs / effective batch | 1 / 16 sequences |
| optimizer steps | 304 |
| optimizer | AdamW, lr 1e-4, betas 0.9/0.999, eps 1e-8, weight decay 0.01 |
| schedule | cosine, warmup ratio 0.05 |
| gradient clipping | 1.0 |
| LoRA | r=64, alpha=32, dropout=0, 12 target module names |
| max sequence length | 4096 |
| precision / hardware | bf16, 1x H100 |
| seed | 0 |
| held-out NLL before -> after | 0.8080 -> 0.1461 |
| final training loss | 0.2080 |
| training wall clock | 272 s |
Rendering: the cookbook renderer qwen3_5_disable_thinking (the empty <think>
block), asserted token-for-token against the model's own chat template; loss falls on
the final assistant turn only, including its turn-end token.
Alpha deviation. r=64 with lora_alpha=32 is an effective LoRA scale of 0.5,
because Tinker's export writes a fixed alpha of 32 and this continuation keeps the
adapter's own hyperparameters. The paper this recipe follows (arXiv 2605.02087) used
alpha 128 at rank 64, i.e. scale 2. The learning rate was not compensated. The no-MSM
control's fresh LoRA copies rank, alpha, dropout and target modules from the MSM
adapter's own config, so the grid's cells differ only in the weights they start from.
The mean over held-out examples of each example's mean NLL on its supervised tokens
(matching the Tinker runs' loss_reduction: mean). The "before" number is measured
on the initial weights over the same held-out rows, so it shows how much of the AFT
data the initialisation already predicts: the midtrained organisms start around 0.80
to 0.89 and a fresh LoRA on the bare instruct model starts around 1.09 to 1.17.