Qwen1.5-MoE-A2.7B-Finetuning API & Inference Endpoint

Cau hinh LoRA

Layer duoc finetune: [8, 16) trong tong so 24 layer (tuong ung khoang 1L/3 -> 2L/3).
Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi thanh phan mot rank rieng qua rank_pattern cua PEFT:
- attention: r = 16
- router: r = 4
- experts: r = 16
alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

L_LM: cross-entropy chuan tren token tiep theo.
L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
lb_loss_coef = None
num_experts = 60, top_k = 4

Du lieu

Alignment data duoc chon qua --alignment_data (flores ntrex ted), gom sample tu cac file: flores.json, ntrex.json, ted.json (moi field ngon ngu trong 1 record duoc coi la 1 sample), shuffle va sort theo do dai token truoc khi gom batch.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png (bieu do tho L_LM / L_LB / L_Total theo tung step) va diagnostics/loss_curve_smoothed.png (cung 3 duong loss nhung da lay trung binh cong moi 50 step lien tiep — de doc xu huong hon vi bieu do tho rat messy o cap do tung step).

Cau hinh LoRA

Layer duoc finetune: [8, 16) trong tong so 24 layer (tuong ung khoang 1L/3 -> 2L/3).
Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi thanh phan mot rank rieng qua rank_pattern cua PEFT:
- attention: r = 16
- router: r = 4
- experts: r = 16
alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

L_LM: cross-entropy chuan tren token tiep theo.
L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
lb_loss_coef = None
num_experts = 60, top_k = 4

Qwen1.5-MoE-A2.7B-Finetuning

README

Cau hinh LoRA

Loss

Du lieu

Diagnostics

Explore FriendliAI today

README

Cau hinh LoRA

Loss

Du lieu

Diagnostics