Macro-Nano-Instruct-MidAlign API & Inference Endpoint

Cau hinh LoRA / Alignment

LoRA + trich xuat hidden state cho contrastive loss CHI ap dung tai layer thu 16 (0-indexed block = 15) trong tong so 28 layer.
Module duoc gan LoRA: attention, router, experts tai layer tren.
r = 16, alpha = 32, dropout = 0.05
Nhiet do contrastive tau = 0.1

Loss (Alternate Training — moi step chi 1 trong 2)

Task step: L_task = L_LM + lb_loss_coef * L_LB
- L_LM: causal LM loss tren cau TARGET LANGUAGE (phia "other" trong cap english-other).
- L_LB: load balancing loss chuan cua MoE tai router trong layer duoc finetune.
- lb_loss_coef = None, num_experts = 232, top_k = 8
Align step: L_align = symmetric InfoNCE / contrastive loss (in-batch negatives) giua mean-pooled hidden state cua cau tieng Anh va cau target tai layer 16.

Du lieu

Cap bitext english-other duoc sample tu cac bo du lieu multiway-parallel: flores.json, bible.json, ntrex.json. Voi moi record, cau eng_Latn duoc ghep voi tung ngon ngu khac trong cung record de tao 1 cap bitext rieng.

Training

3 epoch, batch_size = 64 (per-process).
Multi-GPU: DistributedDataParallel (torchrun), checkpoint chi giu ban moi nhat.
DDP: find_unused_parameters = True (bat buoc True voi MoE + LoRA 1 layer de tranh NCCL Watchdog SIGABRT khi co expert khong nhan token trong 1 step/GPU), NCCL timeout = 30 phut.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step, phan biet step_type=task/align) va diagnostics/loss_curve.png.

Cau hinh LoRA / Alignment

LoRA + trich xuat hidden state cho contrastive loss CHI ap dung tai layer thu 16 (0-indexed block = 15) trong tong so 28 layer.
Module duoc gan LoRA: attention, router, experts tai layer tren.
r = 16, alpha = 32, dropout = 0.05
Nhiet do contrastive tau = 0.1

Loss (Alternate Training — moi step chi 1 trong 2)

Task step: L_task = L_LM + lb_loss_coef * L_LB
- L_LM: causal LM loss tren cau TARGET LANGUAGE (phia "other" trong cap english-other).
- L_LB: load balancing loss chuan cua MoE tai router trong layer duoc finetune.
- lb_loss_coef = None, num_experts = 232, top_k = 8
Align step: L_align = symmetric InfoNCE / contrastive loss (in-batch negatives) giua mean-pooled hidden state cua cau tieng Anh va cau target tai layer 16.

Du lieu

Training

3 epoch, batch_size = 64 (per-process).
Multi-GPU: DistributedDataParallel (torchrun), checkpoint chi giu ban moi nhat.
DDP: find_unused_parameters = True (bat buoc True voi MoE + LoRA 1 layer de tranh NCCL Watchdog SIGABRT khi co expert khong nhan token trong 1 step/GPU), NCCL timeout = 30 phut.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step, phan biet step_type=task/align) va diagnostics/loss_curve.png.

Macro-Nano-Instruct-MidAlign

README

Cau hinh LoRA / Alignment

Loss (Alternate Training — moi step chi 1 trong 2)

Du lieu

Training

Diagnostics

Explore FriendliAI today

README

Cau hinh LoRA / Alignment

Loss (Alternate Training — moi step chi 1 trong 2)

Du lieu

Training

Diagnostics