ducanhdinh

Macro-Nano-Instruct-Finetuning

Deploy Dedicated

README

License: apache-2.0

Cau hinh LoRA

Layer duoc finetune: [9, 18) trong tong so 28 layer (tuong ung khoang 1L/3 -> 2L/3).
Module duoc gan LoRA: attention, router, experts trong khoang layer tren.
r = 16, alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

L_LM: cross-entropy chuan tren token tiep theo.
L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
lb_loss_coef = None
num_experts = 232, top_k = 8

Du lieu

Cau don ngu duoc gom tu 3 bo du lieu alignment: flores.json, bible.json, ntrex.json (moi field ngon ngu trong 1 record duoc coi la 1 sample), shuffle va sort theo do dai token truoc khi gom batch.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step) va diagnostics/loss_curve.png (bieu do L_LM / L_LB / L_Total theo step).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ducanhdinh

Model Tree

Base

ATH-MaaS/Marco-Nano-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities