GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ducanhdinh
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ducanhdinh
Model Tree
Qwen/Qwen1.5-MoE-A2.7B
Input Modalities
Output Modalities
Supported Functionality
text
The following are multiple choice questions (with answers) about <subject>. <question>A. <choice_A>B. <choice_B>C. <choice_C>D. <choice_D>Answer: <letter>Neu khong co subject (nhu du lieu auxiliary_train.json), header bo phan "about ".
Phan sau Answer: la phan model sinh ra. Prompt nay DUNG CHUAN format pho bien khi eval MMLU
(lm-evaluation-harness), nen dung TRUC TIEP duoc cho zero-shot eval tren cac benchmark trac
nghiem cung dinh dang, vi du MMMLU (ban da ngon ngu cua MMLU): chi can thay cau hoi/lua chon
bang du lieu ngon ngu do, giu nguyen cau truc, roi so sanh xac suat token dap an ung voi
A/B/C/D... (hoac generate() va lay chu cai dau tien sinh ra).
[8, 16) trong tong so 24 layer
(tuong ung khoang 1L/3 -> 2L/3), giong het setting cua cac code SNLI/SQuAD/alignment.rank_pattern cua PEFT:
Loss MoE tieu chuan:
L_total = L_LM + lb_loss_coef * L_LB
L_LM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan <letter><eos> (prompt
cau hoi + cac lua chon bi mask, giong instruction-tuning/SFT tieu chuan).L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam
trong khoang layer duoc finetune.lb_loss_coef (lambda) = 0.01num_experts = 60, top_k = 4MMLU, doc tu data/english_task/mmlu/auxiliary_train.json (list object
{"question", "choices": [...], "answer": int, "subject"}). Sample nao co full_text vuot qua
--max_length bi bo qua (khong truncate, de tranh cat mat chu cai dap an nam o cuoi chuoi).
Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png,
diagnostics/loss_curve_smoothed.png va diagnostics/accuracy_curve.png (ty le token dung xap
xi tren toan bo token cua nhan <letter><eos> — teacher forcing, tinh tren batch train, da lam
min moi 50 step; day KHONG phai do chinh xac chuan tren tap eval MMLU/MMMLU,
chi la diagnostics theo doi qua trinh train).
The following are multiple choice questions (with answers) about <subject>.<question>A. <choice_A>B. <choice_B>C. <choice_C>D. <choice_D>Answer: <letter>