GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ducanhdinh
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ducanhdinh
Model Tree
Qwen/Qwen1.5-MoE-A2.7B
Input Modalities
Output Modalities
Supported Functionality
text
Context: <context>Question: <question>Answer: <answer_text>Phan sau Answer: la phan model sinh ra. Prompt nay duoc thiet ke de dung TRUC TIEP cho
zero-shot eval tren cac benchmark QA extractive khac cung dinh dang, vi du XQuAD: chi can
generate() tu prompt (khong co phan dap an) roi cat lay text truoc token eos.
[8, 16) trong tong so 24 layer
(tuong ung khoang 1L/3 -> 2L/3), giong het setting cua code SNLI/alignment finetuning.rank_pattern cua PEFT:
Loss MoE tieu chuan:
L_total = L_LM + lb_loss_coef * L_LB
L_LM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan <answer_text><eos>
(prompt Context/Question bi mask, giong instruction-tuning/SFT tieu chuan).L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam
trong khoang layer duoc finetune.lb_loss_coef (lambda) = 0.01num_experts = 60, top_k = 4SQuAD, doc tu data/english_task/squad/train.json (list object
{"id", "title", "context", "question", "answers": {"text", "answer_start"}}). Cac cau hoi
khong co dap an (SQuAD 2.0 style) bi loc bo. Context qua dai (vuot --max_length) duoc "windowing"
quanh vi tri answer thay vi bi truncate tho, de dam bao answer luon nam trong phan context duoc
giu lai (xem select_context_window trong script training).
Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png,
diagnostics/loss_curve_smoothed.png va diagnostics/accuracy_curve.png (ty le token dung xap
xi tren TOAN BO token cua answer — teacher forcing, tinh tren batch train, da lam min moi
50 step; day KHONG phai Exact Match/F1 chuan cua SQuAD, chi la diagnostics
theo doi qua trinh train).