GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
DeependraVerma
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
DeependraVerma
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Independently recomputed (not the in-training number) over the full held-out validation set: loss 0.3939, perplexity 1.483.
All numbers from lm-evaluation-harness, 0-shot, run on this checkpoint as part of its own publish pipeline (not hand-picked afterwards).
| Task | Metric | Value |
|---|---|---|
| hellaswag | acc,none | 0.3209 |
| hellaswag | acc_norm,none | 0.3610 |
| arc_easy | acc,none | 0.5025 |
| arc_easy | acc_norm,none | 0.4693 |
| piqa | acc,none | 0.6333 |
| piqa | acc_norm,none | 0.6240 |
| Task | Metric | Value |
|---|---|---|
| mmlu_professional_law | acc,none | 0.2425 |
| mmlu_jurisprudence | acc,none | 0.2315 |
| mmlu_international_law | acc,none | 0.2397 |
| Task | Metric | Value |
|---|---|---|
| case_hold | acc,none | 0.1311 |
| case_hold | acc_norm,none | 0.2214 |
| Task | Metric | Value |
|---|---|---|
| legalbench_citation_prediction_classification | acc,none | 0.5093 |
| legalbench_consumer_contracts_qa | acc,none | 0.5606 |
| legalbench_contract_nli_confidentiality_of_agreement | acc,none | 0.5122 |
| legalbench_contract_nli_explicit_identification | acc,none | 0.8165 |
Read the legal-task numbers carefully. A 125M model does not reliably do legal reasoning — CaseHOLD, MMLU-legal, and LegalBench scores in the neighborhood of chance/majority-class are expected and were seen on this project's earlier 125M SFT model too; they are reported here in full rather than omitted.
Pretraining — warm-started from run 1's base weights
(slm-125m-ultimate-base), continued
on markup-stripped legal/financial + general data. See the base model card for
full pretraining details.
Fine-tuning — teacher-distilled, judge-filtered Q&A pairs, covering legal
Q&A, contract drafting, full-document generation, and general chit-chat/refusal.
See run_post_pretrain_to_sft.py
for the exact pipeline. Validated with an independent re-evaluation
(evaluate_sft.py) and the full benchmark suite above before publishing —
this model card's numbers are read directly from that run's own report, not
retyped by hand.
| Parameters | 125,847,552 (tied embeddings) |
| Layers / hidden / heads | 12 / 768 / 12 (no GQA) |
| Context | 4096 |
| Vocabulary | 16,384 (custom BPE, domain-trained, shared across every model in this project) |
| Precision | bfloat16 |
Llama architecture, so it loads with stock transformers.
slm-125m-ultimate-baselegal-slm-125m-ultimate-sft-onnxMIT.
| legalbench_contract_nli_limited_use | acc,none | 0.5337 |
| legalbench_contract_nli_no_licensing | acc,none | 0.5556 |
| legalbench_contract_nli_notice_on_compelled_disclosure | acc,none | 0.6127 |
| legalbench_contract_nli_permissible_copy | acc,none | 0.7701 |
| legalbench_contract_nli_return_of_confidential_information | acc,none | 0.5000 |
| legalbench_contract_nli_sharing_with_employees | acc,none | 0.5588 |
| legalbench_contract_nli_survival_of_obligations | acc,none | 0.4777 |
| legalbench_contract_qa | acc,none | 0.5125 |
| legalbench_telemarketing_sales_rule | acc,none | 0.4043 |