GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
thediymaker
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
thediymaker
Model Tree
Input Modalities
Output Modalities
Supported Functionality
python
from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("thediymaker/gaudi2-fineweb-edu-1b")model = AutoModelForCausalLM.from_pretrained("thediymaker/gaudi2-fineweb-edu-1b", torch_dtype="bfloat16")ids = tok("Photosynthesis is the process by which", return_tensors="pt").input_idsprint(tok.decode(model.generate(ids, max_new_tokens=32, do_sample=False)[0]))Greedy completions from this checkpoint:
| Architecture | LlamaForCausalLM: 22 layers, hidden size 2,048, SiLU MLP of 5,632, 32 attention heads with 4 KV heads, RoPE (theta 10,000), RMSNorm, tied input and output embeddings |
| Parameters | 1,069,639,680 |
| Context length | 2,048 tokens |
| Tokenizer | The SmolLM2 tokenizer (HuggingFaceTB/SmolLM2-135M @ 93efa2f), 49,152 tokens |
| Weights | bf16 (trained with fp32 master weights) |
FineWeb-Edu, default config, revision 87f0914: all 2,410 parquet files, tokenized to 1,598,580,326,400 training tokens. Every token was trained on exactly once. Validation loss is measured on 10M tokens from a held-out file.
| Method | DiLoCo: each worker trains its own copy of the model for one round, then a coordinator averages the workers' parameter changes (weighted by tokens) and applies them with an outer optimizer |
| Workers | Up to 32 workers of 4 Gaudi2 cards each, data-parallel inside a worker. Serving load preempted workers and freed cards started new ones: 30.9 live on average, as few as 16, 658 worker starts in all |
| Inner optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1, gradient clip 1.0, 131,072 tokens per step; peak LR 4e-4, 2B-token warmup, cosine to 4e-5 over the global token count |
| Outer optimizer | SGD with Nesterov momentum 0.9, LR 0.7. 100 chunks of 1,048,576 tokens (800 AdamW steps) per worker per round, after 10 rounds of 50; deltas sent in bf16 |
| Rounds | 503 |
| Precision | bf16 autocast, fp32 master weights and optimizer state |
| Wall clock | 7.2 days |
| Compute | 21,579 Gaudi2 card-hours; 2.56M tokens/s overall (30.8% MFU) |
| Sync overhead | 15.8% of each round |
| Work lost to preemption | 0.4% of trained steps, up to 0.5% counting workers that died mid-round without reporting their steps |
| Final validation loss | 2.079 |
lm-evaluation-harness 0.4.9, bf16, batch size 32, one Gaudi2 card, the same setup for every model below. Scores are acc_norm where the task defines it, else acc; 0-shot except MMLU (5-shot). The reference models were evaluated here, not copied from their cards.
| Model | Tokens | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | OBQA | LAMBADA | MMLU (5-shot) |
|---|---|---|---|---|---|---|---|---|---|
| This model | 1,598.6B | 0.608 | 0.689 | 0.422 | 0.746 | 0.605 | 0.422 | 0.514 | 0.246 |
| This run, 1T milestone | 1,000B | 0.582 | 0.668 | 0.404 | 0.736 | 0.610 | 0.388 | 0.511 | 0.250 |
| This run, 650B milestone | 650B | 0.558 | 0.649 | 0.391 | 0.733 | 0.577 | 0.402 | 0.502 | 0.251 |
| This run, 300B milestone | 300B | 0.530 | 0.628 | 0.358 | 0.717 | 0.590 | 0.386 | 0.488 | 0.264 |
| This run, 150B milestone | 150B | 0.497 | 0.598 | 0.340 | 0.699 | 0.555 | 0.370 | 0.465 | 0.261 |
| This run, 50B milestone | 50B | 0.413 | 0.537 | 0.292 | 0.686 | 0.540 | 0.338 | 0.383 | 0.270 |
| TinyLlama 1.1B base (v1.1) @ ff3c701 | see its card | 0.614 | 0.556 | 0.328 | 0.737 | 0.591 | 0.364 | 0.560 | 0.262 |
| SmolLM2-1.7B @ effd688 | see its card | 0.714 | 0.732 | 0.472 | 0.775 | 0.662 | 0.444 | 0.674 | 0.501 |
MMLU has four choices, so 0.25 is chance: this model and TinyLlama do not score above it at 5-shot.
Apache-2.0. The training data, FineWeb-Edu, is released under ODC-By 1.0 and is subject to the Common Crawl terms of use. The tokenizer is from SmolLM2 (Apache-2.0).
from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("thediymaker/gaudi2-fineweb-edu-1b")model = AutoModelForCausalLM.from_pretrained("thediymaker/gaudi2-fineweb-edu-1b", torch_dtype="bfloat16")ids = tok("Photosynthesis is the process by which", return_tensors="pt").input_idsprint(tok.decode(model.generate(ids, max_new_tokens=32, do_sample=False)[0]))