GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
bizmod
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
bizmod
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Artefact | What it is | Status |
|---|---|---|
model.safetensors (289 MB) | 72,410,880 params — 8 layers, hidden 384, FFN 1024, ctx 256, tied embeddings. Trained 300 steps on a Tesla T4 (loss 12.00 → 4.98). | Real, trained, loadable |
config_10b.json | The 10.00B Datlas 2.5 target architecture: 43 layers, d=4096, FFN 14336, 32 heads / 8 KV heads (GQA), ctx 4096, vocab 151,936, tied embeddings → 10,001,414,144 parameters. | Spec + config only — no weights yet |
train_datlas_10b.py | The exact training engine for that 10B architecture (streaming parquet, prompt-masked loss, OneCycleLR, AdamW, checkpointing). | Runnable on multi-GPU |
DATLAS_10B_SPEC.md | Hardware + data budget for the 10B run. | Reference |
train_datlas_kaggle.py + training_log.json | The exact 72M training script and its metrics. | Archive |
A 10B from-scratch model is 20.4 GB of bf16 weights before a single optimizer state, i.e. ≈80 GB of AdamW state on top. A Kaggle T4 has 16 GB. A 10B checkpoint therefore cannot be produced in the Kaggle notebook, and this repository will not publish weights that were never trained.
To build it, run train_datlas_10b.py on rented multi-GPU:
bash
torchrun --nproc_per_node=8 train_datlas_10b.py \ --seq-len 4096 --micro-batch 1 --grad-accum 16 --steps 20000Minimum: 8×A100-80GB, or 4×H100-80GB with FSDP/ZeRO-3. Budget ≈2–3 T tokens for a Chinchilla-optimal 10B, so the 260K-row corpus (~692M tokens) gives a domain-specialised model, not a general one. If a usable model today matters more than the from-scratch purity, QLoRA a pre-trained ~9–10B base on the same corpus instead.
python
from transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained("bizmod/Datlas-2.5")model = AutoModelForCausalLM.from_pretrained("bizmod/Datlas-2.5")msgs = [{"role": "user", "content": "What should I check before deploying a crawler?"}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")print(tok.decode(model.generate(ids, max_new_tokens=64)[0][ids.shape[-1]:], skip_special_tokens=True))user_prompt / agent_reasoning with
role/industry/os metadata; prompt tokens masked from the loss (−100).Two published routes, both shipped in this repo (see DATLAS_5B_SPEC.md):
config_5b.json = 4,896,473,088 params (44 layers, d=3072, ffn 8192,
24 heads / 8 KV heads, ctx 4096, tied embeddings). Trains with train_datlas_5b.py on >= 8xA100-80GB.config_5b_dus.json = 4,939,213,824 params.
build_datlas_5b_upscale.py takes the pretrained Qwen2.5-3B (36 layers) and duplicates 24 decoder
blocks (LLaMA Pro / SOLAR style) to 60 layers, then train_datlas_5b.py --init-from ./datlas-5b-dus
continues pretraining. Because the base weights are already trained, a few hundred steps heal the
duplicated blocks instead of starting from noise.train_datlas_5b_kaggle.py loads the depth up-scaled 5B in 4-bit NF4
(2.7 GB) with LoRA adapters and continues pretraining on the Edge-Agent-Reasoning corpus.torchrun --nproc_per_node=8 train_datlas_10b.py \ --seq-len 4096 --micro-batch 1 --grad-accum 16 --steps 20000from transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained("bizmod/Datlas-2.5")model = AutoModelForCausalLM.from_pretrained("bizmod/Datlas-2.5")msgs = [{"role": "user", "content": "What should I check before deploying a crawler?"}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")print(tok.decode(model.generate(ids, max_new_tokens=64)[0][ids.shape[-1]:], skip_special_tokens=True))