GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Raghav-Singhal
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Raghav-Singhal
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Llama-style decoder, 24 layers, hidden 2,048, FFN 8,192 (SwiGLU), attention heads / KV heads 16 / 4 (head dim 128), RMSNorm, RoPE base 10,000, untied embeddings, no biases, no QK-norm, sequence length 4,096. Tokenizer: SmolLM2 vocabulary (49,152) plus <|pad|>; <|endoftext|> is the end-of-document token.
Data: the 1PP conversations rewritten from those documents; loss on user and assistant turns (no loss on <|endoftext|>). One pass over 47.8M documents (66.2B tokens of original documents; 63.0B tokens as conversations), 31,777 steps at global batch 512 × 4,096 tokens, cross-document attention masking, best-fit packing with step-aligned document assignment. Optimizer: Muon (shape scaling, matrix LR 0.005) with Adam for embeddings and norms, warmup 2,000 steps, constant, linear decay over the last 10% to 1/100, weight decay 0.1, bf16.
Validation loss (per token, 2,433 held-out documents, final checkpoint):
| assistant text | user text | document text |
|---|---|---|
| 1.433 | 1.339 | 3.123 |
ChatML without a system turn (the models never saw one):
markdown
<|im_start|>user\n{message}<|im_end|>\n<|im_start|>assistant\n{reply}<|im_end|>\n
The bundled chat_template renders exactly this. Generation stops at <|im_end|> (id 2) or <|endoftext|> (id 0); both are listed in eos_token_id. This is a base model; the conversation conditions produce chat-formatted text, the raw baseline plain text.
The HF weights were checked against the Megatron checkpoint by recomputing validation losses with this model:
| set | HF loss | Megatron reference | abs. diff |
|---|---|---|---|
| val50m segments [3] | 1.4312 | 1.4325 | 0.0014 |
| raw_val50m segments [8] | 3.1243 | 3.1226 | 0.0017 |