GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
spitfire4794
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
spitfire4794
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Specification | Value |
|---|---|
| Architecture | LlamaForCausalLM |
| Parameters | ~50M |
| Vocabulary Size | 32,002 (Expanded for ChatML) |
| Hidden Size | 512 |
| Layers | 12 |
| Attention Heads | 8 |
| KV Heads | 4 |
| Context Length | 5,120 tokens |
| Tokenizer | BPE tokenizer (ChatML Jinja template applied) |
| Native EOS Token | `< |
The model's 3B token CPT corpus included raw ChatML text. As a result, the base model originally learned the tags <|im_start|> and <|im_end|> as sequences of 7 separate subwords.
To convert these tags into single tokens without destroying the learned pre-trained representations, we used subword-mean initialization:
This ensures stable loss at the start of fine-tuning and allows inference frameworks to natively stop generation when <|im_end|> is predicted.
This is a base model, not an instruct model. Training used packed raw text with standard causal language-modeling loss:
labels = input_idsThe training mix prepared for the CPT run was:
This model is intended for Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), including reasoning-style (R1) alignments.
You can use standard Hugging Face tools and call tokenizer.apply_chat_template() directly on your datasets.
LoRA Configuration Note: Because the vocabulary size was expanded from 32,000 to 32,002, you must train the embedding layers during fine-tuning so the model can update the new ChatML tokens. Add the following to your PEFT/LoRA configuration:
python
modules_to_save=["embed_tokens", "lm_head"]modules_to_save=["embed_tokens", "lm_head"]