GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Hplm
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Hplm
Model Tree
Input Modalities
Output Modalities
Supported Functionality
LlamaForCausalLM), trained from scratch| Parameters | 631,504,512 |
| Layers | 42 |
| Hidden size | 1152 |
| Feed-forward size | 3072 |
| Attention heads | 18 (head dim 64) |
| Key/value heads | 6 (grouped-query attention, 3:1) |
| Max positions | 1024 |
| Vocabulary | 32,000 (BPE) |
| Embeddings | tied input/output |
| Weights | float32, safetensors |
The design is deep and thin with grouped-query attention and tied embeddings, scaled up from the BabyLlama-2 recipe.
python
from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("hplm/edgar-base")model = AutoModelForCausalLM.from_pretrained("hplm/edgar-base") prompt = "It was upon a dreary evening in the autumn of the year"inputs = tok(prompt, return_tensors="pt")out = model.generate(**inputs, max_new_tokens=120, do_sample=True, top_p=0.95, temperature=0.8)print(tok.decode(out[0], skip_special_tokens=True))The corpus mixes two streams:
The assembled training set is 363.7M tokens; the dev set is 5.8M tokens.
Training follows the BabyLlama-style ensemble distillation recipe:
Overparamaterization relative to data sizing is intentional; the lever is high weight decay
| Hyperparameter | Value |
|---|---|
| Epochs | 4 (about 1.45B tokens seen) |
| Optimizer steps | 22,196 |
| Sequence length | 512 |
| Effective batch size | 128 |
| Learning rate | 7e-4 |
| Weight decay | 5.0 |
| Warmup steps | 600 |
| Distillation alpha / temperature | 0.5 / 1.0 |
| Precision | bf16 |
Results on the held-out dev set at the end of training:
| Model | Cross-entropy loss | Perplexity |
|---|---|---|
| Teacher 1 | 3.297 | 27.0 |
| Teacher 2 | 3.295 | 27.0 |
| Student (this model) | 3.202 | 24.6 |
Filtered BLIMP: 0.72
Currently, if you use this model, please cite the paper the training framework accompanies: Pretraining Language Models for Diachronic Linguistic Change Discovery. An additional preprint will come shortly.
from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("hplm/edgar-base")model = AutoModelForCausalLM.from_pretrained("hplm/edgar-base") prompt = "It was upon a dreary evening in the autumn of the year"inputs = tok(prompt, return_tensors="pt")out = model.generate(**inputs, max_new_tokens=120, do_sample=True, top_p=0.95, temperature=0.8)print(tok.decode(out[0], skip_special_tokens=True))