GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
alea-institute
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
alea-institute
Model Tree
Input Modalities
Output Modalities
Supported Functionality
alea-institute/kl3m-007-500m-step0 (stacked from kl3m-006-170m-checkpoint-63000)| Step Range | Loss (100-avg) | Improvement |
|---|---|---|
| 1-100 | ~7.66 | Baseline (stacked initialization) |
| 5000-5100 | ~2.63 | Layers diverging, specializing |
| 11000-12000 | ~2.52 | Continued improvement |
| 14000-15000 | ~2.63 | Stabilizing |
| 15400-15600 | 2.62-2.81 | Current |
Overall improvement: ~65% reduction in loss from initialization to step 15500.
Based on "Stacking Your Transformers" (NeurIPS 2024, arXiv:2405.15319):
Stacking Pattern (G_stack = G_direct depthwise):
text
Source (30 layers): [0, 1, 2, ..., 28, 29]Target (120 layers): [0-29, 0-29, 0-29, 0-29] └─ Cyclic repetition 4 times (G_stack/G_direct)Method: Direct duplication in depthwise manner (G_stack operator from paper)
Training Philosophy:
Muon Learning Rate: 0.000365 (depth-scaled)
Auxiliary Learning Rate: 0.0005
Muon Parameters:
Per-Layer Learning Rate Multipliers (Phase A):
self_attn.q_proj: 0.7× (slower for better conditioning)self_attn.o_proj: 0.7× (slower for better conditioning)self_attn.k_proj: 0.9×self_attn.v_proj: 0.9×mlp.*: 1.0× (baseline)lm_head: 0.85×Adjusted for 120-layer model with more frequent attention clamping:
Attention layers (q_proj, k_proj, v_proj, o_proj):
MLP layers (gate_proj, up_proj, down_proj):
LM head:
Source: alea-institute/kl3m-data-sample-004-balanced (streaming with buffer=32)
Document Distribution:
Data Characteristics:
Note: The training data is heavily weighted toward court documents (RECAP) which influences the model's generation style and domain expertise.
Analysis of Q-projection weight norms across checkpoints (Steps 14600-15000):
Divergence Timeline:
Steady increase indicates continued layer specialization and differentiation of the 4 duplicated stacks.
Fastest evolving (most specialization):
Slowest evolving (most stable):
Stack-Level Changes (14600 → 15000):
Interpretation: The 4 stacked copies are successfully diverging and specializing, with Stack 1 (early layers) showing the most evolution and middle Stack 3 being most stable.
python
temperature = 0.5top_p = 0.9repetition_penalty = 1.2max_new_tokens = 128Strengths:
Limitations:
Context Confusion Example:
Recommended Use Cases:
Not Recommended:
python
from transformers import AutoModelForCausalLM, AutoTokenizer # Load model and tokenizermodel = AutoModelForCausalLM.from_pretrained( "alea-institute/kl3m-007-500m-checkpoint-15500", torch_dtype="auto", device_map="auto")tokenizer = AutoTokenizer.from_pretrained("alea-institute/kl3m-007-500m-checkpoint-15500") # Generate with recommended parametersinputs = tokenizer( "<|start|>This Agreement is entered into as of", return_tensors="pt").to(model.device) outputs = model.generate( **inputs, max_new_tokens=128, temperature=0.5, top_p=0.9, repetition_penalty=1.2, do_sample=True) print(tokenizer.decode(outputs[0], skip_special_tokens=False))python
from transformers import pipeline generator = pipeline( "text-generation", model="alea-institute/kl3m-007-500m-checkpoint-15500", torch_dtype="auto", device_map="auto") outputs = generator( "<|start|>WHEREAS, the parties desire to enter into", max_new_tokens=128, temperature=0.5, top_p=0.9, repetition_penalty=1.2) print(outputs[0]['generated_text'])| Model | Layers | Params | Steps | Tokens | Loss | Use Case |
|---|---|---|---|---|---|---|
| kl3m-006-170m-checkpoint-63000 | 30 | 181.7M | 63K | 15.83B | 3.18 | Production 170M |
| kl3m-007-500m-step0 | 120 | 500.3M | 0 | 0 | ~8.0 | Stacked init |
✓ Successful layer divergence (6.48% cross-stack variance) ✓ Stable gradient norms (mean 2.27, no clipping needed) ✓ 65% loss reduction from initialization ✓ Multi-domain legal text generation capability ✓ Proper spectral conditioning maintained
⚠️ Domain mixing issues (court vs contract language) ⚠️ Data distribution imbalance affecting generation style ↻ Continued layer specialization (stacks still differentiating) ↻ Loss descent toward production target
Target Performance (by step 50-100K):
G_stack Efficiency Gains (per NeurIPS 2024 paper):
Inherited from source checkpoint + Phase A regularization:
Attention Layers:
MLP Layers:
LM Head:
Training Stability: Zero gradient clipping events in recent window indicates healthy optimization landscape.
json
{ "stacking_method": "G_stack (G_direct depthwise)", "implementation": "cyclic_duplication", "paper": "Stacking Your Transformers (NeurIPS 2024, arXiv:2405.15319)", "authors": "Du et al.", "source_checkpoint": "checkpoints/muon_170m_phase2/step-00063000", "growth_factor": 4, "source_layers": 30, "target_layers": 120, "stacking_pattern": "[0-29] repeated 4 times", "training_steps_post_stack": 15500, "tokens_processed_post_stack": 2160107520, "layer_divergence_pct": 6.48, "expected_efficiency_gain": "54.6% fewer tokens (per paper)"}Alea Institute
If you use this model, please cite the G_stack paper:
bibtex
@inproceedings{gstack2024, title={Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training}, author={Du, Wenyu and Luo, Tongxu and Qiu, Zihan and Huang, Zeyu and Shen, Yikang and Cheng, Reynold and Guo, Yike and Fu, Jie}, booktitle={NeurIPS}, year={2024}, note={arXiv:2405.15319}}Apache 2.0
| kl3m-007-500m-checkpoint-15500 | 120 | 500.3M | 15.5K | 2.16B | 2.62 | Current |
| kl3m-007-500m-checkpoint-* (future) | 120 | 500.3M | 100K+ | 14B+ | <2.5 | Production 500M target |