GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Ace-2504
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Ace-2504
Model Tree
Base
this model
Input Modalities
Text
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
Container
This model exists to answer a research question: is the catastrophic forgetting seen in
one-shot continued pretraining reversible? The slm-125m-extended run had trained on
515M tokens containing no SEC filings, and SEC ability degraded +8.1% as a result.
This run trained on a corpus that is ~35% SEC. Every domain improved, and the earlier forgetting reversed — measured on the base model's own held-out validation set (a fair within-family comparison, identical tokenizer, neither model trained on it):
| Domain | slm-125m-extended → e2 (perplexity) | change |
|---|---|---|
| SEC filings | 7.41 → 5.74 | −12.8% (was +8.1% before) |
| US case law | 11.51 → 9.80 | −6.6% |
| Educational web | 27.65 → 23.52 | −4.9% |
Conclusion: continued-pretraining forgetting is a function of data composition, not an inevitable cost — reintroduce a domain and the model recovers it. SEC, the model's strongest register, is now better than in any previous checkpoint.
| Architecture | LlamaForCausalLM, 125.8M params |
| Layers / hidden / heads | 12 / 768 / 12 (head dim 64) |
| Context length | 1,024 |
| Vocabulary | 16,384 (byte-level BPE, trained on the domain corpus) |
| Positional / norm / MLP | RoPE (θ=10,000) · RMSNorm · SwiGLU · tied embeddings |
| Precision | bf16 |
| Initialised from | Ace-2504/slm-125m-extended (weights only; optimizer reset) |
| Corpus | 2.500B tokens — SEC 869M / US case law 855M / FineWeb-Edu 776M |
| Epochs / steps | 2 epochs · 9,440 steps · 4.95B tokens processed |
| Schedule | 100-step warmup → peak LR 3e-4 → cosine decay → 3e-5 floor |
| Batch | 524,288 tokens/step (micro-batch 32 × grad-accum 16) |
| Optimizer | AdamW (β 0.9/0.95, wd 0.1), grad-clip 1.0 |
| Hardware | 1× A100 (SXM4), ~12.5 h |
| Checkpoint | cumulative tokens seen | val perplexity |
|---|---|---|
slm-125m-base (from scratch, 1 epoch) | 2.04B | ~11.3 |
slm-125m-extended (+515M) | 2.55B | 11.72 |
slm-125m-e2 (this model, +4.95B / 2 epochs) | ~7.5B | 9.86 |
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torchtok = AutoTokenizer.from_pretrained("Ace-2504/slm-125m-e2")model = AutoModelForCausalLM.from_pretrained("Ace-2504/slm-125m-e2", torch_dtype=torch.float32).eval()prompt = "Pursuant to Section 10(b) of the Securities Exchange Act,"ids = tok(prompt, return_tensors="pt").input_idsout = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95, top_k=50)print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Ace-2504/slm-125m-base — the original from-scratch model.Ace-2504/slm-125m-extended — the 1-epoch +515M checkpoint this one continues from.