GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
rohithck
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
rohithck
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Delphi is the Marin team's first open scaling suite, inspired by Pythia. It has three parts:
A pre-registered forecast from that scaling law predicted the final loss of the largest Delphi run (1 × 10²³ FLOPs, 25 B parameters, 600 B tokens) within 0.2%, using 300× less compute than the training run itself. The same process forecasts downstream benchmarks — MMLU, HumanEval, and GSM8K — via a two-step regression combining compute and observational scaling laws.
See "Scaling Laws That Extrapolate 300× Past the Fit"
for the recipe, fit, and downstream-eval projections. The full set of Delphi
checkpoints — IsoFLOP grid points, held-out optima at 1e21/1e22/1e23 with
multiple random seeds, and training intermediates — lives on
marin-community on the Hub.
Research artifact. This model has not been properly tested or evaluated and is not necessarily secure. Use at your own risk in production settings.
| Architecture | Qwen 3 (pre-norm decoder, RMSNorm, RoPE, QK-norm with learned scaling, SwiGLU MLPs) |
| Parameters | 2,895,178,112 |
| Hidden size | 2432 |
| Layers | 24 |
| Attention heads | 19 |
| KV heads | 19 (no GQA) |
| Head dim | 128 |
| FFN intermediate | 9728 (MLP ratio 4) |
| Vocab size | 128,256 (Llama 3 tokenizer) |
| Max sequence length | 4096 |
| Position encoding | RoPE (θ = 500000, Llama 3-style scaling) |
| Bias terms | None |
| Tied embeddings | No |
| Compute | 5.5e20 FLOPs |
| Tokens | 29,916,397,568 |
| Steps | 57,061 |
| Training seed | 0 |
| Final export step (zero-based) | 57,060 |
| Sequence length | 4096 |
| Optimizer | AdamH (Adam with Hyperball) |
| Recipe | Delphi (Complete(d)P-style scaling with (T₀/T)^0.3 token-horizon LR correction) |
| LR schedule | WSD: 10% linear warmup, 20% linear decay, 0 floor |
| Precision | f32 master params, bf16 compute |
| Parallelism | FSDP |
| Data mixture | Nemotron-CC + StarCoderData + ProofPile 2 |
| Tokenizer | Llama 3 (vocab 128,256) |
AdamH, Adam
with Hyperball, constrains every projection weight to stay on the Frobenius-
norm sphere it was initialized on, so weight decay has nothing to regularize
away and falls out of the recipe. A Complete(d)P-style transfer rule with a
(T₀/T)^0.3 correction sets learning rate as token horizon grows. Reference
constants: B₀ = 64, T₀ = 2.5 B tokens, η₀ = 0.00630, η₀,Adam = 0.000656,
ε₀ = 1.85 × 10⁻⁸. Recipe code:
experiments/references/completed_adamh.py.
The recorded training configuration is in
experiments/references/train_delphi_seeds.py.
See the reproduction instructions
and the training run.
The linked code revision publishes the recorded recipe; it does not identify
the checkout used for the completed training run. The inference export has
architectures: ["Qwen3ForCausalLM"]; its weights are unchanged from training.
marin-community on the Hub.marin-community/delphi-blog-data (one config per figure, with wandb_url on every row).Evaluation results for this checkpoint are not included in this release.
marin-community.bibtex
@misc{held2026delphi, title = {Scaling Laws That Extrapolate 300× Past the Fit}, author = {Held, Will and {Marin Community}}, year = {2026}, url = {https://openathena.ai/blog/delphi}}@misc{held2026delphi, title = {Scaling Laws That Extrapolate 300× Past the Fit}, author = {Held, Will and {Marin Community}}, year = {2026}, url = {https://openathena.ai/blog/delphi}}