GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
thu-pacman
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
thu-pacman
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Puro-2B is intended to make billion-parameter pretraining inspectable and affordable for smaller research groups. The release covers more than the final weights:
The main recipe combines RTX 5090 infrastructure, blockwise FP8, MuonH with hyperball constraints, proxy-guided data selection, and a curriculum-aware late continuation followed by checkpoint averaging.
The collection contains multiple checkpoints with different Phase 2 budgets and recipes. The report's approximately $4.4K result is an observed uniform-recipe checkpoint that already exceeds Qwen2-1.5B on the report's 15-task aggregate. It is not the canonical final checkpoint.
The canonical Puro-2B-Base model is the strongest released endpoint. Its
production run used 22,514 measured active-training GPU-hours, corresponding
to **6,891∗∗underthereport′snormalizedRTX5090rentalrate,andremainedw7,478 accelerator budget.
These figures are accelerator-only reproduction estimates. They exclude data acquisition and preprocessing, proxy and ablation experiments, failed runs, post-training, evaluation, storage, networking, and research labor. They should not be read as the total cost of developing the project.
The scaling-law panel labels points by cumulative reproduction cost. The model catalog below maps those costs to Phase 2 budget fractions. Each fraction applies only to Phase 2 data exposure, while the cost includes the shared Phase 1 run.
| Property | Value |
|---|---|
| Model type | Dense decoder-only causal language model |
| Parameters | Approximately 2B |
| Initialization | From scratch |
| Architecture | Qwen3-1.7B configuration with untied embeddings |
| Hidden size | 2,048 |
| Transformer layers | 28 |
| Attention heads / KV heads | 16 / 8 |
| Feed-forward size | 6,144 |
| Vocabulary size |
This is a pretrained base model. It has not been instruction-tuned or preference-aligned and should not be expected to behave like a chat assistant.
Use a Transformers release that supports the Qwen3 configuration:
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel_id = "thu-pacman/Puro-2B-Base"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id,torch_dtype="auto",device_map="auto",)prompt = "The central limit theorem states that"inputs = tokenizer(prompt, return_tensors="pt").to(model.device)outputs = model.generate(**inputs,max_new_tokens=128,do_sample=True,temperature=0.7,top_p=0.9,)print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For reproducible evaluation, pin the model revision, Transformers version, prompts, decoding settings, dataset snapshots, and answer postprocessors.
All numbers below come from the same deterministic OpenCompass pipeline in the technical report. The comparison uses pretrained/base checkpoints throughout. Generation tasks use greedy decoding; multiple-choice tasks use fixed token-likelihood ranking. Scores are percentages.
| Model | Math + Code (4) | Reasoning + Knowledge (11) | Overall (15) |
|---|---|---|---|
| Qwen2-1.5B | 40.29 | 60.54 | 55.14 |
| Puro-2B | 43.50 | 63.02 | 57.81 |
| Qwen2.5-1.5B | 47.52 | 65.53 | 60.73 |
The four math and code tasks are GSM8K, MATH, sanitized-MBPP, and HumanEval. The eleven reasoning and knowledge tasks are MMLU, MMLU-Pro, ARC-Challenge, ARC-Easy, BoolQ, CommonsenseQA, HellaSwag, PIQA, SocialIQA, WinoGrande, and BBH. Each displayed average is an unweighted arithmetic mean.
The Puro Cost Scaling Law fits five single-run Phase 2 uniform-budget points. It is a recipe-specific empirical scale-down relationship, not a universal law. The fit has no uncertainty interval, and the available experiments do not isolate curriculum ordering, constant-LR continuation, and checkpoint averaging as independent causal gains.
| Setting | Phase 1 | Phase 2 |
|---|---|---|
| Tokens consumed | 439B | 961B |
| RTX 5090 GPUs | 24 | 96 |
| Parallelism (TP / PP / DP) | 1 / 2 / 12 | 1 / 4 / 24 |
| Base learning rate | 5.00e-3 -> 1.04e-3 | 1.04e-3 -> 1.00e-5 |
| Schedule | Power decay | Linear decay, then selected constant-LR continuation |
Both phases use a sequence length of 4,096, a global batch size of 1,536 sequences, and a micro-batch size of 2. Main Transformer linear-layer GEMMs use blockwise E4M3 FP8; numerically sensitive operations, master weights, and optimizer states remain in BF16 or FP32 as appropriate.
Selected approximately scale-invariant matrix weights are updated by MuonH with hyperball projection and zero weight decay. The remaining parameters use AdamW with weight decay 0.1. The MuonH matrix group applies a 10x multiplier to the shared base learning-rate schedule.
The final model uses an equal-weight parameter average of six checkpoints from the constant-LR branch resumed at optimizer step 218,000:
text
222100, 222200, 222300, 222400, 222500, 222569
Only model parameters are averaged; optimizer moments are not.
| Repository | Role |
|---|---|
| Puro-2B-Base | Canonical final model; equal-weight average of six late Phase 2 checkpoints. |
| Puro-2B-Base-Phase1 | Phase 1 endpoint, before the Phase 2 distribution. |
| Puro-2B-Curriculum-DecayFinal | Curriculum Phase 2 endpoint before the selected constant-LR continuation and averaging. |
| Puro-2B-Curriculum-SMA6-Inputs | Six complete inputs to the final equal-weight average; an artifact set, not another averaged model. |
All released model artifacts in the collection use the Apache License 2.0.
The budget fractions above refer to Phase 2 data exposure. The dollar values follow Figure 2(b) of the report and include the fixed Phase 1 training cost; they are cumulative costs to reproduce each checkpoint, not incremental Phase 2 costs.
Puro-2B is intended for research on pretraining, data recipes, optimization, model scaling, continued pretraining, and downstream adaptation. It can also be used as a compact base model for task-specific post-training.
The model may produce inaccurate, biased, unsafe, offensive, or copyrighted content. Its pretraining data includes web text, code, mathematics, Chinese and English material, synthetic data, and instruction-formatted examples. The release does not claim exhaustive removal of personal information, benchmark contamination, or undesirable content. Evaluate and post-train the model for your domain before deployment, and add application-specific safeguards where people could be affected by its outputs.
The Puro-2B model weights are released under the Apache License 2.0. The training data remains subject to the dataset repository's documented upstream licenses and terms.
Please cite our technical report if you find our work useful:
bibtex
@misc{luo2026puro2b,title={Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090},author={Kairong Luo and Jiarui Cui and Yaorui Yin and Shengqi Chen and Yiming Yang and Linxiang Gao and Yanmohan Wang and Mingzhe Zhang and Kaiyue Wen and Kaifeng Lyu and Wenguang Chen},year={2026},eprint={2608.27370},archivePrefix={arXiv},primaryClass={cs.CL},url={https://arxiv.org/abs/2608.27370},}
We thank Yanfu Investments for providing computational resources. See the technical report for the complete acknowledgments and contributor list.
| 151,936 |
| Context length | 4,096 tokens |
| Export class | Qwen3ForCausalLM |
| Weight format | Safetensors |
| Median TFLOP/s/GPU | 238 | 192 |
| Uniform-ordering control at the full Phase 2 budget; approximately $6.9K cumulative reproduction cost. |
| Puro-2B-Uniform-Phase2-1of2 | Uniform Phase 2 run at 1/2 budget; approximately $4.4K cumulative reproduction cost. |
| Puro-2B-Uniform-Phase2-1of4 | Uniform Phase 2 run at 1/4 budget; approximately $3.1K cumulative reproduction cost. |
| Puro-2B-Uniform-Phase2-1of8 | Uniform Phase 2 run at 1/8 budget; approximately $2.5K cumulative reproduction cost. |
| Puro-2B-Uniform-Phase2-1of16 | Uniform Phase 2 run at 1/16 budget; approximately $2.2K cumulative reproduction cost. |