apiary-7B-A1B API & Inference Endpoint

Training

Data: first 20 parquet files of HuggingFaceFW/fineweb-edu sample/100BT, tokenized with the SmolLM2 tokenizer (EOS-separated, no document masking), consumed sequentially — see data.py in the code repo.
Optimizer: AdamW lr 3e-4 (betas 0.9/0.95, wd 0.1), 1.05M-token batches, 500 warmup steps, constant then linear decay over the last 20% of the wall-clock budget; grad clip 1.0; load-balance loss 0.01, router z-loss 1e-3.
Systems: PyTorch 2.13, FSDP2 + torch.compile + grouped_mm experts, bf16 with fp32 master weights; median 613k tokens/s (28.2% MFU) on 8xB200; 9382 optimizer steps.
Final train CE 2.497; held-out FineWeb-Edu CE 2.462 (ppl 11.7) on 262k unseen tokens.

loss curve

Evaluation (lm-eval-harness, 0-shot)

Table with columns: HellaSwag, ARC-e, ARC-c, PIQA, Winogrande, MMLU, wikitext word ppl
HellaSwag	ARC-e	ARC-c	PIQA	Winogrande	MMLU	wikitext word ppl
48.8	59.6	33.4	70.9	52.5	25.5	19.2

(acc_norm where the task defines it.) Logits parity vs the training code: argmax agreement 90.6%, mean |diff| 0.077.

Provenance

Weights sha256 (first 16 hex): {"model-00001-of-00004.safetensors": "f9266617b050d0fa", "model-00002-of-00004.safetensors": "cf862b24644ea799", "model-00003-of-00004.safetensors": "b09df28b263a22f6", "model-00004-of-00004.safetensors": "7ea4c3d916870908"}. Full per-10-step training log: metrics.jsonl.

Training

Data: first 20 parquet files of HuggingFaceFW/fineweb-edu sample/100BT, tokenized with the SmolLM2 tokenizer (EOS-separated, no document masking), consumed sequentially — see data.py in the code repo.
Optimizer: AdamW lr 3e-4 (betas 0.9/0.95, wd 0.1), 1.05M-token batches, 500 warmup steps, constant then linear decay over the last 20% of the wall-clock budget; grad clip 1.0; load-balance loss 0.01, router z-loss 1e-3.
Systems: PyTorch 2.13, FSDP2 + torch.compile + grouped_mm experts, bf16 with fp32 master weights; median 613k tokens/s (28.2% MFU) on 8xB200; 9382 optimizer steps.
Final train CE 2.497; held-out FineWeb-Edu CE 2.462 (ppl 11.7) on 262k unseen tokens.

loss curve

Evaluation (lm-eval-harness, 0-shot)

Table with columns: HellaSwag, ARC-e, ARC-c, PIQA, Winogrande, MMLU, wikitext word ppl
HellaSwag	ARC-e	ARC-c	PIQA	Winogrande	MMLU	wikitext word ppl
48.8	59.6	33.4	70.9	52.5	25.5	19.2

(acc_norm where the task defines it.) Logits parity vs the training code: argmax agreement 90.6%, mean |diff| 0.077.

apiary-7B-A1B

README

Training

Evaluation (lm-eval-harness, 0-shot)

Provenance

Explore FriendliAI today

README

Training

Evaluation (lm-eval-harness, 0-shot)

Provenance