Model
Table | |
|---|
| Architecture | Qwen3MoeForCausalLM (Qwen3-30B-A3B config) |
| Layers | 48 |
| Experts | 128 routed experts, 8 active per token |
| Attention | GQA, 32 query heads / 4 KV heads, head_dim 128 |
| Parameters | 30.5B total / ~3.3B active |
| Tokenizer | Qwen3 (vocab 151,936) |
| Precision | bf16 |
Data
- FineWeb-Edu (
sample/100BT), 101 shards used for training (~76B-token pool with the Qwen3 tokenizer); one shard held out for validation and never trained on.
- 85B tokens processed in total (~43B unique). Due to restarts during training, early portions of the data were seen up to 4x; the rest was seen once.
Training
- Training framework: torchtitan v0.2.2 with minor local patches (node-local checkpointing, data-loader skip-ahead).
- Hardware: 8 nodes x 8 AMD MI325X (64 GPUs). Parallelism: FSDP-64 for the first ~59B tokens, then HSDP (8 replicas x 8 shards), expert parallel 8.
- Sequence length 4096; global batch 768 sequences = 3.1M tokens/step; 27,017 steps.
- AdamW, peak LR 3e-4, 2,000 warmup steps, WSD schedule: constant LR to 68B tokens, then linear decay to 0 at 85B tokens.
- Wall-clock: about 4 days including restarts and incident recovery.
Evaluation
Held-out validation loss (FineWeb-Edu held-out shard, 512 sequences x 1024 tokens):
Table with columns: step, tokens, val loss| step | tokens | val loss |
|---|
| 21,597 (decay start) | 68.0B | 2.449 |
| 27,017 (final) | 85.0B | 2.339 |
For reference, the official Qwen3-1.7B-Base scores 2.506 on the same held-out set.
Note: the held-out shard is from the training distribution (FineWeb-Edu), so this comparison favors our model; Qwen3-1.7B-Base was trained on a much broader mix. This is not a general quality comparison.
lm-evaluation-harness (0-shot, full test sets, HF backend):
Table with columns: task, metric, step 13,800 (~43B tokens), final (85B tokens)| task | metric | step 13,800 (~43B tokens) | final (85B tokens) |
|---|
| hellaswag | acc_norm | 0.571 | 0.634 |
| arc_easy | acc | 0.718 | 0.764 |
| piqa | acc | 0.719 | 0.749 |
| winogrande | acc | 0.564 | 0.585 |
Intended use & limitations
Research artifact only. Base (pretrained-only) model: not instruction-tuned, not safety-aligned, not suitable for chat or deployment. May produce factually wrong, biased, or harmful text. English-centric (FineWeb-Edu); weak on code, math, and multilingual.
Usage
Plain text completion (this is a base model; do not use a chat template):
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "ziliangpeng/Qwen3-30B-A3B-FineWeb-Edu-85B"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")inputs = tok("The water cycle is", return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=64, eos_token_id=151643)print(tok.decode(out[0], skip_special_tokens=True))
License
Model weights: Apache-2.0 (same as the Qwen3 architecture/tokenizer). Training data: FineWeb-Edu, released under ODC-By 1.0; please follow its terms.