Model Description
Marco-Nano is built on a decoder-only Transformer architecture with sparse MoE layers replacing standard FFN layers. It is upcycled from Qwen3-0.6B-Base using a fine-grained sub-matrix splitting strategy combined with Drop-Upcycling to promote expert diversification.
Table with columns: Configuration, Value| Configuration | Value |
|---|
| Total Parameters | 8B |
| Activated Parameters | 0.6B |
| Activation Ratio | 7.5% |
| Num Layers | 28 |
| Model Dimension | 1024 |
| FFN Intermediate Dimension | 3072 |
| Q-Heads | 16 |
| KV-Heads | 8 |
| Head Dimension | 128 |
| Expert Dimension | 384 |
| Total Experts | 232 |
| Activated Experts | 8 |
| Tie Embeddings | True |
| Training FLOPs | 1.40×1023 |
Training Details
Marco-Nano was pre-trained on 5.1 trillion tokens using a four-stage curriculum:
- Stage 1 (0 - 2.4T tokens): Foundational Training — High-quality English data (Nemotron-CC-v2), reasoning and instruction data, and multilingual web/QA data for 19 languages.
- Stage 2 (2.4T - 4.1T tokens): Optimization & Upsampling — Upsampled reasoning corpora, downsampled English web data, and upsampled Chinese data with learning rate decay.
- Stage 3 (4.1T - 4.6T tokens): Language Expansion — Added 9 new languages (Bengali, Czech, Urdu, Kazakh, Greek, Romanian, Hungarian, Nepali, Azerbaijani) and upsampled medium-resource languages.
- Stage 4 (4.6T - 5.1T tokens): Synthetic Data Integration — Curated multilingual synthetic data including cultural content (Fineweb2-Culture) and synthetic regional MCQs.
Supported Languages
English, Chinese, Arabic, German, Spanish, French, Korean, Japanese, Portuguese, Turkish, Indonesian, Italian, Dutch, Polish, Russian, Vietnamese, Thai, Hebrew, Ukrainian, Malay, Bengali, Czech, Urdu, Kazakh, Greek, Romanian, Hungarian, Nepali, Azerbaijani
Evaluation
We compare Marco-Nano against size-matched baselines: Qwen3-1.7B (1.7B activated), Trinity Nano (1.09B activated), and Granite4-Tiny (1.47B activated). Marco-Nano uses only 0.6B activated parameters — the smallest among all baselines.
English
Table with columns: Benchmark, # Shots, Qwen3-1.7B, Trinity Nano, Granite4-Tiny, Marco-Nano| Benchmark | # Shots | Qwen3-1.7B | Trinity Nano | Granite4-Tiny | Marco-Nano |
|---|
| MMLU (Acc) | 5-shot | 65.1 | 64.7 | 69.1 | 64.7 |
| MMLU-Redux (Acc) | 0-shot | 61.2 | 60.1 | 65.8 | 62.9 |
| MMLU-Pro (Acc) |
Multilingual — General
Table with columns: Benchmark, # Shots, Qwen3-1.7B, Trinity Nano, Granite4-Tiny, Marco-Nano| Benchmark | # Shots | Qwen3-1.7B | Trinity Nano | Granite4-Tiny | Marco-Nano |
|---|
| GlobalMMLU (Acc) | 5-shot | 49.6 | 43.6 | 54.8 | 52.2 |
| MMMLU (Acc) | 0-shot | 48.6 | 41.2 | 52.3 | 52.6 |
| MMLU-ProX-Lite |
Multilingual — Cultural & Regional
Table with columns: Benchmark, # Shots, Qwen3-1.7B, Trinity Nano, Granite4-Tiny, Marco-Nano| Benchmark | # Shots | Qwen3-1.7B | Trinity Nano | Granite4-Tiny | Marco-Nano |
|---|
| INCLUDE (Acc) | 5-shot | 51.2 | 43.9 | 52.1 | 53.2 |
| Global-PIQA (Acc_norm) | 0-shot | 60.3 | 52.3 | 64.0 | 64.3 |
| CMMLU |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "AIDC-AI/Marco-Nano-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
input_text = "The capital of France is"
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation
@article{marco-moe,
title={Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling},
author={Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi, Yichao Du, Feihu Jiang, Longyue Wang and Weihua Luo},
year={2026}
}
License
This model is released under the Apache 2.0 License.