Model Description
Marco-Mini-Global shares the same architecture as Marco-Mini-Base: a decoder-only Transformer with sparse MoE layers replacing standard FFN layers, upcycled from Qwen3-0.6B-Base using fine-grained sub-matrix splitting combined with Drop-Upcycling.
Table with columns: Configuration, Value| Configuration | Value |
|---|
| Total Parameters | 17.3B |
| Activated Parameters | 0.86B |
| Activation Ratio | 5% |
| Num Layers | 28 |
| Model Dimension | 1024 |
| FFN Intermediate Dimension | 3072 |
| Q-Heads | 16 |
| KV-Heads | 8 |
| Head Dimension | 128 |
| Expert Dimension | 768 |
| Total Experts | 256 |
| Activated Experts | 8 |
| Tie Embeddings | True |
| Training FLOPs | 1.584×1023 |
Training Details
Marco-Mini-Global-Base branches from the Stage-2 checkpoint of Marco-Mini-Base and recalibrates the data mixtures in Stages 3 and 4 to integrate pre-training corpora for 35 newly introduced languages. In total it was trained on 5.5T tokens.
The four-stage curriculum follows the same structure as Marco-Mini-Base:
- Stage 1 (0 - 2.4T tokens): Foundational Training — High-quality English data (Nemotron-CC-v2), reasoning and instruction data, and multilingual web/QA data for 19 languages.
- Stage 2 (2.4T - 4.1T tokens): Optimization & Upsampling — Upsampled reasoning corpora, downsampled English web data, and upsampled Chinese data with learning rate decay.
- Stage 3 (4.1T - 5T tokens): Language Expansion — Recalibrated data mixtures to integrate 35 new languages alongside the original 29.
- Stage 4 (5T - 5.5T tokens): Synthetic Data Integration — Curated multilingual synthetic data including cultural content and synthetic regional MCQs for all 64 languages.
Supported Languages
Original 29 languages: English, Chinese, Arabic, German, Spanish, French, Korean, Japanese, Portuguese, Turkish, Indonesian, Italian, Dutch, Polish, Russian, Vietnamese, Thai, Hebrew, Ukrainian, Malay, Bengali, Czech, Urdu, Kazakh, Greek, Romanian, Hungarian, Nepali, Azerbaijani
35 newly introduced languages: Danish, Swedish, Norwegian, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Serbian, Bulgarian, Persian, Maltese, Hindi, Marathi, Gujarati, Punjabi, Tamil, Telugu, Tagalog, Javanese, Khmer, Lao, Burmese, Amharic, Swahili, Yoruba, Igbo, Zulu
Evaluation
We compare Marco-Mini-Global-Base against strong multilingual baselines: Gemma3-4B (4B activated), Tiny-Aya-3.35B (3.35B activated), and Qwen3-4B (4B activated). All benchmarks are evaluated across the full 64-language set. Marco-Mini-Global uses only 0.86B activated parameters while preserving robust English proficiency (63.6 vs. 63.7 for the 29-language Marco-Mini) and increasing the multilingual advantage over Qwen3-4B from +2.6% to +3.6%.
English
Table with columns: Benchmark, # Shots, Gemma3-4B, Tiny-Aya-3.35B, Qwen3-4B, Marco-Mini-Global| Benchmark | # Shots | Gemma3-4B | Tiny-Aya-3.35B | Qwen3-4B | Marco-Mini-Global |
|---|
| MMLU (Acc) | 5-shot | 61.1 | 58.6 | 75.2 | 72.9 |
| MMLU-Redux (Acc) | 0-shot | 57.7 | 51.7 | 71.3 | 68.9 |
| MMLU-Pro |
Multilingual — General
Table with columns: Benchmark, # Shots, Gemma3-4B, Tiny-Aya-3.35B, Qwen3-4B, Marco-Mini-Global| Benchmark | # Shots | Gemma3-4B | Tiny-Aya-3.35B | Qwen3-4B | Marco-Mini-Global |
|---|
| GlobalMMLU (Acc) | 5-shot | 49.1 | 48.4 | 57.8 | 60.9 |
| MMMLU (Acc) | 0-shot | 45.0 | 42.8 | 54.8 | 58.2 |
| MMLU-ProX-Lite |
Multilingual — Cultural & Regional
Table with columns: Benchmark, # Shots, Gemma3-4B, Tiny-Aya-3.35B, Qwen3-4B, Marco-Mini-Global| Benchmark | # Shots | Gemma3-4B | Tiny-Aya-3.35B | Qwen3-4B | Marco-Mini-Global |
|---|
| INCLUDE (Acc) | 5-shot | 52.3 | 53.5 | 60.0 | 61.1 |
| Global-PIQA (Acc_norm) | 0-shot | 67.8 | 66.7 | 61.8 | 70.2 |
| CMMLU |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "AIDC-AI/Marco-Mini-Global-Base"tokenizer = AutoTokenizer.from_pretrained(model_name)model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto") input_text = "The capital of France is"inputs = tokenizer(input_text, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=50)print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation
@article{marco-moe, title={Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling}, author={Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi, Yichao Du, Feihu Jiang, Longyue Wang and Weihua Luo}, year={2026}}
License
This model is released under the Apache 2.0 License.