GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sepsy070716
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sepsy070716
Model Tree
sepsy070716/Qwen3.5-4B-A3B-Upcycled
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Total parameters | 4,036,686,336 |
| Active parameters | 2,998,596,096 |
| Experts / selected | 8 / top-2 |
| Shared / routed width | 1,536 / 704 |
| Distilled layers | 32 / 32 |
Training used a balanced ODC-By-1.0 FineWeb/FineWeb2 sample in Korean, English, Mandarin Chinese, Japanese, Spanish, and German. Every layer received 24 local updates (four documents per language). Evaluation used a disjoint split with two documents per language and layer.
Language means: de 6.83%, en 5.12%, es 7.15%, ja 7.47%, ko 7.42%, zh 5.60%.
On four disjoint 128-token documents per language, mean causal-LM loss improved
from 10.378 for the untrained upcycle to 9.162 for this checkpoint, an
11.72% relative reduction. All six languages improved. The dense teacher scored
about 2.94, which quantifies the large remaining recovery gap.
A two-prompt greedy generation smoke test loaded and executed successfully, but the next four tokens were whitespace-dominated in both Korean and English. Therefore this checkpoint is not yet suitable for chat, completion, benchmark, or production claims. Joint end-to-end recovery is required.
python
from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGenerationmodel_id = "sepsy070716/Qwen3.5-4B-A3B-Multilingual-Distilled-v1"model = Qwen3_5MoeForConditionalGeneration.from_pretrained(model_id, device_map="auto")tokenizer = AutoTokenizer.from_pretrained(model_id)
See distillation_manifest.json and distillation_summary.json for complete
per-layer and per-language measurements.