GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sepsy070716
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sepsy070716
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Total parameters | 4,036,686,336 |
| Active parameters including vision | 2,998,596,096 |
| Experts per layer | 8 |
| Experts selected per token | 2 |
| Shared expert width | 1536 |
| Routed expert width | 704 |
The original 9,216-wide dense FFN is reduced to a 1,536-wide shared expert and eight 704-wide routed experts with top-2 routing. Neurons are ranked per layer by the product of their gate, up, and down projection norms. The strongest shared and routed slices are retained. Routed experts start identically so the untrained router does not make the initial function nondeterministic.
This checkpoint is intended as an initialization for router warm-up, knowledge
distillation from Qwen/Qwen3.5-4B, and continued pretraining. It is not
recommended for production or user-facing inference before recovery training
and evaluation.
Load with a Transformers release that provides Qwen3_5MoeForConditionalGeneration:
python
from transformers import Qwen3_5MoeForConditionalGeneration, AutoProcessormodel = Qwen3_5MoeForConditionalGeneration.from_pretrained("sepsy070716/Qwen3.5-4B-A3B-Upcycled",device_map="auto",)processor = AutoProcessor.from_pretrained("sepsy070716/Qwen3.5-4B-A3B-Upcycled")
See conversion_manifest.json and neuron_selection.json for reproducibility.
A router-only MPS warm-up artifact is published under
research/router-warmup-v1/. It updates 655,360 router parameters and leaves
all attention, expert, embedding, and vision weights untouched.
On 32 held-out FineWeb2 Korean documents (8,109 valid tokens), normalized
routing entropy improved from 0.99599 to 0.99748, while the maximum/minimum
expert usage ratio improved from 1.528 to 1.390. This adapter balances
routing but does not recover the quality lost by dense-FFN compression;
expert distillation is still required.
Accepted layer adapters for depths 0, 16, and 31 are published under
research/layer-distillation-pilots/. Each was trained for 10 local updates
with the v1 router frozen. On eight held-out Korean documents, dense-FFN
relative MSE improved by 4.07%, 8.24%, and 8.33% respectively, with improvement
on all 24 document/layer comparisons.
These local reconstruction results are proof of direction, not an end-to-end model benchmark. The root checkpoint has not been modified by the pilot adapters.