bayliner1980

Qwen2.5-0.5B-MoE-A0.3B

README

License: apache-2.0

Architecture

Table

Layers	24: layers 0-22 are MoE, layer 23 is a dense MLP (`mlp_only_layers=[23]`)
Routed experts	32 per MoE layer, top-8, `norm_topk_prob=True`
Expert size	128
Shared expert	768, sigmoid-gated
Active MLP compute	37% of the dense model's in each MoE layer, 39.5% averaged over all layers
Active parameters	~168M of 358M non-embedding parameters per token
Total parameters	494.7M (bf16, ~990 MB)
Context length, vocabulary	Same as Qwen2.5-0.5B

Evaluation

Perplexity uses 1024-token windows. Benchmarks are zero-shot and follow lm-evaluation-harness conventions (HellaSwag on 3,000 validation items; full sets for the others).

Table with columns: WikiText-2 test ppl, C4 val ppl, HellaSwag (norm), ARC-Easy (acc), ARC-Challenge (norm), WinoGrande, OpenBookQA (norm), Mean acc_norm
	WikiText-2 test ppl	C4 val ppl	HellaSwag (norm)	ARC-Easy (acc)	ARC-Challenge (norm)	WinoGrande	OpenBookQA (norm)	Mean acc_norm
Qwen2.5-0.5B (dense)	14.66	21.29	0.497	0.646	0.321	0.569	0.354	0.467
This model

Mean acc_norm averages acc_norm over all five tasks (ARC-Easy acc_norm: 0.508).

Changes from the previous version

The previous upload scored better on WikiText-2 (17.99) but worse on general web text (C4 ~35) and benchmarks (mean acc_norm 0.393). This version is more general-purpose. It uses slightly more compute per token than before (39.5% vs 37% of the MLP) because its last layer is dense.

Speed

The sparsity only saves compute with an inference engine that runs just the selected experts, such as transformers' MoE kernels, vLLM or llama.cpp. At this model size attention and the embedding/LM head make up a large share of the cost, so expect smaller end-to-end speedups than the MLP figures suggest.

Limitations

Behind the dense model on reasoning and knowledge. It keeps most of the dense model's language-modelling quality, but loses about 0.06 mean benchmark accuracy. Expect factual errors and topic drift in long generations.
Code and non-English text not evaluated. Quality there is likely weaker than on English prose.
Base model, not instruction-tuned. It continues text; it does not follow chat instructions.
Research artefact. It is an experiment in sparse conversion, not a model for production use.

Support

This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model Qwen/Qwen2.5-0.5B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

bayliner1980

Model Tree

Base

Qwen/Qwen2.5-0.5B

Fine-tuned

this model

Input Modalities

Text

Output Modalities