bayliner1980

SmolLM2-360M-MoE-A0.2B

README

License: apache-2.0

Architecture

Table

Layers	32: layers 3 and 31 are dense MLPs (`mlp_only_layers=[3, 31]`), the other 30 are MoE
Routed experts	32 per MoE layer, top-8, `norm_topk_prob=True`
Expert size	64
Shared expert	512, sigmoid-gated
Active MLP compute	40% of the dense model's in each MoE layer, 44% averaged over all layers
Active parameters	~183M of 315M non-embedding parameters per token
Total parameters	362.8M (bf16, ~726 MB)
Context length, vocabulary	Same as SmolLM2-360M (8,192 tokens, 49,152 vocabulary)

Evaluation

Perplexity uses 1024-token windows. Benchmarks are zero-shot and follow lm-evaluation-harness conventions (HellaSwag on 3,000 validation items; full sets for the others).

Table with columns: WikiText-2 test ppl, C4 val ppl, HellaSwag (norm), ARC-Easy (acc), ARC-Challenge (norm), WinoGrande, OpenBookQA (norm), Mean acc_norm
	WikiText-2 test ppl	C4 val ppl	HellaSwag (norm)	ARC-Easy (acc)	ARC-Challenge (norm)	WinoGrande	OpenBookQA (norm)	Mean acc_norm
SmolLM2-360M (dense)	12.94	18.09	0.525	0.702	0.386	0.590	0.374	0.514
This model

Mean acc_norm averages acc_norm over all five tasks (ARC-Easy acc_norm: dense 0.693, this model 0.527).

Speed

The sparsity only saves compute with an inference engine that runs just the selected experts, such as transformers' MoE kernels, vLLM or llama.cpp. At this model size attention and the embedding/LM head make up a large share of the cost, so expect smaller end-to-end speedups than the MLP figures suggest.

Limitations

Behind the dense model on reasoning and knowledge. It keeps much of the dense model's language-modelling quality, but loses about 0.10 mean benchmark accuracy, most on ARC (science questions). Expect factual errors and topic drift in long generations.
Code, math and non-English text not evaluated. Quality there is likely weaker than on English prose.
Base model, not instruction-tuned. It continues text; it does not follow chat instructions.
Research artefact. It is an experiment in sparse conversion, not a model for production use.

Support

This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model HuggingFaceTB/SmolLM2-360M.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

bayliner1980

Model Tree

Base

HuggingFaceTB/SmolLM2-360M

Fine-tuned

this model

Input Modalities

Text

Output Modalities