GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
bayliner1980
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
bayliner1980
Model Tree
Qwen/Qwen2.5-1.5B
Input Modalities
Output Modalities
Supported Functionality
python
from transformers import AutoModelForCausalLM, AutoTokenizer path = "bayliner1980/Qwen2.5-1.5B-MoE-A0.9B"tok = AutoTokenizer.from_pretrained(path)model = AutoModelForCausalLM.from_pretrained(path, dtype="bfloat16", device_map="auto") ids = tok("The history of the printing press", return_tensors="pt").to(model.device)out = model.generate(**ids, max_new_tokens=80, do_sample=False)print(tok.decode(out[0], skip_special_tokens=True))It loads as a standard Qwen2MoeForCausalLM; no custom code or trust_remote_code is needed.
| Qwen2.5-1.5B (dense) | This model | |
|---|---|---|
| Layers / hidden size / heads | 28 / 1536 / 12 (2 KV) | same |
| MLP per layer | 8960 neurons, all active | shared expert 1792 + 32 experts × 224, top-8 |
| Active MLP neurons per token | 8960 | 3584 (40%) |
| Active parameters per token* | 1.54B | 0.85B |
| Total parameters | 1.54B | 1.78B |
| Output head | tied to embeddings | untied (llama.cpp's qwen2moe requires a separate output tensor) |
* Active parameters exclude the embedding lookup and include the output head. The total grows only because the output head is untied; the experts together hold the same number of neurons as the original MLP.
The routing uses norm_topk_prob: false, the only mode llama.cpp's qwen2moe supports. The expert width (224) is
a multiple of 32, so every expert tensor quantizes with llama.cpp's k-quants.
Perplexity: lower is better. Tasks: zero-shot acc_norm, higher is better, on the first 1000 items of each benchmark
(500 for OpenBookQA), with the dense model scored the same way.
| Dense Qwen2.5-1.5B | This model | |
|---|---|---|
| WikiText-2 test perplexity (1024-token windows) | 10.41 | 14.63 (+41%) |
| C4 validation perplexity (64 × 1024 tokens) | 15.76 | 22.88 (+45%) |
| Zero-shot tasks, mean acc_norm | 0.565 | 0.462 |
| Task | Dense | This model |
|---|---|---|
| HellaSwag | 0.592 | 0.510 |
| ARC-Easy | 0.723 | 0.572 |
| ARC-Challenge | 0.452 | 0.311 |
| WinoGrande | 0.646 | 0.588 |
| OpenBookQA | 0.410 | 0.328 |
llama.cpp llama-bench, Q4_K_M for both models, tokens per second (RTX 4060 8 GB; Ryzen 5 7500F for CPU):
| Dense Qwen2.5-1.5B | This model | Change | |
|---|---|---|---|
| GPU, text generation | 191 | 253 | +32% |
| GPU, prompt processing (512 tokens) | 10,329 | 11,353 | +10% |
| CPU, text generation | 54.9 | 77.5 | +41% |
| CPU, prompt processing (512 tokens) | 3,615 | 3,646 | ≈ same |
The GGUF files are larger than the dense model's (Q4_K_M: 1125 vs 935 MiB) because of the untied output head.
The dense model's MLP neurons were grouped into a shared expert and routed experts, a router was fitted for each
layer. Everything ran on a single 8 GB consumer GPU. No layer was kept dense, so the model fits
llama.cpp's qwen2moe architecture as is.
This model was built entirely on my own hardware, as an independent experiment. The results are promising, and
there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations
should all improve quality, but they need far more compute than my current hardware can provide.
With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new
models will be shared openly here.
If you find this work useful or interesting, any contribution helps.
Apache 2.0, the same as the base model Qwen/Qwen2.5-1.5B.
from transformers import AutoModelForCausalLM, AutoTokenizer path = "bayliner1980/Qwen2.5-1.5B-MoE-A0.9B"tok = AutoTokenizer.from_pretrained(path)model = AutoModelForCausalLM.from_pretrained(path, dtype="bfloat16", device_map="auto") ids = tok("The history of the printing press", return_tensors="pt").to(model.device)out = model.generate(**ids, max_new_tokens=80, do_sample=False)print(tok.decode(out[0], skip_special_tokens=True))