GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
FlameF0X
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
FlameF0X
Model Tree
Input Modalities
Output Modalities
Supported Functionality
num_experts_per_tok": 2)Unlike a standard dense model, an MoE model stores a larger footprint of parameters on disk but selectively activates only a subset for any given token during a forward pass:
| Component | Total Parameters | Status During Inference |
|---|---|---|
| Embeddings (Input + LM Head) | 24,576,000 | Always Active |
| Attention Blocks (10 Layers) | 4,423,680 | Always Active |
| MoE Routers (10 Layers) | 30,720 | Always Active |
| Experts (8 Total across 10 Layers) | 70,778,880 | 2 of 8 Active per Layer (~17.6M active) |
| Overall Footprint | 99,809,280 | 22,544,640 Active per Token |
This model was trained on a HuggingFaceTB/smollm-corpus subsets cosmopedia-v2 and fineweb-edu-dedup
You can load and experiment with this model using the Hugging Face transformers library:
python
from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "FlameF0X/TinyMoE-100m-2x8-retrained"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id) input_text = "Wikipedia is a free"inputs = tokenizer(input_text, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=50)print(tokenizer.decode(outputs[0], skip_special_tokens=True))text
> Wikipedia is a free, open source of information about the world. It is a great resource for anyone who has been able to read and write in a way that is easy to read.The first thing that is in the world of the internet is that it is notfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "FlameF0X/TinyMoE-100m-2x8-retrained"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id) input_text = "Wikipedia is a free"inputs = tokenizer(input_text, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=50)print(tokenizer.decode(outputs[0], skip_special_tokens=True))> Wikipedia is a free, open source of information about the world. It is a great resource for anyone who has been able to read and write in a way that is easy to read.The first thing that is in the world of the internet is that it is not