GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Kiy-K
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Kiy-K
Model Tree
Kiy-K/moe-gpt-neox-125m-v1.1
Input Modalities
Output Modalities
Supported Functionality
| Feature | Specification |
|---|---|
| Base Model | Kiy-K/moe-gpt-neox-125m-v1.1 |
| Architecture | MoE-Enhanced GPT-NeoX |
| Total Parameters | ~125M (base) + MoE layers |
| MoE Experts | 8 specialized experts |
| Top-K Routing | 2 (sparse activation) |
| Training Method | Frozen base + MoE layer training |
| Training Hardware | Kaggle 2x Tesla T4 GPUs |
| Format | 🔒 Safetensors (secure) |
The model enhances every other transformer layer with a MoE routing system:
text
Input Token → Gating Network → Route to Top-2 Experts → Weighted Combination → OutputKey Components:
Advantages:
yaml
Training Steps: 6,000Batch Size: 16Gradient Accumulation: 2Effective Batch Size: 96Learning Rate: 1e-4Warmup Ratio: 0.06Optimizer: AdamW (β1=0.9, β2=0.95, weight_decay=0.1)Max Sequence Length: 512Training Time: ~3 hoursTrained on diverse, high-quality datasets:
| Dataset | Samples | Purpose |
|---|---|---|
| FineWeb-Edu | 7,000 | Educational web content |
| UltraChat 200k | 7,000 | Conversational data |
| The Stack (Python) | 7,000 | Code understanding |
Total Training Samples: 21,000
Data Quality:
python
import torchfrom transformers import AutoTokenizer, GPTNeoXForCausalLM # Load tokenizer (from Pythia for better template support)tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-160m") # Load base modelbase_model = GPTNeoXForCausalLM.from_pretrained("Kiy-K/moe-gpt-neox-125m-v1.1") # For full MoE functionality, you'll need the MoEGPTNeoX wrapper# (See architecture code in repository) # Basic generation example (with base model)text = "def fibonacci(n):"inputs = tokenizer(text, return_tensors="pt") with torch.no_grad(): outputs = base_model.generate( **inputs, max_length=100, temperature=0.7, top_p=0.9, do_sample=True ) print(tokenizer.decode(outputs[0], skip_special_tokens=True))python
from safetensors.torch import load_file # Load MoE weightsmoe_weights = load_file("moe_layers.safetensors") # Apply to your MoEGPTNeoX wrapper# (Full implementation in training script)Recommended settings:
python
# For creative tasks (code, stories)generation_config = { "temperature": 0.7, "top_p": 0.9, "top_k": 50, "repetition_penalty": 1.1} # For factual tasks (Q&A, documentation)generation_config = { "temperature": 0.3, "top_p": 0.85, "top_k": 40, "repetition_penalty": 1.0}This model excels at:
Best For:
python
class MoELayer: hidden_size: 512 intermediate_size: 2048 num_experts: 8 top_k: 2 dropout: 0.1 # Gating Network gate: Linear(512 -> 8) # Routes to experts # Expert Networks experts: [ ExpertNetwork(512 -> 2048 -> 512) # x8 ] # Load Balancing Loss auxiliary_loss: 0.01 * load_balancing_losstext
Initial Loss: ~3.5Final Loss: ~2.1Best Loss: ~2.0Training Steps: 6,000All 8 experts show balanced activation patterns thanks to load balancing loss, ensuring efficient parameter usage.
text
.├── model.safetensors # Base GPT-NeoX weights (secure format)├── moe_layers.safetensors # MoE expert weights (secure format)├── moe_config.json # MoE architecture configuration├── config.json # Model configuration├── tokenizer.json # Tokenizer vocabulary├── tokenizer_config.json # Tokenizer settings├── special_tokens_map.json # Special token mappings├── training_info.json # Training metrics & config└── README.md # This fileAll model weights use the safetensors format, providing:
Training data methodology ensures:
bash
pip install torch>=2.0.0pip install transformers>=4.35.0pip install safetensors>=0.4.0| Use Case | Minimum | Recommended |
|---|---|---|
| Inference | 2GB VRAM | 4GB VRAM |
| Fine-tuning | 8GB VRAM | 16GB VRAM |
| CPU Inference | 4GB RAM | 8GB RAM |
python
# Use float16 for faster inferencemodel = model.half().to("cuda") # Enable torch.compile (PyTorch 2.0+)model = torch.compile(model) # Use gradient checkpointing for fine-tuningmodel.gradient_checkpointing_enable()v1.5 (Current)
v1.1 (Base)
If you use this model in your research or projects, please cite:
bibtex
@misc{moe_gpt_neox_125m_v15, author = {Kiy-K}, title = {MoE GPT-NeoX 125M v1.5: Efficient Mixture of Experts Language Model}, year = {2025}, publisher = {HuggingFace}, url = {https://huggingface.co/Kiy-K/moe-gpt-neox-125m-v1.5}}Found a bug or have a suggestion? Please open an issue on the repository!
Apache 2.0 License
This model inherits the Apache 2.0 license from the base GPT-NeoX model. You are free to:
See LICENSE for full terms.
Built with ❤️ for the open-source AI community
Everyone deserves their own AI model - not just expensive API access.
Training Steps: 6,000Batch Size: 16Gradient Accumulation: 2Effective Batch Size: 96Learning Rate: 1e-4Warmup Ratio: 0.06Optimizer: AdamW (β1=0.9, β2=0.95, weight_decay=0.1)Max Sequence Length: 512Training Time: ~3 hoursimport torchfrom transformers import AutoTokenizer, GPTNeoXForCausalLM # Load tokenizer (from Pythia for better template support)tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-160m") # Load base modelbase_model = GPTNeoXForCausalLM.from_pretrained("Kiy-K/moe-gpt-neox-125m-v1.1") # For full MoE functionality, you'll need the MoEGPTNeoX wrapper# (See architecture code in repository) # Basic generation example (with base model)text = "def fibonacci(n):"inputs = tokenizer(text, return_tensors="pt") with torch.no_grad(): outputs = base_model.generate( **inputs, max_length=100, temperature=0.7, top_p=0.9, do_sample=True ) print(tokenizer.decode(outputs[0], skip_special_tokens=True))from safetensors.torch import load_file # Load MoE weightsmoe_weights = load_file("moe_layers.safetensors") # Apply to your MoEGPTNeoX wrapper# (Full implementation in training script)# For creative tasks (code, stories)generation_config = { "temperature": 0.7, "top_p": 0.9, "top_k": 50, "repetition_penalty": 1.1} # For factual tasks (Q&A, documentation)generation_config = { "temperature": 0.3, "top_p": 0.85, "top_k": 40, "repetition_penalty": 1.0}class MoELayer: hidden_size: 512 intermediate_size: 2048 num_experts: 8 top_k: 2 dropout: 0.1 # Gating Network gate: Linear(512 -> 8) # Routes to experts # Expert Networks experts: [ ExpertNetwork(512 -> 2048 -> 512) # x8 ] # Load Balancing Loss auxiliary_loss: 0.01 * load_balancing_lossInitial Loss: ~3.5Final Loss: ~2.1Best Loss: ~2.0Training Steps: 6,000.├── model.safetensors # Base GPT-NeoX weights (secure format)├── moe_layers.safetensors # MoE expert weights (secure format)├── moe_config.json # MoE architecture configuration├── config.json # Model configuration├── tokenizer.json # Tokenizer vocabulary├── tokenizer_config.json # Tokenizer settings├── special_tokens_map.json # Special token mappings├── training_info.json # Training metrics & config└── README.md # This filepip install torch>=2.0.0pip install transformers>=4.35.0pip install safetensors>=0.4.0# Use float16 for faster inferencemodel = model.half().to("cuda") # Enable torch.compile (PyTorch 2.0+)model = torch.compile(model) # Use gradient checkpointing for fine-tuningmodel.gradient_checkpointing_enable()@misc{moe_gpt_neox_125m_v15, author = {Kiy-K}, title = {MoE GPT-NeoX 125M v1.5: Efficient Mixture of Experts Language Model}, year = {2025}, publisher = {HuggingFace}, url = {https://huggingface.co/Kiy-K/moe-gpt-neox-125m-v1.5}}