GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
kirikir13
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
kirikir13
Model Tree
mistralai/Mistral-7B-Instruct-v0.2
Input Modalities
Output Modalities
Supported Functionality
MOEmoe = "Mistral of Experts". Three Mistral-family 7B models whose initials spell M-O-E:
They share the same tokenizer and tensor shapes, so they frankenmerge cleanly into one Mixtral-style MoE.
The original upload kirikir13/MOEmoe-3x7B-BF16 (and its bucket copies) is incomplete: 19.3 GB of the ~37.0 GB model. It only has layers 0–15 and part of layer 16. Layers 16–31, all 32 router (gate) weights, model.norm and lm_head are missing (upload/hard-drive failure). The original is left untouched. This repo is a full rebuild from the same source models with mergekit-moe.
| Class | MixtralForCausalLM (model_type: mixtral) |
| Experts | 3 (num_local_experts: 3), top-2 routing (num_experts_per_tok: 2) |
| Layers / hidden / FFN | 32 / 4096 / 14336 |
| Attention | 32 heads, 8 KV heads (GQA), head_dim 128 |
| RoPE theta / context | 1,000,000 / 32768 (from Mistral-7B-Instruct-v0.2), no sliding window |
| Vocab | 32000 (Mistral SentencePiece), bos <s>=1, eos </s>=2 |
| Params / size | ~18.5B total (~12.9B active per token with top-2), 37.0 GB BF16, 8 shards |
| Expert | Source model | Role |
|---|---|---|
| 0 | mistralai/Mistral-7B-Instruct-v0.2 | Main talker: personality, chat, tools (also supplies attention, embeddings, norms, lm_head) |
| 1 | openchat/openchat_3.5 | Logic and reasoning |
| 2 | openbmb/Eurus-7b-kto | Math |
The source weights came from George's bucket kirikir13/M-O-E_moe. The original config's _name_or_path was Mistral-7B-Instruct-v0.2, so that is the base. "Minstrel" in the notes above is Mistral-7B-Instruct-v0.2.
Router: gate_mode: cheap_embed. Each expert's router vector comes from the base model's token embeddings of its positive prompts (see mergekit_moe_config.yml). hidden mode needs a full forward pass of the base model, which did not fit in the 15 GB RAM build box. The router is a sensible starting point, not a trained one. Fine-tuning the gates (or a hidden rebuild on a bigger machine) will sharpen the routing.
The tokenizer is Mistral-7B-Instruct-v0.2's (32000 tokens). It matches vocab_size and the embed_tokens/lm_head shape [32000, 4096] exactly. The embeddings come from the Mistral base. OpenChat's tokenizer has 32002 tokens (<|end_of_turn|>, <|pad_0|>), which this model has no embeddings for, so the OpenChat "GPT4 Correct User" template is not used. The chat template is Mistral [INST]:
text
<s> [INST] {system}\n\n{user} [/INST] {assistant}</s>python
from transformers import AutoTokenizer, AutoModelForCausalLMimport torchrepo = "kirikir13/MOEmoe-3x7B-BF16-v2"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")msgs = [{"role": "user", "content": "Solve 3x + 5 = 20 and explain."}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(ids, max_new_tokens=256)print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))You need about 38 GB RAM/VRAM in BF16. Use 4-bit (bitsandbytes) or a GGUF quant for smaller machines.
"num_experts_per_tok": 1 in config.json, or config.num_experts_per_tok = 1 before loading. This is faster, but the router still picks per token.The weights use the classic Mixtral tensor names (block_sparse_moe.experts.N.w1/w2/w3, block_sparse_moe.gate), so llama.cpp converts them directly:
bash
python convert_hf_to_gguf.py ./MOEmoe-3x7B-BF16-v2 --outtype bf16 --outfile moemoe-3x7b-bf16.gguf./llama-quantize moemoe-3x7b-bf16.gguf moemoe-3x7b-Q4_K_M.gguf Q4_K_M # ~11 GB./llama-server -m moemoe-3x7b-Q4_K_M.gguf --chat-template mistral -c 8192To run on a single expert in llama.cpp, add --override-kv llama.expert_used_count=int:1.
model-0000{1..8}-of-00008.safetensors and model.safetensors.index.json: 515 tensors, 37,032,828,928 bytes, BF16config.json, generation_config.json (bos 1, eos 2)tokenizer.json, tokenizer.model, tokenizer_config.json, special_tokens_map.json, chat_template.jinjamergekit_moe_config.yml: the exact merge recipemergekit-moe --lazy-unpickle, CPU, gate_mode: cheap_embed, top-2) from Mistral-7B-Instruct-v0.2 + openchat_3.5 + Eurus-7b-kto. Converted to classic Mixtral tensor names for llama.cpp/vLLM/transformers-4.x compatibility. Added the tokenizer, [INST] chat template, generation config and index. Verified: the index matches all shard headers; transformers 5.20.2 loads it with 0 missing / 0 unexpected / 0 mismatched keys; a 2-layer forward pass gives finite logits and a working router.from transformers import AutoTokenizer, AutoModelForCausalLMimport torchrepo = "kirikir13/MOEmoe-3x7B-BF16-v2"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")msgs = [{"role": "user", "content": "Solve 3x + 5 = 20 and explain."}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(ids, max_new_tokens=256)print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))python convert_hf_to_gguf.py ./MOEmoe-3x7B-BF16-v2 --outtype bf16 --outfile moemoe-3x7b-bf16.gguf./llama-quantize moemoe-3x7b-bf16.gguf moemoe-3x7b-Q4_K_M.gguf Q4_K_M # ~11 GB./llama-server -m moemoe-3x7b-Q4_K_M.gguf --chat-template mistral -c 8192