Petrouil

FrankenCPM-4x1B-A2B

Highlights

4x1B MoE architecture: Mixtral-style with 4 experts, 2 active per token
Domain-specialized experts: Agent (tool use), Coding, Reasoning, and General (base model)
BAR-inspired training: All experts train on all data — specialization via LoRA configs + router
Hidden-gate router: Initialized with domain-specific positive/negative prompts for expert routing
Efficient active compute: ~1.3B active params per token (vs 2.61B total)
Built from MiniCPM5-1B: Strong multilingual base with Chinese + English support

Model Specifications

Table with columns: Property, Value
Property	Value
Architecture	`MixtralForCausalLM` (Mixtral-style MoE)
Base Model	openbmb/MiniCPM5-1B
Total Parameters	~2.61B
Active Parameters / Token	~1.3B
Experts	4
Experts per Token	2
Context Length	32,768
Gate Mode	`hidden` (domain-supervised)
Dtype	`bfloat16`
License	Apache 2.0

Expert Composition

Table with columns: Expert, Domain, LoRA Rank, LoRA Alpha, Training Tokens, LR, Specialization
Expert	Domain	LoRA Rank	LoRA Alpha	Training Tokens	LR	Specialization
0 — Agent	Agentic tool use	64	128	70M	2e-4	Multi-turn tool calls, bash, read, edit, write, grep, todowrite
1 — Coding	Code generation	64	128	150M	2e-4

All LoRA experts train on MLP layers only. Attention layers remain frozen to the base model weights.

Training Details

Expert Fine-Tuning (LoRA)

All 3 domain experts are fine-tuned from openbmb/MiniCPM5-1B using LoRA adapters targeting MLP layers only. This follows the BAR (Branch-Adapt-Route) recipe from Ai2 — all experts train on ALL data, with specialization emerging from different LoRA hyperparameters and the MergeKit router.

Table with columns: Expert, Dataset, Max Samples, Batch, Grad Accum, Optim, Scheduler
Expert	Dataset	Max Samples	Batch	Grad Accum	Optim	Scheduler
Agent	`Petrouil/opencode-agentic-mini`	5,000	1	16	paged_adamw_8bit	cosine
Coding	CodeDataset (50K samples)	50,000	4	4

All experts use 4-bit NF4 quantization during training with QLoRA.

Router Training

After MoE assembly via MergeKit, the router (gate) is trained on a stratified 5% sample of the SFT dataset. All expert and shared weights remain frozen. The router learns to route tokens to the appropriate expert based on domain signals from the hidden-gate initialization.

MergeKit MoE Assembly

The 4 experts are assembled into a Mixtral-style MoE using MergeKit with gate_mode: hidden. The router is initialized using positive/negative prompt pairs that define each expert's domain affinity.

Usage

Transformers

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Petrouil/FrankenCPM-4x1B-A2B",
    device_map="auto",
    torch_dtype="bfloat16",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "Petrouil/FrankenCPM-4x1B-A2B",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Write a Python function to find all prime numbers up to n using the Sieve of Eratosthenes."}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM

python
from vllm import LLM, SamplingParams

llm = LLM(
    model="Petrouil/FrankenCPM-4x1B-A2B",
    tensor_parallel_size=1,
    trust_remote_code=True,
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(
    ["Explain the difference between a stack and a queue."],
    sampling_params,
)
print(outputs[0].outputs[0].text)

Evaluation

Agent Benchmark (Tool Use)

Evaluated on 10 agentic tool-use tasks across 5 categories. Base model (MiniCPM5-1B) used as reference.

Table with columns: Task, Category, Score, Tool Detected, Content Correct
Task	Category	Score	Tool Detected	Content Correct
list_files	single_tool	1.0	No	Yes
read_file	single_tool	1.0	No	Yes
tool_selection	tool_selection	1.0	No	Yes

Overall: 9/10 tasks passed (90%)

Reasoning Checkpoints

The reasoning expert was evaluated across training checkpoints on math proofs and logic puzzles. Base model responses are long-chain-of-thought with correct reasoning. Fine-tuned responses show improved conciseness and structure while maintaining correctness.

Table with columns: Prompt, Domain, Base Model Response Style, Fine-tuned Response Style
Prompt	Domain	Base Model Response Style	Fine-tuned Response Style
Number theory proof (p² ≡ 1 mod 24)	Math	Detailed CRT decomposition, verbose	More structured, step-by-step
Combinatorial identity proof	Math	Vandermonde identity attempt, lengthy	Cleaner combinatorial argument
Invariant reasoning (5×5 board game)	Logic	Case analysis, lengthy reasoning	More concise, focused analysis

Limitations

Tool detection: The model does not natively detect or emit structured tool calls — it generates natural language descriptions of tool usage
Single expert routing: Only 2 of 4 experts activate per token, so domain knowledge is not always perfectly routed
Base model constraints: Inherits MiniCPM5-1B's limitations (1B parameter capacity, 32K context)
Training data: Experts were trained on specific datasets — performance may degrade outside those domains
No function calling: Not designed for structured function calling APIs — intended for conversational tool-use scenarios

Citation

bibtex
@software{frankencpm2026,
  title  = {FrankenCPM-4x1B-A2B: A Domain-Specialized Mixture-of-Experts Model},
  author = {Petrouil},
  year   = {2026},
  url    = {https://huggingface.co/Petrouil/FrankenCPM-4x1B-A2B}
}

License

Apache 2.0 — see LICENSE.

Acknowledgments

openbmb/MiniCPM5-1B — Base model
MergeKit — MoE assembly toolkit
BAR (Branch-Adapt-Route) — Training recipe inspiration from Ai2
Unsloth — LoRA fine-tuning framework

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Petrouil

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Highlights

4x1B MoE architecture: Mixtral-style with 4 experts, 2 active per token
Domain-specialized experts: Agent (tool use), Coding, Reasoning, and General (base model)
BAR-inspired training: All experts train on all data — specialization via LoRA configs + router
Hidden-gate router: Initialized with domain-specific positive/negative prompts for expert routing
Efficient active compute: ~1.3B active params per token (vs 2.61B total)
Built from MiniCPM5-1B: Strong multilingual base with Chinese + English support

Model Specifications

Table with columns: Property, Value
Property	Value
Architecture	`MixtralForCausalLM` (Mixtral-style MoE)
Base Model	openbmb/MiniCPM5-1B
Total Parameters	~2.61B
Active Parameters / Token	~1.3B
Experts	4
Experts per Token	2
Context Length	32,768
Gate Mode	`hidden` (domain-supervised)
Dtype	`bfloat16`
License	Apache 2.0

Expert Composition

Table with columns: Expert, Domain, LoRA Rank, LoRA Alpha, Training Tokens, LR, Specialization
Expert	Domain	LoRA Rank	LoRA Alpha	Training Tokens	LR	Specialization
0 — Agent	Agentic tool use	64	128	70M	2e-4	Multi-turn tool calls, bash, read, edit, write, grep, todowrite
1 — Coding	Code generation	64	128	150M	2e-4

All LoRA experts train on MLP layers only. Attention layers remain frozen to the base model weights.

Training Details

Expert Fine-Tuning (LoRA)

Table with columns: Expert, Dataset, Max Samples, Batch, Grad Accum, Optim, Scheduler
Expert	Dataset	Max Samples	Batch	Grad Accum	Optim	Scheduler
Agent	`Petrouil/opencode-agentic-mini`	5,000	1	16	paged_adamw_8bit	cosine
Coding	CodeDataset (50K samples)	50,000	4	4

All experts use 4-bit NF4 quantization during training with QLoRA.

Router Training

MergeKit MoE Assembly

The 4 experts are assembled into a Mixtral-style MoE using MergeKit with gate_mode: hidden. The router is initialized using positive/negative prompt pairs that define each expert's domain affinity.

Usage

Transformers

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Petrouil/FrankenCPM-4x1B-A2B",
    device_map="auto",
    torch_dtype="bfloat16",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "Petrouil/FrankenCPM-4x1B-A2B",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Write a Python function to find all prime numbers up to n using the Sieve of Eratosthenes."}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM

python
from vllm import LLM, SamplingParams

llm = LLM(
    model="Petrouil/FrankenCPM-4x1B-A2B",
    tensor_parallel_size=1,
    trust_remote_code=True,
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(
    ["Explain the difference between a stack and a queue."],
    sampling_params,
)
print(outputs[0].outputs[0].text)

Evaluation

Agent Benchmark (Tool Use)

Evaluated on 10 agentic tool-use tasks across 5 categories. Base model (MiniCPM5-1B) used as reference.

Table with columns: Task, Category, Score, Tool Detected, Content Correct
Task	Category	Score	Tool Detected	Content Correct
list_files	single_tool	1.0	No	Yes
read_file	single_tool	1.0	No	Yes
tool_selection	tool_selection	1.0	No	Yes

Overall: 9/10 tasks passed (90%)

Reasoning Checkpoints

Table with columns: Prompt, Domain, Base Model Response Style, Fine-tuned Response Style
Prompt	Domain	Base Model Response Style	Fine-tuned Response Style
Number theory proof (p² ≡ 1 mod 24)	Math	Detailed CRT decomposition, verbose	More structured, step-by-step
Combinatorial identity proof	Math	Vandermonde identity attempt, lengthy	Cleaner combinatorial argument
Invariant reasoning (5×5 board game)	Logic	Case analysis, lengthy reasoning	More concise, focused analysis

Limitations

Tool detection: The model does not natively detect or emit structured tool calls — it generates natural language descriptions of tool usage
Single expert routing: Only 2 of 4 experts activate per token, so domain knowledge is not always perfectly routed
Base model constraints: Inherits MiniCPM5-1B's limitations (1B parameter capacity, 32K context)
Training data: Experts were trained on specific datasets — performance may degrade outside those domains
No function calling: Not designed for structured function calling APIs — intended for conversational tool-use scenarios

Citation

bibtex
@software{frankencpm2026,
  title  = {FrankenCPM-4x1B-A2B: A Domain-Specialized Mixture-of-Experts Model},
  author = {Petrouil},
  year   = {2026},
  url    = {https://huggingface.co/Petrouil/FrankenCPM-4x1B-A2B}
}

License

Apache 2.0 — see LICENSE.

Acknowledgments

openbmb/MiniCPM5-1B — Base model
MergeKit — MoE assembly toolkit
BAR (Branch-Adapt-Route) — Training recipe inspiration from Ai2
Unsloth — LoRA fine-tuning framework

FrankenCPM-4x1B-A2B

README

Highlights

Model Specifications

Expert Composition

Training Details

Expert Fine-Tuning (LoRA)

Router Training

MergeKit MoE Assembly

Usage

Transformers

vLLM

Evaluation

Agent Benchmark (Tool Use)

Reasoning Checkpoints

Limitations

Citation

License

Acknowledgments

Explore FriendliAI today

README

Highlights

Model Specifications

Expert Composition

Training Details

Expert Fine-Tuning (LoRA)

Router Training

MergeKit MoE Assembly

Usage

Transformers

vLLM

Evaluation

Agent Benchmark (Tool Use)

Reasoning Checkpoints

Limitations

Citation

License

Acknowledgments