Highlights
- Most accurate open-weight model on our ERP benchmark (accuracy 87.1), ahead of open models up to 50x larger.
- General ability retained: matches or exceeds Qwen3-8B on GSM8K, AIME 2024, and MMLU-Pro, and on BFCL function calling.
- Single GPU: bfloat16 weights, standard Qwen3 chat template and tooling.
Usage
GoudERP keeps Qwen3's reasoning mode: it writes its reasoning inside a <think> block before the answer.
vllm serve SutanshuRaj/GoudERP --reasoning-parser qwen3 --max-model-len 40960
from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "SutanshuRaj/GoudERP"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "How do I set up an intercompany balancing rule between two primary ledgers?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, top_k=20)print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
Recommended sampling: temperature 0.6, top-p 0.95, top-k 20 (the Qwen3 reasoning-mode defaults).
Important: close the reasoning tag. In 1.7% of our benchmark answers the model ends generation without emitting
</think>, and the answer sits inside the unclosed block. A client that hides unclosed reasoning will show the user
nothing. Close the block at the end of generation instead of discarding it.
Training
Table with columns: Stage, What it does| Stage | What it does |
|---|
| Domain-adaptive pre-training | ERP text (documentation, API references, schemas, filtered web) mixed with general text |
| SFT Phase 1 | ERP conversations (single-turn, multiple-choice, multi-turn) mixed with general instruction data |
| SFT Phase 2 | ERP reasoning, business rules, tool calling, and general reasoning data |
| TIES merge | For each weight tensor, the 30% of entries that fine-tuning moved furthest are pulled 70% of the way back to Qwen3-8B |
The merge: theta_final = theta_SFT + 0.7 * trim_0.3(theta_base - theta_SFT).
Training data were generated from an ERP taxonomy: search queries and questions written by an LLM, web pages retrieved
and scored for relevance, and answers grounded in the retained pages.
Evaluation
ERP benchmark
Our ERP benchmark holds single-turn, multiple-choice, and multi-turn questions across ERP modules, processes, and
complexity levels. 2,614 turns (1,832 conversations) answered by every model. Answers are scored by the open-weight judge
gpt-oss-120b, which sees them without model names and with reasoning removed. Confidence intervals and p-values
come from a paired bootstrap over conversations (4,000 resamples).
Table with columns: Model, Params, Overall, Accuracy| Model | Params | Overall | Accuracy |
|---|
| GoudERP-8B | 8B | 87.7 | 87.1 |
| Nemotron-Cascade-2-30B-A3B | 30B (3B active) | 87.6 | 85.3 |
| Nemotron-3-Nano-30B-A3B | 30B (3B active) | 87.1 | 86.2 |
| Qwen3-235B-A22B | 235B (22B active) | 87.0 | 86.5 |
| Qwen3-32B | 32B | 86.9 | 86.5 |
| Sarvam-105B | 105B | 86.1 | 86.1 |
| Qwen3.5-397B-A17B | 397B (17B active) | 85.8 | 86.2 |
| Qwen3-8B (base) | 8B | 85.1 | 85.2 |
| Gemma-4-26B-A4B-it | 26B (4B active) | 84.3 | 85.6 |
GoudERP has the highest accuracy of every model compared (p <= 0.016) and the highest overall score,
statistically tied with Nemotron-Cascade-2-30B-A3B (p = 0.51) and ahead of the rest (p < 0.001).
General capabilities
Table with columns: Benchmark, Qwen3-8B (base), After fine-tuning, GoudERP-8B| Benchmark | Qwen3-8B (base) | After fine-tuning | GoudERP-8B |
|---|
| GSM8K | 95.1 | 72.3 | 95.2 |
| MATH-500 | 95.8 | 63.0 | 95.0 |
| AIME 2024 | 74.8 | 52.2 | 74.9 |
| AIME 2025 | 64.8 | 32.3 | 62.8 |
| MMLU-Pro | 73.1 | 46.5 | 75.4 |
| IFEval | 84.1 | 52.1 | 79.5 |
AIME is averaged over 64 samples per problem; IFEval is prompt-level strict accuracy.
All runs use the sampling settings above.
Function calling and agentic tasks
BFCL v4, restricted to its 17 standard categories, so the overall column is not comparable to the public BFCL leaderboard:
Table with columns: Model, Overall, Non-live, Live, Multi-turn, Hallucination| Model | Overall | Non-live | Live | Multi-turn | Hallucination |
|---|
| Qwen3.5-397B-A17B | 72.5 | 85.0 | 82.1 | 60.8 | 86.0 |
| Nemotron-3-Nano-30B-A3B | 66.1 | 82.4 | 80.5 | 50.1 | 83.0 |
| Gemma-4-26B-A4B-it | 61.6 | 75.5 | 80.8 | 44.4 | 79.9 |
| Qwen3-8B (base) | 58.7 | 84.1 | 80.9 | 35.8 | 79.7 |
| GoudERP-8B | 58.5 | 83.9 | 80.3 | 35.3 | 81.2 |
| Nemotron-Cascade-2-30B-A3B | 54.2 | 40.4 | 77.3 | 46.4 | 68.2 |
On tau2-bench (pass^1, 3 trials, gpt-oss-120b as the user simulator; not leaderboard-comparable) GoudERP scores
airline 0.427, retail 0.307, telecom 0.211, against 0.273 / 0.342 / 0.170 for Qwen3-8B. Both Nemotron 30B-A3B models
are ahead of it on these tasks.
Perplexity
Table with columns: Model, Held-out ERP text, WikiText-2| Model | Held-out ERP text | WikiText-2 |
|---|
| Qwen3-8B (base) | 5.61 | 8.36 |
| After fine-tuning | 3.27 | 11.18 |
| GoudERP-8B | 3.86 | 8.50 |
Limitations
- Instruction following is below the base model (IFEval 79.5 vs 84.1).
- Tool use is at base-model level, not above it, and larger models remain ahead on multi-turn tool calling.
- Scope: the training data centre on Oracle Fusion, SAP, Workday. Coverage of other ERP vendors is thinner.
- Data provenance: the synthetic training data were generated with closed models. Seed is absent from code.
- Smaller sizes: the same recipe applied to a 4B base model did not improve it.
License
Apache-2.0, following the Qwen3-8B base model.