ANMOLGPT-4B-v0.5 API & Inference Endpoint

Model Details

Model: ANMOLGPT-4B-v0.5
Base model: Qwen/Qwen3.5-4B
Parameters: ~4B
Architecture: Causal Language Model
Language: English
Primary task: Text generation and reasoning
Training focus: Mathematical reasoning
Fine-tuning dataset: GSM8K
Framework: Transformers / Unsloth
Quantization during training: 4-bit
Fine-tuning approach: Parameter-efficient fine-tuning

What's New in v0.5?

The main change in v0.5 is targeted training on GSM8K to improve mathematical reasoning.

Compared with ANMOLGPT-4B-v0.4, v0.5 shows a substantial improvement on GSM8K while largely preserving performance across several general-purpose benchmarks.

Key GSM8K Results

Table with columns: Metric, v0.4, v0.5, Change
Metric	v0.4	v0.5	Change
GSM8K Flexible Extract	54.36%	60.42%	+6.06 pp
GSM8K Strict Match	46.85%	56.18%	+9.33 pp

The improvement in strict-match accuracy is particularly notable, suggesting that the model became better at producing answers in the expected mathematical format.

Benchmark Results

All evaluations below were performed using lm-evaluation-harness.

GSM8K

5-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Flexible Extract	60.42% ± 1.35%
Strict Match	56.18% ± 1.37%

HellaSwag

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	55.32% ± 0.50%
Normalized Accuracy	73.30% ± 0.44%

PIQA

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	78.24% ± 0.96%
Normalized Accuracy	78.94% ± 0.95%

ARC-Easy

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	84.30% ± 0.75%
Normalized Accuracy	80.98% ± 0.81%

Winogrande

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	71.03% ± 1.27%

TruthfulQA

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
MC2 Accuracy	49.01% ± 1.49%

Note: TruthfulQA decreased compared with v0.4. This is an observed trade-off following the GSM8K-focused fine-tuning and is reported transparently.

MMLU

0-shot evaluation.

Overall

73.78% ± 0.35%

Table with columns: Category, Accuracy
Category	Accuracy
Humanities	66.16%
Other	77.57%
Social Sciences	83.13%
STEM	72.28%

Selected Mathematical Subjects

Table with columns: Subject, Accuracy
Subject	Accuracy
Abstract Algebra	52.00%
College Mathematics	62.00%
Elementary Mathematics	67.72%
High School Mathematics	47.78%
High School Statistics	**70.37%

v0.4 → v0.5 Comparison

Table with columns: Benchmark, v0.4, v0.5, Change
Benchmark	v0.4	v0.5	Change
GSM8K Flexible	54.36%	60.42%	+6.06 pp
GSM8K Strict	46.85%	56.18%	+9.33 pp
HellaSwag	55.06%	55.32%	+0.26 pp
PIQA	77.86%	78.24%	+0.38 pp

Interpretation

The results suggest that GSM8K-focused fine-tuning substantially improved mathematical reasoning without causing a broad degradation across the evaluated general-purpose benchmarks.

The largest improvement was observed on GSM8K:

54.36% → 60.42%

while MMLU, ARC-Easy, PIQA, HellaSwag and Winogrande remained broadly stable or improved.

However, TruthfulQA decreased:

54.61% → 49.01%

This highlights an important trade-off in targeted fine-tuning of smaller language models: improving a specific capability can affect other capabilities.

Future versions of ANMOLGPT will investigate methods for improving mathematical reasoning while preserving factuality and general reasoning performance.

Intended Use

ANMOLGPT-4B-v0.5 is intended primarily for:

Research into small language models
Mathematical reasoning experiments
Instruction-following research
Local inference experimentation
Model fine-tuning experiments
Benchmarking and evaluation
Educational experimentation

It can also be used as a general-purpose text-generation model, although it should not be considered a replacement for significantly larger frontier models.

Limitations

ANMOLGPT-4B-v0.5 is an experimental research model.

Known limitations include:

Mathematical reasoning remains imperfect.
The model may produce incorrect calculations or reasoning.
Truthfulness performance decreased after GSM8K-focused training.
The model can hallucinate information.
Benchmark performance does not necessarily translate directly into real-world reliability.
The model has substantially fewer parameters than modern frontier models.
No claim is made that the model is suitable for safety-critical or high-stakes applications.

Outputs should therefore be independently verified, especially for financial, medical, legal, scientific or other high-impact applications.

Training Approach

The model was developed through iterative experimentation.

The ANMOLGPT development process has focused on understanding how targeted datasets affect the capabilities of a relatively small language model.

v0.5 specifically investigates:

Can targeted mathematical reasoning training significantly improve a ~4B parameter model without causing widespread capability degradation?

The results indicate that the answer is partially yes.

Evaluation

Evaluation was performed using:

EleutherAI LM Evaluation Harness

The model was evaluated using the following tasks:

GSM8K
MMLU
ARC-Easy
PIQA
HellaSwag
Winogrande
TruthfulQA

Unless otherwise specified, evaluations used 0-shot settings.

GSM8K was evaluated using 5-shot evaluation.

Example Usage

Transformers

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "anmoldhandhania93/ANMOLGPT-4B-v0.5"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto"
)

prompt = "Solve this step by step: If a train travels 60 km in 1.5 hours, what is its average speed?"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Details

Model: ANMOLGPT-4B-v0.5
Base model: Qwen/Qwen3.5-4B
Parameters: ~4B
Architecture: Causal Language Model
Language: English
Primary task: Text generation and reasoning
Training focus: Mathematical reasoning
Fine-tuning dataset: GSM8K
Framework: Transformers / Unsloth
Quantization during training: 4-bit
Fine-tuning approach: Parameter-efficient fine-tuning

What's New in v0.5?

The main change in v0.5 is targeted training on GSM8K to improve mathematical reasoning.

Compared with ANMOLGPT-4B-v0.4, v0.5 shows a substantial improvement on GSM8K while largely preserving performance across several general-purpose benchmarks.

Key GSM8K Results

Table with columns: Metric, v0.4, v0.5, Change
Metric	v0.4	v0.5	Change
GSM8K Flexible Extract	54.36%	60.42%	+6.06 pp
GSM8K Strict Match	46.85%	56.18%	+9.33 pp

The improvement in strict-match accuracy is particularly notable, suggesting that the model became better at producing answers in the expected mathematical format.

Benchmark Results

All evaluations below were performed using lm-evaluation-harness.

GSM8K

5-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Flexible Extract	60.42% ± 1.35%
Strict Match	56.18% ± 1.37%

HellaSwag

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	55.32% ± 0.50%
Normalized Accuracy	73.30% ± 0.44%

PIQA

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	78.24% ± 0.96%
Normalized Accuracy	78.94% ± 0.95%

ARC-Easy

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	84.30% ± 0.75%
Normalized Accuracy	80.98% ± 0.81%

Winogrande

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
Accuracy	71.03% ± 1.27%

TruthfulQA

0-shot evaluation.

Table with columns: Metric, Score
Metric	Score
MC2 Accuracy	49.01% ± 1.49%

Note: TruthfulQA decreased compared with v0.4. This is an observed trade-off following the GSM8K-focused fine-tuning and is reported transparently.

MMLU

0-shot evaluation.

Overall

73.78% ± 0.35%

Table with columns: Category, Accuracy
Category	Accuracy
Humanities	66.16%
Other	77.57%
Social Sciences	83.13%
STEM	72.28%

Selected Mathematical Subjects

Table with columns: Subject, Accuracy
Subject	Accuracy
Abstract Algebra	52.00%
College Mathematics	62.00%
Elementary Mathematics	67.72%
High School Mathematics	47.78%
High School Statistics	**70.37%

v0.4 → v0.5 Comparison

Table with columns: Benchmark, v0.4, v0.5, Change
Benchmark	v0.4	v0.5	Change
GSM8K Flexible	54.36%	60.42%	+6.06 pp
GSM8K Strict	46.85%	56.18%	+9.33 pp
HellaSwag	55.06%	55.32%	+0.26 pp
PIQA	77.86%	78.24%	+0.38 pp

Interpretation

The results suggest that GSM8K-focused fine-tuning substantially improved mathematical reasoning without causing a broad degradation across the evaluated general-purpose benchmarks.

The largest improvement was observed on GSM8K:

54.36% → 60.42%

while MMLU, ARC-Easy, PIQA, HellaSwag and Winogrande remained broadly stable or improved.

However, TruthfulQA decreased:

54.61% → 49.01%

This highlights an important trade-off in targeted fine-tuning of smaller language models: improving a specific capability can affect other capabilities.

Future versions of ANMOLGPT will investigate methods for improving mathematical reasoning while preserving factuality and general reasoning performance.

Intended Use

ANMOLGPT-4B-v0.5 is intended primarily for:

Research into small language models
Mathematical reasoning experiments
Instruction-following research
Local inference experimentation
Model fine-tuning experiments
Benchmarking and evaluation
Educational experimentation

It can also be used as a general-purpose text-generation model, although it should not be considered a replacement for significantly larger frontier models.

Limitations

ANMOLGPT-4B-v0.5 is an experimental research model.

Known limitations include:

Mathematical reasoning remains imperfect.
The model may produce incorrect calculations or reasoning.
Truthfulness performance decreased after GSM8K-focused training.
The model can hallucinate information.
Benchmark performance does not necessarily translate directly into real-world reliability.
The model has substantially fewer parameters than modern frontier models.
No claim is made that the model is suitable for safety-critical or high-stakes applications.

Outputs should therefore be independently verified, especially for financial, medical, legal, scientific or other high-impact applications.

Training Approach

The model was developed through iterative experimentation.

The ANMOLGPT development process has focused on understanding how targeted datasets affect the capabilities of a relatively small language model.

v0.5 specifically investigates:

Can targeted mathematical reasoning training significantly improve a ~4B parameter model without causing widespread capability degradation?

The results indicate that the answer is partially yes.

Evaluation

Evaluation was performed using:

EleutherAI LM Evaluation Harness

The model was evaluated using the following tasks:

GSM8K
MMLU
ARC-Easy
PIQA
HellaSwag
Winogrande
TruthfulQA

Unless otherwise specified, evaluations used 0-shot settings.

GSM8K was evaluated using 5-shot evaluation.

Example Usage

Transformers

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "anmoldhandhania93/ANMOLGPT-4B-v0.5"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto"
)

prompt = "Solve this step by step: If a train travels 60 km in 1.5 hours, what is its average speed?"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

ANMOLGPT-4B-v0.5

README

Model Details

What's New in v0.5?

Key GSM8K Results

Benchmark Results

GSM8K

HellaSwag

PIQA

ARC-Easy

Winogrande

TruthfulQA

MMLU

Overall

Selected Mathematical Subjects

v0.4 → v0.5 Comparison

Interpretation

Intended Use

Limitations

Training Approach

Evaluation

Example Usage

Transformers

Explore FriendliAI today

README

Model Details

What's New in v0.5?

Key GSM8K Results

Benchmark Results

GSM8K

HellaSwag

PIQA

ARC-Easy

Winogrande

TruthfulQA

MMLU

Overall

Selected Mathematical Subjects

v0.4 → v0.5 Comparison

Interpretation

Intended Use

Limitations

Training Approach

Evaluation

Example Usage

Transformers