Math-tuned base

This checkpoint starts from SupraLabs/Supra-50M-Base and was continually pretrained as a base causal language model on a math-heavy mixture:

• 34% HuggingFaceTB/finemath (finemath-4plus)
• 25% openbmb/UltraData-Math (UltraData-Math-L2-preview)
• 15% meta-math/MetaMathQA
• 14% local synthetic arithmetic
• 6% microsoft/orca-math-word-problems-200k
• 5% Menouar/LinearEquations
• 3% openai/gsm8k (main)

Training used packed 1024-token causal-LM blocks for about 10,000 optimizer steps.

This is a base model, not an instruction-tuned assistant.

Important limitation

This model is only about 50M parameters and was continued-pretrained from a weak base model. It is not usable as a reliable math model. It mostly follows the surface pattern of math text and word-problem solutions, and its generated answers should not be expected to be correct. Treat outputs as experimental pattern completions, not as solved arithmetic or mathematical reasoning.

Usage

python
Copy code
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Abhiram1009/Supra-50M-Math-CPT"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

inputs = tokenizer(
    "John has 22 apples, he eats 10 of them, now john has",
    return_tensors="pt",
)
outputs = model.generate(
    **inputs,
    max_new_tokens=32,
    do_sample=True,
    temperature=0.7,
    top_k=40,
    top_p=0.95,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))