nareshmeena12

fluxion-370m-instruct

README

License: apache-2.0

What is Fluxion?

Fluxion-370M-Instruct is a 370M-parameter instruction-tuned language model built from scratch, focused on mathematical reasoning and code generation. Its core design philosophy is simple: do more with less.

Trained on just 227B tokens — a fraction of what comparable models consume — Fluxion achieves competitive performance on math and coding benchmarks against models that have seen 18–28× more data. On token efficiency (MATH-500 score per trillion training tokens), Fluxion ranks 1st across all models in its class by a dominant margin.

The model went through three training stages: large-scale pretraining, supervised fine-tuning for instruction following, and GRPO reinforcement learning specifically targeting mathematical reasoning with rule-based rewards.

⚡ Token Efficiency — The Core Story

Table with columns: Model, Training Tokens, MATH-500, MATH-500 / Trillion Tokens
Model	Training Tokens	MATH-500	MATH-500 / Trillion Tokens
Fluxion-370M	0.227T	17.40%	🥇 76.7×
MobileLLM-R1-360M	4.2T	25.60%	6.1×
Gemma3-270M	6T	8.20%	1.4×
Granite-350M	11T	19.20%	1.7×
Qwen2.5-0.5B	18T	29.80%	1.7×
LFM2.5-350M	28T	13.40%	0.5×
SmolLM2-360M	4T	3.40%	0.9×

Fluxion delivers 76.7× higher MATH-500 score per trillion training tokens than any other model in this comparison. This is a structural advantage from architecture choices, custom tokenization, and a high-signal data pipeline — not a measurement artifact.

🏗️ Architecture

Fluxion is a clean, modern decoder-only transformer. No hybrid layers, no custom operators — just well-chosen standard components assembled deliberately.

markdown
FluxionForCausalLM
├── Embedding          (vocab_size=50,000 · hidden_size=1024)
├── 26 × DecoderBlock
│   ├── RMSNorm        (ε = 1e-6)
│   ├── GroupedQueryAttention
│   │   ├── 16 Query heads · 8 KV heads (GQA 2:1)
│   │   └── RoPE (θ = 500,000) + NTK scaling
│   ├── RMSNorm        (ε = 1e-6)
│   └── SwiGLU FFN     (hidden → 3072 → hidden)
└── RMSNorm → LM Head  (weight-tied to embedding)

Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Total Parameters	~370M
Layers	26
Hidden Size	1024
FFN Intermediate Size	3072
Attention Heads (Q)	16
KV Heads (GQA)	8
GQA Group Ratio	2 : 1
Head Dimension	64

Design Choices & Rationale

Grouped Query Attention (16Q / 8KV): Halves KV cache memory vs standard MHA with no meaningful quality loss at this scale. Larger effective batch sizes at both training and inference time.

RoPE with θ = 500,000 + NTK scaling: High base frequency improves length generalisation beyond the training window. NTK scaling further stabilises attention patterns at longer sequences without fine-tuning.

SwiGLU activation: Consistently stronger than GeLU/ReLU on reasoning tasks. The gated structure gives the FFN richer expressiveness per parameter.

Custom BPE tokenizer (50k vocab): Trained from scratch on 100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. A smaller, domain-tuned vocabulary means fewer tokens per math expression and code snippet, directly improving effective context utilisation and reducing sequence lengths vs generic 128k tokenizers.

Weight tying (embed ↔ lm_head): Saves ~50M parameters, keeps the model compact, and forces the input and output representation spaces to align.

Flash Attention: Full Flash Attention used throughout training for memory efficiency and throughput. TF32 enabled for matrix multiplications.

📊 Benchmarks

All results are 0-shot unless noted. Evaluation is strict.

Math & Code

Table with columns: Model, Tokens, GSM8K, MATH-500, HumanEval, MBPP, Avg
Model	Tokens	GSM8K	MATH-500	HumanEval	MBPP	Avg
Qwen2.5-0.5B	18T	43.44%	29.80%	30.49%	38.91%	35.66%
Granite-350M	11T	31.69%	19.20%	29.88%	42.02%	30.70%

General Reasoning

Table with columns: Model, ARC, PIQA, OBQA, CSQA, MMLU, Wino, IFEval, Avg
Model	ARC	PIQA	OBQA	CSQA	MMLU	Wino	IFEval	Avg
Qwen2.5-0.5B	50.9%	70.1%	36.2%	42.3%	45.9%	51.1%	74.1%	52.9%
Granite-350M	51.8%	68.9%

Rankings Summary

Table with columns: Category, Score, Rank
Category	Score	Rank
Math Average (GSM8K + MATH-500)	23.49%	4th
Code Average (HumanEval + MBPP)	18.35%	5th
Math + Code Combined	20.92%	4th
General Average	39.19%	7th
Token Efficiency (MATH-500 / T tokens)	76.7×	🥇 1st

Fluxion's design target is math and code under tight data constraints. On general benchmarks like IFEval and MMLU it trails models trained on significantly more diverse data — this is expected and intentional. On token efficiency it has no close competitor.

💬 Chat Template & Tokenizer

Fluxion-Instruct uses the ChatML format:

markdown
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{your prompt here}<|im_end|>
<|im_start|>assistant

Special Tokens

Table with columns: Token, ID, Role
Token	ID	Role
`<\|im_start\|>`	100278	Turn start marker
`<\|im_end\|>`	100279	Turn end marker
`<\|pad\|>`	—	Padding token

Stopping Token IDs

Always pass eos_token_id=[1, 4] during generation. Without these, the model may continue generating beyond the intended response boundary:

python
output = model.generate(
    **inputs,
    eos_token_id=[1, 4],
    ...
)

Token ID 1 is the primary EOS token and token ID 4 is the <|im_end|> marker. Including both ensures clean turn termination in all inference scenarios.

Tokenizer

Fluxion uses a custom BPE tokenizer trained from scratch on ~100GB of domain-focused text — a curated subset of the pretraining corpus weighted toward mathematics, code, and English prose. With a 50,000 token vocabulary, it is significantly more compact than general-purpose tokenizers (32k–128k). This means:

Fewer tokens per math expression and code snippet
Better effective context utilisation within the 4096 token window
Faster embedding lookups and a smaller lm_head projection

🚀 Quick Start

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "nareshmeena12/fluxion-370m-instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "nareshmeena12/fluxion-370m-instruct"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant that solves math problems step by step."},
    {"role": "user",   "content": "What is the sum of the first 100 natural numbers?"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        eos_token_id=[1, 4],
        do_sample=True,
        top_p=0.9,
    )

response = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True
)
print(response)

Note: Always include eos_token_id=[1, 4] in all generate() calls regardless of quantization mode.

🗂️ Training

Fluxion was trained in three stages.

Stage 1 — Pretraining

Large-scale unsupervised pretraining on a curated mix of math, code, and English text.

Table with columns: Parameter, Value
Parameter	Value
Total Tokens	~227B (0.227T)
Steps	264,000
Batch Size	10 sequences
Gradient Accumulation	21 steps
Effective Batch	~860K tokens/step
Learning Rate	3e-4
Min Learning Rate	9e-5
LR Schedule	Cosine decay

Pretraining data: (to be added)

Stage 2 — Supervised Fine-Tuning (SFT)

Instruction tuning using ChatML format to teach the model to follow user instructions across math, code, and general tasks.

Table with columns: Parameter, Value
Parameter	Value
Format	ChatML
Data	(to be added)

Stage 3 — GRPO (Math Reasoning)

Group Relative Policy Optimisation applied specifically to strengthen mathematical reasoning.

Table with columns: Parameter, Value
Parameter	Value
Algorithm	GRPO
Reward Signal	Rule-based answer correctness
Target Domain	Mathematics
Data	(to be added)

Rule-based rewards verify final answer correctness against ground truth — no LLM judge, no model-in-the-loop. This keeps the reward signal clean, fast, and free from reward hacking through verbosity or style.

🎯 Intended Use

Fluxion-370M-Instruct is designed for:

Mathematical problem solving and step-by-step reasoning
Code generation and explanation (Python, general purpose)
Instruction-following tasks in math and code domains
Fine-tuning base for domain-specific math/code applications
Efficient deployment — capable performance on modest hardware
Research into token-efficient pretraining and RL for reasoning

Fluxion-370M-Instruct is NOT designed for:

Knowledge-intensive QA requiring broad world knowledge
Long-document tasks beyond 4096 tokens
Multilingual use cases
Safety-critical production deployments without additional alignment

⚠️ Limitations

Narrow training data scope. 0.227T tokens is intentionally small and domain-focused. The model has limited world knowledge outside math, code, and general English.
Context length is 4096 tokens. Not suitable for long-document tasks without chunking strategies.
General benchmarks lag. IFEval (66.9%) and MMLU (25.3%) reflect that the model was not optimised for instruction following diversity or broad knowledge recall — this is a deliberate trade-off.
Custom tokenizer. The 50k BPE vocabulary is domain-tuned. Rare tokens, non-English text, and highly specialised symbols outside the training distribution may tokenise suboptimally.
Small scale. At 370M parameters, the model is outperformed by larger models on complex multi-step reasoning chains requiring deep world knowledge.
Always set eos_token_id=[1, 4]. Without explicit stopping token IDs the model may continue generating past the intended response boundary.

📰 News

 — Fluxion-370M-Instruct released on Hugging Face
 — Training code released on GitHub
 — Technical report available on arXiv

📄 Citation

bibtex
@misc{fluxion2025,
  title   = {Fluxion-370M: Token-Efficient Pretraining and Reinforcement Learning for Mathematical and Code Reasoning},
  author  = {Meena, Naresh},
  year    = {2025},
  note    = {<!-- ADD: arXiv link or report URL -->}
}

📬 Contact

Less data. Less compute. More signal.

🤗 Model · 💻 Code · 📄 Paper

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.