Siddh07ETH

Pluto-Genesis-0.5B-LLM

README

License: apache-2.0

Model Description

Pluto-Genesis-0.6B is a fine-tuned instruction-following language model built on top of Qwen3-0.6B. It was trained using QLoRA (Quantized Low-Rank Adaptation) on a curated mixture of 80,000 high-quality instruction-response pairs spanning general reasoning, mathematical problem solving, and code generation.

This model is part of the Pluto AI research project by Siddharth N.R., exploring efficient fine-tuning of sub-1B language models on consumer-grade hardware.

Research Goal: Demonstrate that a sub-1B model fine-tuned on carefully curated data can achieve competitive performance on reasoning, math, and coding benchmarks while remaining deployable on consumer hardware.

Training Details

Table with columns: Property, Value
Property	Value
Base Model	Qwen/Qwen3-0.6B
Parameters	~620 Million
Method	QLoRA (4-bit NF4 + LoRA)
LoRA Rank	r=64, α=128
LoRA Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training Steps	2,475
Final Training Loss	0.2741
Precision	FP16
Optimizer	Paged AdamW 8-bit
Learning Rate	2e-4 (cosine schedule)
Effective Batch Size	32 (2 × 16 grad accum)
Sequence Length	1024 tokens
Hardware	Tesla T4 (16 GB)
Framework	Transformers + PEFT + TRL

Training Data

Table with columns: Domain, Dataset, Samples, Skills Targeted
Domain	Dataset	Samples	Skills Targeted
🧠 General Reasoning	OpenHermes-2.5	30,000	Instruction following, reasoning
🔢 Mathematics	Orca-Math-200K	30,000	Word problems, step-by-step math
💻 Code	CodeFeedback-Filtered

Benchmarks

📊 Benchmarks will be added shortly. The model is currently being evaluated on standard benchmarks.

Table with columns: Benchmark, Pluto-Genesis-0.6B, Qwen3-0.6B (Base), SmolLM2-360M
Benchmark	Pluto-Genesis-0.6B	Qwen3-0.6B (Base)	SmolLM2-360M
ARC-Easy	Coming soon	—	—
ARC-Challenge	Coming soon	—	—
MMLU	Coming soon	—	—
GSM8K	Coming soon	—	—

Usage

Basic Inference

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Siddh07ETH/Pluto-Genesis-0.5B-LLM",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Siddh07ETH/Pluto-Genesis-0.5B-LLM")

messages = [
    {"role": "user", "content": "Explain what a neural network is."}
]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.3,
        do_sample=True,
        top_p=0.9,
        repetition_penalty=1.1,
    )

response = tokenizer.decode(
    output[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)
print(response)

Low Memory Inference (4-bit)

python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)
model = AutoModelForCausalLM.from_pretrained(
    "Siddh07ETH/Pluto-Genesis-0.5B-LLM",
    quantization_config=quant_config,
    device_map="auto",
)

Recommended Generation Settings

Table with columns: Setting, Value, Reason
Setting	Value	Reason
`temperature`	0.3	Conservative — reduces hallucinations
`top_p`	0.9	Focused vocabulary
`repetition_penalty`	1.1	Prevents rambling
`max_new_tokens`	200–512	Keeps answers concise

Limitations

Model size: At 620M parameters this model will hallucinate on topics outside its training distribution. Always verify factual claims.
Context length: Trained on sequences up to 1024 tokens. Performance may degrade on longer contexts.
Knowledge cutoff: The model does not have access to real-time information.
Research only: Not intended for production deployment without further evaluation and safety testing.

Author

Siddharth N.R. HuggingFace

Citation

bibtex
@misc{plutogenesis2026,
  author    = {Siddharth N.R.},
  title     = {Pluto-Genesis-0.6B: An Instruction-Tuned Sub-1B Language Model},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/Siddh07ETH/Pluto-Genesis-0.5B-LLM}
}

License

Apache 2.0 — see LICENSE. Base model Qwen3-0.6B is also Apache 2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Siddh07ETH

Model Tree

Base

Qwen/Qwen3-0.6B

Fine-tuned

this model

Input Modalities

Text

Output Modalities