GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Siddh07ETH
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Siddh07ETH
Model Tree
Base
Qwen/Qwen2.5-Coder-0.5B
Fine-tuned
this model
Input Modalities
Text
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
Container
Atlas-Coder-0.5B is a coding-specialized language model instruction-tuned from scratch on top of Qwen2.5-Coder-0.5B base (not instruct). Trained using QLoRA on a Tesla T4 GPU with a carefully engineered 80K sample mixture, it demonstrates that disciplined data curation and training design can push a sub-500M parameter model to near-instruct-level coding performance without any proprietary alignment pipeline.
This model is part of the Pluto AI research project by Siddharth N.R., following the Pluto-Genesis-0.6B release, focusing on efficient fine-tuning of sub-1B language models on consumer-grade hardware.
Research Goal: Prove that a sub-1B coding model fine-tuned on curated, decontaminated open-source data can match or exceed the coding performance of officially instruction-tuned variants of the same architecture — without RLHF, proprietary data, or large-scale compute.
⚠️ Note: This is an Infrastructure Case Study, not a SOTA Benchmark model.
Why is the score low?
This V1 model was fine-tuned on the Qwen2.5-Coder-0.5B-Base model using ChatML format. Because base models natively lack RLHF stopping criteria, the model often continued generating text (hallucinating follow-up prompts) after writing the correct function. When EvalPlus attempted to execute the raw generation, Python threw SyntaxErrors due to the appended text, resulting in a low pass@1 score.
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-Coder-0.5B (Base, not Instruct) |
| Parameters | ~494 Million |
| Method | QLoRA (4-bit NF4 + LoRA) |
| LoRA Rank | r=64, α=128 |
| LoRA Dropout | 0.05 |
| LoRA Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| modules_to_save | embed_tokens, lm_head (fully unfrozen for base model adaptation) |
| Trainable Parameters |
| Domain | Dataset | Samples | Purpose |
|---|---|---|---|
| 🧬 Synthetic Complexity | Magicoder-Evol-Instruct-110K | 15,000 | Multi-step code synthesis |
| ✅ Exec-Verified OSS | self-oss-instruct-sc2-exec-filter-50k | 50,000 | Single-function completion (HumanEval+ aligned) |
| 🔍 Real-World Debug | CodeFeedback-Filtered-Instruction |
All datasets were scanned using n-gram Jaccard similarity (8-gram, threshold 0.3) against the full HumanEval test set before training. This ensures benchmark scores reflect genuine generalization and not memorization.
| Dataset | Pre-decontam | Removed | Post-decontam |
|---|---|---|---|
| Magicoder | 15,000 | 7 | 14,993 |
| OSS-Instruct | 50,000 | 0 | 50,000 |
| CodeFeedback | 10,000 | 5 | 9,995 |
| TACO | 5,000 | 0 | 5,000 |
1. Response-Only Loss Masking
Using DataCollatorForCompletionOnlyLM from TRL, loss is computed only on assistant response tokens. This prevents the model from wasting gradient steps learning to predict system prompts and user messages — the single highest-ROI change for HumanEval+ performance.
2. Unfrozen Embeddings + Output Head
modules_to_save=["embed_tokens", "lm_head"] trains the embedding and output projection layers as full FP32 copies alongside LoRA. Critical when fine-tuning from a base (not instruct) model — the token distribution needs to shift significantly to learn the ChatML instruction format.
3. FP32 LoRA Cast on T4 PEFT 0.17 initializes LoRA matrices in BF16 by default. Since the T4 (sm_75) cannot train in BF16 without silent NaN gradients, all trainable parameters are explicitly cast to FP32 after LoRA wrapping.
4. Exec-Verified OSS Data as Primary Source
50K of 80K samples (62.5%) come from self-oss-instruct-sc2-exec-filter-50k — execution-verified, single-function Python completions derived from real open-source code. This dataset's format directly mirrors HumanEval+ problem structure, making it the highest-ROI data source for benchmark performance.
5. 3-Layer Checkpoint Recovery Training was designed to survive Kaggle's 12-hour session limit via a 3-layer resume system: local checkpoint scan → HuggingFace Hub download → fresh start. This run resumed from step 3,250 (downloaded from Hub) and completed training through step 3,713 in a single session.
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torchmodel = AutoModelForCausalLM.from_pretrained("Siddh07ETH/Atlas-Coder-0.5B",torch_dtype=torch.float16,device_map="auto",)tokenizer = AutoTokenizer.from_pretrained("Siddh07ETH/Atlas-Coder-0.5B")messages = [{"role": "user", "content": "Write a Python function to find the longest common subsequence of two strings."}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device)with torch.no_grad():output = model.generate(**inputs,max_new_tokens=512,temperature=0.3,do_sample=True,top_p=0.9,repetition_penalty=1.1,)response = tokenizer.decode(output[0][inputs.input_ids.shape[1]:],skip_special_tokens=True)print(response)
python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigimport torchquant_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type="nf4",bnb_4bit_compute_dtype=torch.float16,)model = AutoModelForCausalLM.from_pretrained("Siddh07ETH/Atlas-Coder-0.5B",quantization_config=quant_config,device_map="auto",)tokenizer = AutoTokenizer.from_pretrained("Siddh07ETH/Atlas-Coder-0.5B")
GGUF quantizations for CPU inference are available at:
Runs at 40+ tokens/second on a laptop CPU using LM Studio, Ollama, or llama.cpp.
| Setting | Value | Reason |
|---|---|---|
temperature | 0.2–0.4 | Conservative — reduces hallucinations in code |
top_p | 0.9 | Focused vocabulary sampling |
repetition_penalty | 1.1 | Prevents repetitive patterns |
max_new_tokens | 256–512 | Sufficient for most coding tasks |
This model was fine-tuned from Qwen2.5-Coder-0.5B base, not instruct. The gap this training bridges:
| Model | HumanEval+ | Notes |
|---|---|---|
| Qwen2.5-Coder-0.5B Base | ~23.8% | Starting point before this fine-tune |
| Atlas-Coder-0.5B | TBD — running | This model |
| Qwen2.5-Coder-0.5B Instruct | ~57.3% | Alibaba's full alignment pipeline (ceiling benchmark) |
Benchmark results will be published shortly via EvalPlus.
| Model | Parameters | Description |
|---|---|---|
| Pluto-Genesis-0.6B | 596M | General reasoning, math, and code — Pluto AI's first release |
| Atlas-Coder-0.5B (this) | 494M | Coding-specialized, trained from base |
Siddharth N.R. (Siddhu) Final-year B.Tech — AI & Data Science Pluto AI Research
bibtex
@misc{atlascoder2026,author = {Siddharth N.R.},title = {Atlas-Coder-0.5B: A QLoRA-Trained Sub-1B Coding Model from Base},year = {2026},publisher = {HuggingFace},url = {https://huggingface.co/Siddh07ETH/Atlas-Coder-0.5B}}
Apache 2.0 — see LICENSE. Base model Qwen2.5-Coder-0.5B is also Apache 2.0.
| 35,192,832 / 350,312,320 (10.05%) |
| Loss Masking | Response-only (DataCollatorForCompletionOnlyLM) |
| Training Epochs | 3 |
| Total Steps | ~7,424 (resumed from checkpoint 3,250 → completed at 3,713) |
| Final Training Loss | 0.0294 |
| Precision | FP16 (forced — T4 sm_75 does not support BF16) |
| Optimizer | AdamW 8-bit (Paged) |
| Learning Rate | 1e-4 (cosine schedule) |
| Warmup Steps | max(150, 5% of total steps) |
| Effective Batch Size | 32 (2 × 16 grad accum) |
| Sequence Length | 1024 tokens |
| Hardware | Tesla T4 (16 GB VRAM) — Kaggle free tier |
| Training Time | 42h 44m 31s |
| Framework | Transformers 4.52.4 + PEFT 0.17.0 + TRL 0.19.1 |
| Chat Template | ChatML |
| 10,000 |
| Stack Overflow Q&A, debugging |
| 🧠 Algorithms | BAAI/TACO | 5,000 | Algorithmic reasoning |
| Total | 80,000 → 79,988 after decontamination |
| 80,000 |
| 12 |
| 79,988 |
do_sample | True | Required when temperature < 1.0 |
