tensorov

qwen2.5-coder-7b-dpo

README

License: apache-2.0

Model Details

Base model: Qwen2.5-Coder-7B-Instruct (4-bit quantized via unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit)
Architecture: Qwen2ForCausalLM, 7B parameters (80.7M LoRA trainable)
Adapter type: LoRA (rank=32, alpha=32, dropout=0)
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training stages: SFT → DPO
License: Apache 2.0
Language: English

Training Data

The Combined Reasoning Distill dataset consists of 1,335,511 records aggregated from 41 source datasets covering reasoning traces from:

Table with columns: Source Family, Models
Source Family	Models
Anthropic	Claude Opus 4.5/4.6/4.7, Sonnet 4.5/4.6, Haiku 4.5
OpenAI	GPT 5.1/5.2
Kimi	K2, K2.5, K2.6
GLM	4.6, 4.7, 5.1
Google	Gemini 3 Pro Preview
MiniMax	M2.1
xAI	Grok Code Fast 1

The data covers math, code, science, logic, and general reasoning. Thinking traces are embedded in <think>...</think> tags where present.

Training splits

Table with columns: Split, SFT stage, DPO stage
Split	SFT stage	DPO stage
Train	439,942 records	420,111 preference pairs
Eval	13,607 records	12,996 preference pairs

Training Procedure

Stage 1 — Supervised Fine-Tuning (SFT)

The model was first fine-tuned on the best-ranked answer per question group (think tags removed).

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	2e-5
Schedule	Linear with 5% cosine warmup
Batch size	16 (gradient accumulation)
Epochs	1
Max seq length	4,096 tokens
Optimizer	AdamW (β₁=0.9, β₂=0.999)
Weight decay	0.01
LoRA rank	64 (SFT stage)
LoRA alpha	128

Stage 2 — Direct Preference Optimization (DPO)

The SFT checkpoint was used as the base for DPO training on preference pairs (chosen/rejected from answer rankings).

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	1e-4
Schedule	Linear with 10% cosine warmup
Batch size	16 (effective, accum=16 × batch=1)
Epochs	1
Max seq length	1,024 tokens
β (DPO temperature)	0.1
Optimizer	AdamW 8-bit
Weight decay	0.01
LoRA rank	32 (DPO stage)

Hardware

GPUs: 2× Tesla V100-SXM2 16 GB (no NVLink)
CUDA: 13.2
PyTorch: 2.10.0+cu128
Framework: Unsloth (SFT) + TRL DPOTrainer (DPO)
Precision: fp16 mixed precision (V100 has no bf16 support)
Memory: ~9.6 GiB / ~7.6 GiB (GPU 0 / GPU 1, model split across both via device_map="auto")

Environmental Impact

Hardware: 2× Tesla V100 16 GB
Training time: ~105 hours (DPO) + ~30 hours (SFT) = ~135 hours total
Power: ~300W per GPU × 2 GPUs × 135h ≈ 81 kWh
CO₂ equivalent: ~35–55 kg CO₂ (varies by grid mix)

Evaluation Results

Table with columns: Benchmark, Score, Notes
Benchmark	Score	Notes
GSM8K (CoT, 0-shot)	30.0% (15/50)	Unsloth-based eval
MMLU (0-shot, 100 samples)	48.0% (48/100)	PEFT eval
HumanEval (pass@50)	100.0% (50/50)	PEFT eval, name-presence heuristic

Notes on Evaluation

GSM8K was evaluated with chain-of-thought prompting (0-shot).
MMLU was evaluated 0-shot on 100 samples covering STEM, humanities, and social sciences.
HumanEval pass@50 uses a name-presence heuristic (checks if the expected function name appears in generated code); this likely overestimates true pass@k. Exact match or test-case-based evaluation would yield a lower score.
Benchmarks were run using PEFT (for MMLU and HumanEval) due to a torch.compile incompatibility in the Unsloth inference path. GSM8K was evaluated via Unsloth with a patched environment.

How to Use

Load the adapter

python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Base model (4-bit quantized)
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit",
    device_map="auto",
    torch_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit")

# Load DPO adapter
model = PeftModel.from_pretrained(base_model, "tensorov/qwen2.5-coder-7b-dpo")
model.eval()

Inference example

python
prompt = "Write a Python function to check if a string is a palindrome, ignoring case and spaces."
messages = [
    {"role": "user", "content": prompt}
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        inputs, max_new_tokens=512, temperature=0.7, top_p=0.9, do_sample=True
    )
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

Loading with Unsloth (if available)

python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "tensorov/qwen2.5-coder-7b-dpo",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Limitations

Benchmark coverage is limited. Only GSM8K (50 samples), MMLU (100 samples), and HumanEval were evaluated. Broader evaluation (MATH, MBPP, BigBench, etc.) was not performed.
HumanEval 100% is inflated. The evaluation used a name-presence heuristic, not actual test-case execution.
DPO training used seq=1024. Longer-context reasoning (>2K tokens) was not explicitly trained, though the base model supports up to 131K tokens.
Single-epoch DPO. Training was capped at 1 epoch based on the recommendation in the DPO literature; additional epochs could potentially improve alignment at the cost of overfitting risk.
4-bit quantization. The base model uses 4-bit NormalFloat quantization, which slightly degrades output quality compared to fp16 inference.
English only. The model was trained exclusively on English data.

Citation

If you use this adapter, please cite the base model and the training dataset:

bibtex
@article{qwen2.5-coder,
  title={Qwen2.5-Coder Technical Report},
  author={Qwen Team},
  journal={arXiv preprint},
  year={2024}
}

@misc{combined-reasoning-distill,
  title={Combined Reasoning Distill},
  author={Tensorov},
  year={2025},
  url={https://huggingface.co/tensorov/qwen2.5-coder-7b-dpo}
}

Repository Files

Table with columns: File, Size, Description
File	Size	Description
`adapter_model.safetensors`	323 MB	LoRA adapter weights
`adapter_config.json`	1.2 KB	LoRA configuration
`tokenizer.json`	11.4 MB	Tokenizer
`tokenizer_config.json`	4.6 KB	Tokenizer config

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

tensorov

Model Tree

Base

unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Container

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Base model: Qwen2.5-Coder-7B-Instruct (4-bit quantized via unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit)
Architecture: Qwen2ForCausalLM, 7B parameters (80.7M LoRA trainable)
Adapter type: LoRA (rank=32, alpha=32, dropout=0)
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training stages: SFT → DPO
License: Apache 2.0
Language: English

Training Data

The Combined Reasoning Distill dataset consists of 1,335,511 records aggregated from 41 source datasets covering reasoning traces from:

Table with columns: Source Family, Models
Source Family	Models
Anthropic	Claude Opus 4.5/4.6/4.7, Sonnet 4.5/4.6, Haiku 4.5
OpenAI	GPT 5.1/5.2
Kimi	K2, K2.5, K2.6
GLM	4.6, 4.7, 5.1
Google	Gemini 3 Pro Preview
MiniMax	M2.1
xAI	Grok Code Fast 1

The data covers math, code, science, logic, and general reasoning. Thinking traces are embedded in <think>...</think> tags where present.

Training splits

Table with columns: Split, SFT stage, DPO stage
Split	SFT stage	DPO stage
Train	439,942 records	420,111 preference pairs
Eval	13,607 records	12,996 preference pairs

Training Procedure

Stage 1 — Supervised Fine-Tuning (SFT)

The model was first fine-tuned on the best-ranked answer per question group (think tags removed).

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	2e-5
Schedule	Linear with 5% cosine warmup
Batch size	16 (gradient accumulation)
Epochs	1
Max seq length	4,096 tokens
Optimizer	AdamW (β₁=0.9, β₂=0.999)
Weight decay	0.01
LoRA rank	64 (SFT stage)
LoRA alpha	128

Stage 2 — Direct Preference Optimization (DPO)

The SFT checkpoint was used as the base for DPO training on preference pairs (chosen/rejected from answer rankings).

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	1e-4
Schedule	Linear with 10% cosine warmup
Batch size	16 (effective, accum=16 × batch=1)
Epochs	1
Max seq length	1,024 tokens
β (DPO temperature)	0.1
Optimizer	AdamW 8-bit
Weight decay	0.01
LoRA rank	32 (DPO stage)

Hardware

GPUs: 2× Tesla V100-SXM2 16 GB (no NVLink)
CUDA: 13.2
PyTorch: 2.10.0+cu128
Framework: Unsloth (SFT) + TRL DPOTrainer (DPO)
Precision: fp16 mixed precision (V100 has no bf16 support)
Memory: ~9.6 GiB / ~7.6 GiB (GPU 0 / GPU 1, model split across both via device_map="auto")

Environmental Impact

Hardware: 2× Tesla V100 16 GB
Training time: ~105 hours (DPO) + ~30 hours (SFT) = ~135 hours total
Power: ~300W per GPU × 2 GPUs × 135h ≈ 81 kWh
CO₂ equivalent: ~35–55 kg CO₂ (varies by grid mix)

Evaluation Results

Table with columns: Benchmark, Score, Notes
Benchmark	Score	Notes
GSM8K (CoT, 0-shot)	30.0% (15/50)	Unsloth-based eval
MMLU (0-shot, 100 samples)	48.0% (48/100)	PEFT eval
HumanEval (pass@50)	100.0% (50/50)	PEFT eval, name-presence heuristic

Notes on Evaluation

GSM8K was evaluated with chain-of-thought prompting (0-shot).
MMLU was evaluated 0-shot on 100 samples covering STEM, humanities, and social sciences.
HumanEval pass@50 uses a name-presence heuristic (checks if the expected function name appears in generated code); this likely overestimates true pass@k. Exact match or test-case-based evaluation would yield a lower score.
Benchmarks were run using PEFT (for MMLU and HumanEval) due to a torch.compile incompatibility in the Unsloth inference path. GSM8K was evaluated via Unsloth with a patched environment.

How to Use

Load the adapter

python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Base model (4-bit quantized)
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit",
    device_map="auto",
    torch_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit")

# Load DPO adapter
model = PeftModel.from_pretrained(base_model, "tensorov/qwen2.5-coder-7b-dpo")
model.eval()

Inference example

python
prompt = "Write a Python function to check if a string is a palindrome, ignoring case and spaces."
messages = [
    {"role": "user", "content": prompt}
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        inputs, max_new_tokens=512, temperature=0.7, top_p=0.9, do_sample=True
    )
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

Loading with Unsloth (if available)

python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "tensorov/qwen2.5-coder-7b-dpo",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Limitations

Benchmark coverage is limited. Only GSM8K (50 samples), MMLU (100 samples), and HumanEval were evaluated. Broader evaluation (MATH, MBPP, BigBench, etc.) was not performed.
HumanEval 100% is inflated. The evaluation used a name-presence heuristic, not actual test-case execution.
DPO training used seq=1024. Longer-context reasoning (>2K tokens) was not explicitly trained, though the base model supports up to 131K tokens.
Single-epoch DPO. Training was capped at 1 epoch based on the recommendation in the DPO literature; additional epochs could potentially improve alignment at the cost of overfitting risk.
4-bit quantization. The base model uses 4-bit NormalFloat quantization, which slightly degrades output quality compared to fp16 inference.
English only. The model was trained exclusively on English data.

Citation

If you use this adapter, please cite the base model and the training dataset:

bibtex
@article{qwen2.5-coder,
  title={Qwen2.5-Coder Technical Report},
  author={Qwen Team},
  journal={arXiv preprint},
  year={2024}
}

@misc{combined-reasoning-distill,
  title={Combined Reasoning Distill},
  author={Tensorov},
  year={2025},
  url={https://huggingface.co/tensorov/qwen2.5-coder-7b-dpo}
}

Repository Files

Table with columns: File, Size, Description
File	Size	Description
`adapter_model.safetensors`	323 MB	LoRA adapter weights
`adapter_config.json`	1.2 KB	LoRA configuration
`tokenizer.json`	11.4 MB	Tokenizer
`tokenizer_config.json`	4.6 KB	Tokenizer config