Phase-Technologies

qwen2.5-3b-claude-distilled-reasoning-dpo

README

License: apache-2.0

Overview

qwen2.5-3b-claude-distilled-reasoning-dpo is a post-trained, reasoning-specialized 3.0B parameter causal language model.

This model represents a two-stage post-training alignment pipeline built on top of Qwen/Qwen2.5-3B-Instruct:

Supervised Fine-Tuning (SFT): Fine-tuned on high-quality internal reasoning monologue traces distilled from Claude 3.5 Sonnet, imparting deep step-by-step mathematical, logical, and code-synthesis reasoning behavior.
Direct Preference Optimization (DPO): Aligned using DPOTrainer on preference pairs (argilla/ultrafeedback-binarized-preferences-cleaned). This step eliminates scientific hallucinations (e.g., density vs. thermal conductivity), suppresses infinite token repetition loops, and anchors physical explanations to first-principles facts.

Model Capabilities & Highlights

Distilled Chain-of-Thought (CoT): Thinks through mathematical equations, coding challenges, and logic puzzles step-by-step prior to executing answers.
Factually Grounded: High performance on graduate/research-level physics and mathematics questions, reducing hallucination tendencies present in basic SFT models.
ChatML Ready: Fully compatible with standard Qwen2.5 ChatML chat templates and system prompt instructions.
Low Memory Footprint: Runs comfortably in FP16/SDPA on a single consumer GPU (e.g., NVIDIA T4 / RTX 3060) requiring ~6GB VRAM.

Alignment & Evaluation Pipeline

Table with columns: Feature, Base Model (Qwen2.5-3B-Instruct), SFT Stage (...-reasoning), DPO Stage (...-reasoning-dpo)
Feature	Base Model (`Qwen2.5-3B-Instruct`)	SFT Stage (`...-reasoning`)	DPO Stage (`...-reasoning-dpo`)
Reasoning Engine	Static response generation	Claude CoT monologue traces	Refined CoT monologue traces
Physics/Math Accuracy	Standard textbook baseline	Prone to reasoning hallucinations	First-principles verified
Degeneracy / Loops	Standard EOS handling

Integrated Real-Time Streaming & Safe Inference Script

The code below provides a production-ready, bulletproof inference script. It includes:

Token-by-Token Streaming using TextIteratorStreamer.
Dynamic Temperature Scaling (lowers temperature for simple greetings to prevent creative rambling; elevates it for math/reasoning tasks).
System Prompt Injection to prevent tool/interface hallucinations.
Custom Stopping Criteria to cut off any potential ASCII symbol artifacts or trailing conversational chatter.

python
import os
import sys
from threading import Thread
import torch
import time
from transformers import (
    AutoTokenizer, 
    AutoModelForCausalLM, 
    TextIteratorStreamer,
    StoppingCriteria,
    StoppingCriteriaList
)

# =========================================================================
# 1. CONFIGURATION & MODEL LOADING
# =========================================================================
REPO_ID = "Phase-Technologies/qwen2.5-3b-claude-distilled-reasoning-dpo"

print(f"[*] Hardware Status: CUDA Available: {torch.cuda.is_available()}")

tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForCausalLM.from_pretrained(
    REPO_ID,
    torch_dtype=torch.float16,
    device_map="auto",
    attn_implementation="sdpa"
)

# =========================================================================
# 2. ENHANCED INFERENCE ENGINE
# =========================================================================
def analyze_inference(prompt):
    messages = [
        {"role": "system", "content": "You are a reasoning assistant. Solve the problem step-by-step and provide a final answer in a box."},
        {"role": "user", "content": prompt}
    ]
    formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device)
    
    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
    
    # Increased repetition penalty to 1.3 to stop 'Implication:' loops
    # Added stop_strings for common hallucination patterns
    gen_kwargs = dict(
        **inputs,
        streamer=streamer,
        max_new_tokens=400,
        do_sample=True,
        temperature=0.4,
        top_p=0.9,
        repetition_penalty=1.3,
        stop_strings=["Implication:", "<|im_end|>", "###"],
        tokenizer=tokenizer,
        pad_token_id=tokenizer.eos_token_id
    )

    print(f"\n--- TESTING IMPROVED PARAMETERS ---")
    start_time = time.time()
    thread = Thread(target=model.generate, kwargs=gen_kwargs)
    thread.start()
    
    generated_text = ""
    for new_text in streamer:
        print(new_text, end="", flush=True)
        generated_text += new_text
    
    duration = time.time() - start_time
    print(f"\n\n[Metric] Speed: {len(tokenizer.encode(generated_text))/duration:.2f} tokens/sec")

analyze_inference("Sally has 3 brothers. Each of her brothers has 2 sisters. How many sisters does Sally have?")

Technical Specifications

Architecture: Causal LM (Qwen2.5 architecture)
Parameters: ~3.09 Billion
Context Window: 32,768 tokens (Recommended max inference: 2,048 tokens)
Precision: bfloat16 / float16
License: Apache-2.0

Citation & Acknowledgments

Base Model: Alibaba Qwen Team (Qwen/Qwen2.5-3B-Instruct)
Preference Dataset: Argilla (argilla/ultrafeedback-binarized-preferences-cleaned)
Distillation Framework: Fine-tuned and post-trained using Hugging Face TRL (DPOTrainer), PEFT, and Transformers.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Phase-Technologies

Model Tree

Base

Phase-Technologies/qwen2.5-3b-claude-distilled-reasoning

Fine-tuned

this model

Input Modalities

Text

Output Modalities