maghrane

Qwen-0.8B-unsloth

Deploy Dedicated

README

License: apache-2.0

📌 Model Details

Developed by: maghrane
Base Model: Qwen/Qwen3.5-0.8B
Model Type: Causal Language Model (Decoder-Only)
Fine-Tuning Technique: LoRA (PEFT) + QLoRA (4-bit NF4)
Dataset: maghrane/unsloth-qwen
Prompt Format: <s>[INST] {prompt} [/INST] {response} </s>
License: Apache-2.0

⚙️ Training Hyperparameters & Configuration

🔹 LoRA Configuration (PEFT)

Table with columns: Parameter, Value
Parameter	Value
LoRA Rank ( $r$ )	`64`
LoRA Alpha ( $a l p ha$ )	`16`
LoRA Dropout	`0.1`
Bias	`none`
Target Modules	`q_proj`, `k_proj`, , , , , , ,

🔹 Quantization & Precision (BitsAndBytes)

Table with columns: Parameter, Value
Parameter	Value
Quantization	4-bit (`nf4`)
Compute Dtype	`float16`
Double Quantization	`False`

🔹 SFT Training Arguments

Table with columns: Parameter, Value
Parameter	Value
Optimizer	`paged_adamw_32bit`
Learning Rate	`2e-4`
Learning Rate Scheduler	`constant`
Weight Decay	`0.001`
Warmup Ratio	`0.03`
Epochs

💻 How to Use & Run Inference

python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# 1. Disable cuDNN algorithm search (resolves Qwen 3.5 1D convolution kernel lookup issue)
torch.backends.cudnn.enabled = False

base_model_name = "Qwen/Qwen3.5-0.8B"
adapter_model_name = "maghrane/Qwen-0.8B-unsloth"

# 2. Load Base Model & Adapter
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, adapter_model_name)

# 3. Re-enable KV Cache & set to evaluation mode
model.config.use_cache = True
model.eval()

# 4. Generate Response
prompt = "What is Unsloth and what can it do?"
formatted_prompt = f"<s>[INST] {prompt} [/INST]"
inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.15,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id
    )

generated_text = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(generated_text)

📊 Results & Sample Inference

Input Prompt:

<s>[INST] What is Unsloth and what can it do? [/INST]

Model Output:

Unsloth lets you run and train AI models on your own local hardware via an open-source UI. It supports inference and training for 500+ models across text, audio, embedding, vision, and more. Key features include:

Inference: Search, download, and run models (GGUFs, LoRA adapters, safetensors), self-healing tool calling, auto parameter tuning, model export, and side-by-side output comparison.

Training: ~2x faster training with ~70% less VRAM, full/pre-training, 4-bit/16-bit/FP8 support, auto-dataset creation from PDF/CSV/DOCX, live observability, and efficient reinforcement learning (RL).

📜 Disclaimer

This model is fine-tuned for educational and documentation guidance purposes regarding local LLM fine-tuning and inference.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

maghrane

Model Tree

Base

Qwen/Qwen3.5-0.8B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality