Sources
- Base model:
Qwen/Qwen2.5-0.5B-Instruct at commit 7ae557604adf67be50417f59c2c2f167def9a775
- Tokenizer:
Qwen/Qwen2.5-0.5B-Instruct at commit 7ae557604adf67be50417f59c2c2f167def9a775
- Dataset:
Intel/orca_dpo_pairs at commit 624952e3f420ae18d88b31977ee2ea436c833abb
- Project code: https://github.com/praveenraj9623-sketch/align-lite-dpo-lab
Method
This adapter uses QLoRA fine-tuning and DPO preference alignment. It is not PPO-RLHF, does not include a separate reward model, and does not merge or publish full base-model weights.
Hyperparameters
LoRA
{
"alpha": 32,
"dropout": 0.05,
"rank": 16,
"target_modules": [
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj"
]
}
DPO
{
"beta": 0.1,
"bf16": false,
"epochs": 1,
"eval_steps": 100,
"eval_strategy": "steps",
"fp16": true,
"gradient_accumulation_steps": 8,
"gradient_checkpointing": true,
"learning_rate": 1e-05,
"logging_steps": 10,
"max_length": 1024,
"per_device_eval_batch_size": 1,
"per_device_train_batch_size": 1,
"save_steps": 100,
"save_strategy": "steps",
"save_total_limit": 2
}
Generation
{
"do_sample": false,
"max_new_tokens": 128,
"temperature": 0.0
}
Adapter Artifact Size
Total adapter artifact size: 17642272 bytes (16.824982 MB)
ADAPTER_EXPORT_METADATA.json: 359 bytes (0.000342 MB)
adapter_config.json: 1105 bytes (0.001054 MB)
adapter_model.safetensors: 17640808 bytes (16.823586 MB)
Limitations
- This is a 0.5B-parameter base model with an adapter-only alignment experiment.
- The data is public preference data from
Intel/orca_dpo_pairs, not feedback collected by this project.
- This adapter should be evaluated against the locked held-out split and exact source revisions recorded in
adapter_manifest.json.
- DPO preference alignment is not PPO-RLHF and should not be described as reward-model training.