Model Details
Table with columns: Field, Value| Field | Value |
|---|
| Base model | Qwen/Qwen3-4B |
| Method | DPO + QLoRA |
| Checkpoint | step 521 (final) |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| DPO beta | 0.1 |
| DPO epochs | 2 |
| Learning rate | 5e-7 |
| PEFT version | 0.19.1 |
LoRA target modules
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training data
Preference pairs from the code_vuln_dpo dataset (chosen vs. rejected code responses for vulnerability-related prompts).
Final training metrics (step 521)
Table with columns: Metric, Value| Metric | Value |
|---|
| DPO loss | 0.250 |
| Preference accuracy | 93.8% |
| Reward margin | 1.64 |
| Chosen reward | 2.50 |
| Rejected reward | 0.86 |
Usage
Load the base model and apply this adapter with PEFT:
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE_MODEL = "Qwen/Qwen3-4B"
ADAPTER = "Legendarylibrary/Qwen3-4B-CVE-DPO-LoRA"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()
messages = [
{"role": "user", "content": "Review this code for security vulnerabilities:\n\n```python\ndef login(user, pwd):\n query = f\"SELECT * FROM users WHERE name='{user}' AND pass='{pwd}'\"\n return db.execute(query)\n```"}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Merge adapter into base weights (optional)
merged = model.merge_and_unload()
merged.save_pretrained("./qwen3-4b-cve-dpo-merged")
tokenizer.save_pretrained("./qwen3-4b-cve-dpo-merged")
Limitations
- Specialized for code vulnerability preference alignment; general capabilities inherit from the base model.
- Always validate security recommendations with static analysis, testing, and human review.
- Not a substitute for professional security auditing.
License
This adapter is released under the Apache 2.0 license, consistent with the Qwen3 base model license. Use of the base model is subject to the Qwen3 license terms.
Citation
@misc{qwen3-4b-cve-dpo-lora,
author = {Legendarylibrary},
title = {Qwen3-4B CVE DPO LoRA},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Legendarylibrary/Qwen3-4B-CVE-DPO-LoRA}}
}