GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
peterrs23
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
peterrs23
Model Tree
Qwen/Qwen2.5-3B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This LoRA adapter enhances Qwen/Qwen2.5-3B-Instruct with domain-specific knowledge of:
| Parameter | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-3B-Instruct |
| Quantization | 4-bit NF4 (bitsandbytes) |
| PEFT Method | QLoRA (r=8, α=16, dropout=0.05, bias=none) |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Dataset Size | 253 ChatML Q&A examples (202 train / 51 validation, zero data leakage) |
| Training Hyperparameters | Epochs = 2, Batch Size = 1, Grad Accumulation = 8, Learning Rate = 2e-4, Max Length = 1024 |
| Hardware Used | NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM) |
| Peak VRAM Allocated | 2.61 GB / 6.00 GB |
| Training Duration | 596.07 seconds (9.93 minutes) |
| Final Training Loss | 1.2388 |
| Final Validation Loss | 0.6851 |
| Benchmark Generation Test | PASS |
python
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigMODEL_ID = "Qwen/Qwen2.5-3B-Instruct"ADAPTER_ID = "peterrs23/audit-v-qwen-3b"# Configure 4-bit NF4 Quantizationbnb_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type="nf4",bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=torch.float16,)# Load Tokenizer & Base Modeltokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, trust_remote_code=True)base_model = AutoModelForCausalLM.from_pretrained(MODEL_ID,quantization_config=bnb_config,device_map="auto",trust_remote_code=True,)# Load Fine-Tuned Audit V LoRA Adaptermodel = PeftModel.from_pretrained(base_model, ADAPTER_ID)model.eval()# Sample Querymessages = [{"role": "system", "content": "You are Audit V, an AI financial and tax intelligence assistant."},{"role": "user", "content": "What is the maximum deduction allowed under Section 80C?"}]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(prompt, return_tensors="pt").to("cuda")with torch.no_grad():outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)print(response)
Qwen/Qwen2.5-3B-Instruct.