GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
emonHasan01
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
emonHasan01
Model Tree
Qwen/Qwen2.5-3B-Instruct
Input Modalities
Output Modalities
Supported Functionality
LoRA freezes the base model and trains small adapter matrices. QLoRA additionally stores the frozen base in 4 bits, while gradients still flow through it in fp16. Three pieces:
| Piece | Role |
|---|---|
| NF4 | a 4-bit datatype shaped for normally-distributed weights rather than uniform buckets |
| Double quantization | quantizes the quantization constants, saving roughly 0.4 bits per parameter |
| Paged 8-bit optimizer | pages optimizer state to CPU RAM on a spike instead of crashing |
Greedy decoding, 250 held-out DialogSum test dialogues, ROUGE with stemming.
| ROUGE-1 | ROUGE-2 | ROUGE-L | mean words | |
|---|---|---|---|---|
| lead-3 (extractive) | 26.72 | 5.72 | 20.65 | 37.7 |
Qwen/Qwen2.5-3B-Instruct zero-shot | 28.66 | 7.34 | 21.19 | 45.5 |
| + this adapter | 44.69 | 18.15 | 36.62 | 23.0 |
Both baselines are here deliberately. Lead-3 just copies the first three turns, and it is the standard check that a summarizer is doing more than truncating. The zero-shot base model is the harder and more honest comparison: Qwen2.5-3B-Instruct can already summarize, so the only question worth asking is whether the adapter improves on what it did for free.
Reference summaries average 17 words, against 45.5 for the zero-shot base and 23.0 for the adapter. A real part of the ROUGE gain is the adapter learning that length and register rather than comprehending better — worth holding in mind before reading too much into the numbers.
| Reference | Base, zero-shot | With adapter |
|---|---|---|
| Ms. Dawson helps #Person1# to write a memo to inform every employee that they ha | #Person1# asked #Person2# to take a dictation for an intra-office memorandum. Th | #Person1# asks Ms. Dawson to take a dictation for him. #Person1# tells all emplo |
| In order to prevent employees from wasting time on Instant Message programs, #Pe | #Person1# asked #Person2# to take a dictation for an intra-office memorandum. Th | #Person1# asks Ms. Dawson to take a dictation for him. #Person1# tells all emplo |
| Ms. Dawson takes a dictation for #Person1# about prohibiting the use of Instant | #Person1# asked #Person2# to take a dictation for an intra-office memorandum. Th | #Person1# asks Ms. Dawson to take a dictation for him. #Person1# tells all emplo |
| #Person2# arrives late because of traffic jam. #Person1# persuades #Person2# to | #Person2# explains their experience with traffic and the reasons why they might | #Person2# tells #Person1# that #Person2# got stuck in traffic again and suggests |
The adapter needs its base model, loaded with the same 4-bit configuration:
python
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfigfrom peft import PeftModel bnb = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16, # bfloat16 on Ampere or newer) tok = AutoTokenizer.from_pretrained("emonHasan01/qwen2.5-3b-dialogsum-qlora")base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", quantization_config=bnb, device_map={"": 0})model = PeftModel.from_pretrained(base, "emonHasan01/qwen2.5-3b-dialogsum-qlora") messages = [ {"role": "system", "content": "You are a helpful assistant that summarizes conversations."}, {"role": "user", "content": "Summarize the following dialogue in two or three sentences.\n\n" + dialogue},]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = model.generate(**inputs, max_new_tokens=96, do_sample=False)print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))The prompt format matters. The adapter was trained on Qwen2.5's own chat template with
that exact system message and instruction; a different wrapper will degrade output. The
exact strings, including a rendered example, are in
prompt_format.json.
python
from peft import PeftModelfrom transformers import AutoModelForCausalLM base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", torch_dtype="float16", device_map="auto")merged = PeftModel.from_pretrained(base, "emonHasan01/qwen2.5-3b-dialogsum-qlora").merge_and_unload()merged.save_pretrained("merged-dialogsum")Merging requires loading the base in fp16, not 4-bit — adapters cannot be folded into quantized weights. The merged model is full size.
| Base model | Qwen/Qwen2.5-3B-Instruct (3.1B parameters) |
| Quantization | 4-bit NF4, double quantization, compute dtype float16 |
| Trainable | 29,933,568 parameters (0.96% of the model) |
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | all seven projections: q,k,v,o,gate,up,down |
| Learning rate | 0.0002, cosine schedule, 3% warmup |
| Batch | 2 × 8 accumulation = 16 effective |
Loss on the completion only. Prompt tokens are set to -100. Labelling every token
instead trains the model to generate the dialogue as well as the summary, spending most of
the gradient on text that is always supplied at inference.
max_grad_norm=0.3. fp16 training over a quantized base overflows readily. Without
clipping, the loss reaches NaN partway through and nothing else warns you.
Targeting all seven projections. With a quantized base, adapter coverage matters more
than rank — that is the QLoRA paper's own ablation, and q_proj/v_proj alone
underperforms here.
T4 is Turing (compute capability 7.5): no bfloat16, no Flash Attention 2. Compute dtype
must be float16. Recipes written for A100 or newer frequently assume bfloat16 and will
fail or silently degrade on this hardware.
#Person1# style markers. Meeting transcripts, support tickets, noisy ASR output and
multi-party conversation are all out of distribution.Do not use this where a wrong summary would matter without a human reading the output.
The first version of this card reported the base model as 1.7B parameters and estimated
3.4 GB for fp16 weights. Both came from calling numel() on 4-bit tensors, which
undercounts by about half because bitsandbytes packs two 4-bit values into each uint8.
Corrected to 3.1B parameters and 6.2 GB. The measured VRAM figures were
never affected, and the error understated how much memory QLoRA saves here.
One Colab notebook on a Tesla T4 produced the training, both baselines, all three charts and
this card. Raw numbers in results.json.
bibtex
@article{dettmers2023qlora, title = {QLoRA: Efficient Finetuning of Quantized LLMs}, author = {Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke}, journal = {NeurIPS}, year = {2023}}@inproceedings{chen2021dialogsum, title = {DialogSum: A Real-Life Scenario Dialogue Summarization Dataset}, author = {Chen, Yulong and Liu, Yang and Chen, Liang and Zhang, Yue}, booktitle = {Findings of ACL}, year = {2021}}| Epochs | 3 over 3,000 examples |
| Max length | 768 tokens |
| Optimizer | paged_adamw_8bit, max_grad_norm=0.3 |
| Hardware | Tesla T4 (free Colab tier) |
| Training time | ~104 minutes |