shoron07

mistral-7b-dolly-qlora

README

License: apache-2.0

Training configuration

Table with columns: Setting, Value
Setting	Value
Base model	`mistralai/Mistral-7B-v0.3`
Training examples	11,850
Validation examples	1,447
Test examples	1,446
Epochs	1
Maximum sequence length	1,024
Quantization	4-bit NF4
Double quantization	Enabled
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Learning rate	2e-4
Optimizer	Paged AdamW 8-bit
Completion-only loss	Enabled
Trainable parameters	41,943,040
Trainable percentage	0.5754%

The adapter targeted the q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj linear layers.

Dataset preparation

Eight exact duplicates were removed. Repeated complete inputs were assigned only to training, producing zero instruction-plus-context overlap among the training, validation, and test splits.

Examples longer than 1,024 tokens were filtered instead of silently truncated.

Held-out test results

These results use all 1,446 untouched test examples.

Table with columns: Metric, Base Mistral, QLoRA adapted
Metric	Base Mistral	QLoRA adapted
Test loss	3.0392	2.5306
Perplexity	20.8891	12.5609
Mean token accuracy	0.6565	0.6912

Test-loss reduction: 16.74%
Perplexity reduction: 39.87%
Token-accuracy improvement: 3.4698 percentage points

Generation evaluation

Generation was evaluated on a reproducible balanced sample of 80 held-out examples, containing 10 examples from every Dolly category.

Table with columns: Metric, Base Mistral, QLoRA adapted
Metric	Base Mistral	QLoRA adapted
ROUGE-1 F1	0.1806	0.4392
ROUGE-2 F1	0.0690	0.2628
ROUGE-L F1	0.1380	0.3713
BERTScore F1	0.8188	0.8884

QLoRA achieved a higher BERTScore on 70/80 examples.

Loading the adapter

python
import torch

from peft import PeftModel
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig
)

base_model_id = "mistralai/Mistral-7B-v0.3"
adapter_id = "shoron07/mistral-7b-dolly-qlora"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    revision="caa1feb0e54d415e2df31207e5f4e273e33509b1",
    quantization_config=quantization_config,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    adapter_id
)

prompt = (
    "### Instruction:\n"
    "Explain why the sky appears blue.\n\n"
    "### Response:\n"
)

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=False
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True
    )
)

Limitations

The adapter was trained for one epoch on Dolly-15K.
Dataset answers can contain outdated or incorrect information.
ROUGE and BERTScore do not directly measure factual correctness.
Generation metrics used a balanced 80-example test sample.
The model has not undergone dedicated safety evaluation.
Generated information should be independently verified.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

shoron07

Model Tree

Base

mistralai/Mistral-7B-v0.3

Adapter

this model

Input Modalities

Text

Output Modalities