GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
trustkrem
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
trustkrem
Model Tree
mistralai/Mistral-7B-v0.1
Input Modalities
Output Modalities
Supported Functionality
| Model | Test perplexity ↓ |
|---|---|
| Mistral 7B (base, 4-bit) | 9.09 |
| Mistral 7B + this adapter | 6.83 |
Measured as mean per-chunk perplexity on 97 held-out text chunks not used in training.
Prompt: "I'm afraid I've created a " (greedy decoding)
I'm afraid I've created a monster in putting life and sensation into dead matter. [...] Let me not be confounded with the wretch to whom I gave existence. I shall escape him, and find a place for repentance. Adieu!
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel base_id = "mistralai/Mistral-7B-v0.1"quant_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16,) base = AutoModelForCausalLM.from_pretrained(base_id, quantization_config=quant_config, device_map="auto")model = PeftModel.from_pretrained(base, "trustkrem/frankenstein-qlora-mistral7b")tokenizer = AutoTokenizer.from_pretrained(base_id) inputs = tokenizer("I'm afraid I've created a ", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=100)print(tokenizer.decode(outputs[0], skip_special_tokens=True))Requires an NVIDIA GPU (bitsandbytes 4-bit). A free Colab T4 is enough. For more varied output, add do_sample=True, temperature=0.8.
Data: 481 text chunks of Frankenstein from Project Gutenberg (public domain), split 384 train / 97 test.
| Quantization | |
|---|---|
| Type | 4-bit NF4, double quantization |
| Compute dtype | bfloat16 |
| LoRA | |
|---|---|
Rank r / lora_alpha | 32 / 64 |
| Dropout | 0.05 |
| Target modules | q_proj, v_proj |
| Task type | CAUSAL_LM |
| Training | |
|---|---|
| Epochs / steps | 2 / 384 |
| Batch size | 2 |
| Learning rate | 2e-5, 2 warmup steps |
| Optimizer | paged_adamw_8bit |
| Precision | fp16 mixed precision |
| Hardware / time | Google Colab T4 (16 GB) / ~16.5 min |
| Training loss | ~1.89 (first 20 steps) → ~1.66 (last 20 steps) |
Built as a portfolio project for Codecademy's Finetuning Transformer Models course. Base model by Mistral AI (Apache 2.0). Source text: Mary Shelley, Frankenstein (1818), via Project Gutenberg.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel base_id = "mistralai/Mistral-7B-v0.1"quant_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16,) base = AutoModelForCausalLM.from_pretrained(base_id, quantization_config=quant_config, device_map="auto")model = PeftModel.from_pretrained(base, "trustkrem/frankenstein-qlora-mistral7b")tokenizer = AutoTokenizer.from_pretrained(base_id) inputs = tokenizer("I'm afraid I've created a ", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=100)print(tokenizer.decode(outputs[0], skip_special_tokens=True))