GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
yennj12
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
yennj12
Model Tree
distilbert/distilgpt2
Input Modalities
Output Modalities
Supported Functionality
| Prompt | Base distilgpt2 | With this adapter |
|---|---|---|
Q: What is LoRA?\nA: | "LoRA is a very simple, simple, simple, simple…" | "Fact: LoRA freezes the base weights and trains two small matrices A and B." |
Q: What does a low-rank matrix do?\nA: (held out) | "It's a matrix that is a matrix that is a matrix…" | "Fact: low-rank sets the weights of the edges, q_proj and v_proj first." |
The first answer is memorised verbatim from the training set. The second
prompt was never trained on: the Fact: style transferred, but the content is
wrong — low-rank matrices do not "set the weights of the edges". That gap is
the honest lesson of an 8-example fine-tune. LoRA transferred style, not
knowledge.
python
from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained("distilgpt2")base = AutoModelForCausalLM.from_pretrained("distilgpt2")model = PeftModel.from_pretrained(base, "yennj12/distilgpt2-lora-fact").eval()inputs = tok("Q: What is LoRA?\nA:", return_tensors="pt")print(tok.decode(model.generate(**inputs, max_new_tokens=24,do_sample=False,pad_token_id=tok.eos_token_id)[0]))
Call model.merge_and_unload() to fold the adapter into the base weights for
zero inference overhead.
| Base model | distilgpt2 (82M params) |
| Trainable params | 147,456 (0.18%) |
| Adapter size | ~584 KB, vs 313 MB for the base |
| Config | r=8, lora_alpha=16 (scaling 2.0), lora_dropout=0.05 |
| Target modules | c_attn — GPT-2's fused q/k/v projection, in all 6 blocks |
| Optimiser | AdamW, lr , 60 full-batch epochs |
The learning rate is ~100x a typical full fine-tuning LR. That is normal for LoRA: you are training freshly initialised matrices, not nudging pretrained weights.
Everything about this model is a limitation. It was overfit on purpose to 8 sentences about LoRA, so it will state confident falsehoods on any prompt outside that set, and it inherits all of distilgpt2's biases underneath. Use it to learn how adapters work, not for generation.
Training code and a from-scratch (no-PEFT) reimplementation of the same arithmetic: see the repository this adapter was trained from. Follows the Fine-Tuning with LoRA & QLoRA lesson from AI Engineering from Scratch.
2e-3| Loss | 5.16 -> 0.51 |
| Hardware | Apple Silicon (MPS), ~7 s |
| Seed | torch.manual_seed(0) — reproducible |