GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
medelharchaoui
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
medelharchaoui
Model Tree
openai-community/gpt2-large
Input Modalities
Output Modalities
Supported Functionality
| Stage | WikiText-2 PPL | vs baseline |
|---|---|---|
| GPT-2 Large (baseline) | 27.56 | 1.00× |
| + ASVD rank=640 | 153.82 | 5.58× worse |
| + this LoRA adapter | 24.53 | 0.89× (11% better) |
Inference speedup retained: 1.6× (ASVD factored weights, not LoRA, do inference).
W (5120×1280) is replaced by a rank-640 factorisation U·Σ·Vᵀ, weighted by input activation statistics before SVD. This reduces matmul cost by 1.6×.A·B are added on top of frozen compressed weights, trained for 3 epochs on WikiText-2. Only 8.85M params (1.13% of model) are trainable.python
from transformers import GPT2LMHeadModel, GPT2Tokenizerfrom peft import PeftModel # Load compressed base (you must apply ASVD first — see repo for script)# OR load the full model and merge adapter weights normallybase = GPT2LMHeadModel.from_pretrained("gpt2-large")model = PeftModel.from_pretrained(base, "elharchaoui/gpt2-large-asvd640-lora")tokenizer = GPT2Tokenizer.from_pretrained("gpt2-large") inputs = tokenizer("The history of machine learning", return_tensors="pt")out = model.generate(**inputs, max_new_tokens=50)print(tokenizer.decode(out[0]))Note: This adapter was trained on top of ASVD-compressed weights. Loading it on an uncompressed GPT-2 Large will work (the adapter corrects for the compression error) but the inference speedup requires the factored weights. See the OptIAI repository for the full compression + inference pipeline.
gpt2-large (774M params)c_fc, c_proj layers (MLP + attention projection), 36 layersc_fc, c_projSalesforce/wikitext, wikitext-2-raw-v1), 9,343 train chunks × 256 tokens| Epoch | Time | PPL wiki |
|---|---|---|
| (post-ASVD, pre-LoRA) | 0 min | 153.82 |
| 1 | 18 min | 25.57 |
| 2 | 37 min | 24.70 |
| 3 | 55 min | 24.53 |
Recovery is fast: epoch 1 already crosses below the uncompressed baseline (27.56).
Part of the OptIAI research project investigating cheap inference-time approximations of LLM weight matrices via structured decomposition.
from transformers import GPT2LMHeadModel, GPT2Tokenizerfrom peft import PeftModel # Load compressed base (you must apply ASVD first — see repo for script)# OR load the full model and merge adapter weights normallybase = GPT2LMHeadModel.from_pretrained("gpt2-large")model = PeftModel.from_pretrained(base, "elharchaoui/gpt2-large-asvd640-lora")tokenizer = GPT2Tokenizer.from_pretrained("gpt2-large") inputs = tokenizer("The history of machine learning", return_tensors="pt")out = model.generate(**inputs, max_new_tokens=50)print(tokenizer.decode(out[0]))