GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Falcot922
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Falcot922
Model Tree
TinyLlama/TinyLlama-1.1B-Chat-v1.0
Input Modalities
Output Modalities
Supported Functionality
1,000 IMDB test reviews (shuffled with seed 42, first 400 characters), answer read from the class-token logits.
| Model | Accuracy (95% Wilson CI) | Reviews / s (T4) | Peak VRAM |
|---|---|---|---|
| Teacher: Mistral 7B NF4 + QLoRA | 91.6% (89.7 to 93.2) | 0.87 | 4.1 GiB |
| TinyLlama, no adapter | 65.4% (62.4 to 68.3) | 24.4 | 2.1 GiB |
| TinyLlama, hard labels | 89.7% (87.7 to 91.4) | 24.9 | 2.2 GiB |
| TinyLlama, distilled | 91.3% (89.4 to 92.9) | 24.9 | 2.2 GiB |
This adapter: 91.3% (89.4 to 92.9). Distillation beats hard labels by 1.6 points (paired test, p = 0.033); an earlier run of the same code gave 1.1 points (p = 0.14), so the gain is small. The distilled student is statistically indistinguishable from the teacher (p = 0.81) and 29 times faster.
TinyLlama/TinyLlama-1.1B-Chat-v1.0 in fp16python
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto")model = PeftModel.from_pretrained(model, "Falcot922/tinyllama-sentiment-distilled").eval() prompt = ("Analyze the sentiment of this movie review and respond with exactly one word: " "positive or negative.\n\nReview: {text}\nSentiment:")text = prompt.format(text="A wonderful, moving film.")n = len(tok(text)["input_ids"])pos, neg = (tok(text + w)["input_ids"][n] for w in (" positive", " negative")) # first token of each answerwith torch.no_grad(): last = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1]print("positive" if last[pos] > last[neg] else "negative")import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto")model = PeftModel.from_pretrained(model, "Falcot922/tinyllama-sentiment-distilled").eval() prompt = ("Analyze the sentiment of this movie review and respond with exactly one word: " "positive or negative.\n\nReview: {text}\nSentiment:")text = prompt.format(text="A wonderful, moving film.")n = len(tok(text)["input_ids"])pos, neg = (tok(text + w)["input_ids"][n] for w in (" positive", " negative")) # first token of each answerwith torch.no_grad(): last = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1]print("positive" if last[pos] > last[neg] else "negative")