GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Falcot922
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Falcot922
Model Tree
unsloth/mistral-7b-instruct-v0.3-bnb-4bit
Input Modalities
Output Modalities
Supported Functionality
1,000 IMDB test reviews (shuffled with seed 42, first 400 characters). The answer is read from the logits of the class tokens at the last prompt position, without generation.
| Model | Accuracy (95% Wilson CI) |
|---|---|
| Mistral NF4, no adapter | 85.4% (83.1 to 87.5) |
| Mistral NF4 + this adapter | 91.6% (89.7 to 93.2) |
Paired test on the same reviews: p < 0.001. Peak VRAM 4.1 GiB on a T4, about 0.87 reviews per second.
unsloth/mistral-7b-instruct-v0.3-bnb-4bit (an ungated copy of the official model, NF4)python
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = "unsloth/mistral-7b-instruct-v0.3-bnb-4bit"tok = AutoTokenizer.from_pretrained(base)model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base, device_map="auto"), "Falcot922/mistral-nf4-sentiment").eval() prompt = ("[INST] Analyze the sentiment of this movie review and respond with exactly one word: " "positive or negative.\n\nReview: {text} [/INST]")text = prompt.format(text="A wonderful, moving film.")n = len(tok(text)["input_ids"])pos, neg = (tok(text + w)["input_ids"][n] for w in ("positive", "negative")) # first token of each answerwith torch.no_grad(): last = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1]print("positive" if last[pos] > last[neg] else "negative")The evaluation code also groups the variants "positive" / "Positive" with logsumexp; see the distillation notebook.