GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Crusadersk
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Crusadersk
Model Tree
mistralai/Mistral-7B-Instruct-v0.3
Input Modalities
Output Modalities
Supported Functionality
| Parameter | Value |
|---|---|
| Method | GPTQ |
| Tool | gptqmodel |
| Bits | 4 |
| Group size | 128 |
| Scheme | Symmetric (4-bit, INT32 packing) |
| Calibration dataset | allenai/c4 (en, shard 1 of 1024) |
| Calibration samples | 128 |
| Seed | 42 |
| Quantization time | 542s |
| Hardware | RunPod RTX 6000 Ada (48 GB) |
Pre-quantized checkpoints on HuggingFace typically have unknown calibration provenance — the dataset, sample count, seed, and group size are rarely documented. This checkpoint was self-quantized with controlled, documented settings to enable rigorous cross-method comparison (GGUF k-quant vs AWQ vs GPTQ) in a NeurIPS 2026 submission on quality-safety correlation under quantization.
Evaluation pending — quality and safety benchmarks will be run on this checkpoint and results updated here.
| Format | Repository |
|---|---|
| Original FP16 | mistralai/Mistral-7B-Instruct-v0.3 |
| AWQ 4-bit | Crusadersk/mistral-7b-awq-4bit |
markdown
[INST] {prompt} [/INST]
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel = AutoModelForCausalLM.from_pretrained("Crusadersk/mistral-7b-gptq-4bit",device_map="auto",)tokenizer = AutoTokenizer.from_pretrained("Crusadersk/mistral-7b-gptq-4bit")messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)outputs = model.generate(inputs, max_new_tokens=64, do_sample=False)print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Inference requirements: pip install gptqmodel (Linux only) or optimum+auto-gptq
Windows users: GPTQ inference requires gptqmodel which only builds on Linux. Use Docker or WSL2.
| Framework | Supported |
|---|---|
| Transformers | Yes |
| vLLM | Yes (GPTQ backend) |
| llama.cpp | No (use GGUF format instead) |
| Ollama | No (use GGUF format instead) |
| Windows (native) | No — requires Linux/Docker |
The full quantization pipeline — Dockerfiles, quantization scripts, and a 766-line engineering log documenting every platform failure and solution — is available at:
markdown
research/tr142/expansion/
in the Banterhearts repository.
bibtex
@misc{banterhearts2026mistral7bgptq,title = {Self-Quantized Mistral 7B Instruct v0.3 (GPTQ 4-bit) for Quality-Safety Correlation Research},author = {Kadadekar, Sahil},year = {2026},url = {https://huggingface.co/Crusadersk/mistral-7b-gptq-4bit},note = {Part of the Banterhearts research program. NeurIPS 2026 submission.}}
This work is part of the Chimera/Banterhearts technical-report program on deployment-time LLM behavior, quantization, refusal robustness, batching effects, and inference-stack reliability. Canonical public archive: Chimeraforge Reports; source context: github.com/Sahil170595/Banterhearts.