Crusadersk

qwen2.5-1.5b-gptq-4bit

README

License: apache-2.0

Deployment safety notice

QuantSafe release action: ROUTE.

I retrospectively screened the immutable weight revision 4e1c7d4d78a3... against its matched FP16 baseline. The frozen screen reports RTSI 0.7864 (HIGH, the matrix maximum) and a raw refusal-rate shift from 99% to 47% (-52 percentage points).

This result does not prove that the checkpoint is unsafe. It is the publisher's release-gate decision: do not deploy this revision without direct safety evaluation; route to a safer baseline when the deployment cannot absorb that evaluation.

Open the QuantSafe evidence Â· Read the methodology

This README-only notice does not modify the screened model weights.

Self-quantized GPTQ 4-bit checkpoint of Qwen/Qwen2.5-1.5B-Instruct with fully documented calibration provenance.

Created as part of the Banterhearts research program investigating quality-safety correlation under quantization for consumer LLM deployment.

Table

Base model	Qwen/Qwen2.5-1.5B-Instruct
Parameters	1.54B
Architecture	GQA, 28 layers
Quantization	GPTQ 4-bit, group_size=128
Model size	1.1 GB
VRAM required	~1.6 GB (inference)

Quantization Details

Table with columns: Parameter, Value
Parameter	Value
Method	GPTQ
Tool	gptqmodel 5.8.0
Bits	4
Group size	128
Scheme	Symmetric (4-bit, INT32 packing)
Calibration dataset	allenai/c4 (en, shard 1 of 1024)
Calibration samples	128
Seed	42
Quantization time	555s

Why Self-Quantized?

Pre-quantized checkpoints on HuggingFace typically have unknown calibration provenance â€” the dataset, sample count, seed, and group size are rarely documented. This checkpoint was self-quantized with controlled, documented settings to enable rigorous cross-method comparison (GGUF k-quant vs AWQ vs GPTQ) in a NeurIPS 2026 submission on quality-safety correlation under quantization.

Evaluation Results

Evaluated on 735 quality samples across 7 tasks and 468 safety samples judged by gemma3:12b.

Quality Metrics (generation tasks)

Table with columns: Metric, Score
Metric	Score
BERTScore (F1)	0.614
ROUGE-L	0.270
Coherence	0.688

Accuracy (capability tasks)

Table with columns: Task, Accuracy
Task	Accuracy
MMLU	46.7%
ARC Challenge	70.0%
Classification	22.0%

Safety Metrics (gemma3:12b judge)

Table with columns: Metric, Score
Metric	Score
Refusal Rate (AdvBench)	64.0%
Truthfulness (TruthfulQA)	20.0%
Unbiased Rate (BBQ)	45.5%

Other Quantization Formats

Table with columns: Format, Repository
Format	Repository
AWQ 4-bit	Crusadersk/qwen2.5-1.5b-awq-4bit
Original FP16	Qwen/Qwen2.5-1.5B-Instruct

Prompt Template

markdown
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Crusadersk/qwen2.5-1.5b-gptq-4bit",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Crusadersk/qwen2.5-1.5b-gptq-4bit")

inputs = tokenizer("What is the capital of France?", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference requirements: pip install gptqmodel (Linux only) or optimum+auto-gptq

Windows users: GPTQ inference requires gptqmodel which only builds on Linux. Use Docker or WSL2. See reproduction instructions below.

Compatibility

Table with columns: Framework, Supported
Framework	Supported
Transformers	Yes
vLLM	Yes (GPTQ backend)
llama.cpp	No (use GGUF format instead)
Ollama	No (use GGUF format instead)
Windows (native)	No â€” requires Linux/Docker

Reproduction

The full quantization pipeline â€” Dockerfiles, quantization scripts, and a 766-line engineering log documenting every platform failure and solution â€” is available at:

markdown
research/tr142/expansion/

in the Banterhearts repository. Key files:

Table with columns: File, Purpose
File	Purpose
`QUANTIZATION_LOG.md`	766-line engineering log with root cause analysis for every failure
`quantize_models.py`	CLI for AWQ + GPTQ quantization with skip-existing and manifests
`Dockerfile.gptq` / `Dockerfile.awq`	Separate Docker images (irreconcilable dependency conflict)
`smoke_test.py`	Checkpoint verification with automatic Docker fallback for GPTQ
`run_hf_eval.py`	HuggingFace .generate() evaluation backend

Citation

bibtex
@misc{banterhearts2026qwen2515bgptq,
  title = {Self-Quantized Qwen 2.5 1.5B Instruct (GPTQ 4-bit) for Quality-Safety Correlation Research},
  author = {Kadadekar, Sahil},
  year = {2026},
  url = {https://huggingface.co/Crusadersk/qwen2.5-1.5b-gptq-4bit},
  note = {Part of the Banterhearts research program. NeurIPS 2026 submission.}
}

Acknowledgments

This work is part of the Chimera/Banterhearts technical-report program on deployment-time LLM behavior, quantization, refusal robustness, batching effects, and inference-stack reliability. Canonical public archive: Chimeraforge Reports; source context: github.com/Sahil170595/Banterhearts.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Crusadersk

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Quantized

this model

Input Modalities

Text

Output Modalities