Rainy-W-cy

qwen2-0.5b-instruct-gptq-w8a16

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Base model	`Qwen/Qwen2-0.5B-Instruct`
Base revision	`c540970f9e29518b1d8f06ab8b24cba66ad77b6d`
Offline algorithm	GPTQ
Quantized targets	`Linear`
Weight format	Signed INT8, symmetric
Weight granularity	Per-output-channel (`strategy=channel`, `group_size=null`)
Activation/KV format	Floating point; not quantized by this checkpoint
Ignored module	`lm_head`
Resolved model dtype	BF16
Serialization	`compressed-tensors`, `pack-quantized`
Export provider	`llm-compressor==0.12.0`
Compression library	`compressed-tensors==0.17.1`

Calibration and GPTQ Configuration

Table with columns: Field, Value
Field	Value
Dataset	`rajpurkar/squad`, configuration `plain_text`
Dataset revision	`7b6d24c440a36b6815f21b70d25016731768db1f`
Split / text field	`train` / `context`
Selected samples	8
Maximum sequence length	256 tokens
Seed	42

The SQuAD examples were used only for GPTQ calibration, not for fine-tuning. The 8×256 calibration setting was chosen to close and verify the export pipeline; it should not be presented as a fully optimized or paper-scale GPTQ calibration study.

Usage

Use a Transformers environment that supports the serialized compressed-tensors format. The export environment used Transformers 5.10.1 and compressed-tensors==0.17.1.

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Rainy-W-cy/qwen2-0.5b-instruct-gptq-w8a16"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain GPTQ weight-only quantization briefly."},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)
print(response)

Quantization Files

config.json contains the effective compressed-tensors quantization configuration.
provider_recipe.json records calibration identity and requested/resolved provider parameters.
recipe.yaml contains the effective GPTQ recipe.
model.safetensors contains the compressed model weights.

Evaluation

No formal benchmark results are claimed in this model card yet. The local artifact manifest records software evaluation as not_checked. Before publishing results, evaluate this checkpoint and the BF16 base model with identical task versions, prompts, sample selections, chat templates, generation parameters, and random seeds.

Example with LM Evaluation Harness:

bash
lm-eval run \
  --model hf \
  --model_args pretrained=Rainy-W-cy/qwen2-0.5b-instruct-gptq-w8a16,dtype=auto \
  --tasks arc_challenge \
  --num_fewshot 0 \
  --apply_chat_template \
  --batch_size 1 \
  --log_samples \
  --output_path runs/gptq_w8a16/arc_challenge

Intended Use

Research on GPTQ INT8 weight-only quantization.
Software quality comparison against the pinned BF16 base model and the RTN W8A16 checkpoint.
Input to a separately validated deployment compiler or hardware-runtime workflow.

Limitations

This is W8A16/weight-only, not end-to-end W8A8 or integer-only inference.
lm_head, activations, Attention operations, and KV Cache are not INT8-quantized by this checkpoint.
The small calibration set is suitable for pipeline validation but may not be optimal for model quality.
Runtime speed depends on backend support; reduced weight storage does not guarantee faster inference on every device.
The model inherits the capabilities, limitations, and potential biases of the base Qwen2 model.
No formal accuracy, perplexity, latency, power, or hardware-resource result is bundled yet.

License

The base model is released under the Apache 2.0 license. This quantized derivative follows the same license; users remain responsible for complying with the base model terms.

Citation

bibtex
@article{qwen2,
  title={Qwen2 Technical Report},
  author={Yang, An and others},
  journal={arXiv preprint arXiv:2407.10671},
  year={2024}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Rainy-W-cy

Model Tree

Base

Qwen/Qwen2-0.5B-Instruct

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Base model	`Qwen/Qwen2-0.5B-Instruct`
Base revision	`c540970f9e29518b1d8f06ab8b24cba66ad77b6d`
Offline algorithm	GPTQ
Quantized targets	`Linear`
Weight format	Signed INT8, symmetric
Weight granularity	Per-output-channel (`strategy=channel`, `group_size=null`)
Activation/KV format	Floating point; not quantized by this checkpoint
Ignored module	`lm_head`
Resolved model dtype	BF16
Serialization	`compressed-tensors`, `pack-quantized`
Export provider	`llm-compressor==0.12.0`
Compression library	`compressed-tensors==0.17.1`

Calibration and GPTQ Configuration

Table with columns: Field, Value
Field	Value
Dataset	`rajpurkar/squad`, configuration `plain_text`
Dataset revision	`7b6d24c440a36b6815f21b70d25016731768db1f`
Split / text field	`train` / `context`
Selected samples	8
Maximum sequence length	256 tokens
Seed	42

Usage

Use a Transformers environment that supports the serialized compressed-tensors format. The export environment used Transformers 5.10.1 and compressed-tensors==0.17.1.

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Rainy-W-cy/qwen2-0.5b-instruct-gptq-w8a16"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain GPTQ weight-only quantization briefly."},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)
print(response)

Quantization Files

config.json contains the effective compressed-tensors quantization configuration.
provider_recipe.json records calibration identity and requested/resolved provider parameters.
recipe.yaml contains the effective GPTQ recipe.
model.safetensors contains the compressed model weights.

Evaluation

Example with LM Evaluation Harness:

bash
lm-eval run \
  --model hf \
  --model_args pretrained=Rainy-W-cy/qwen2-0.5b-instruct-gptq-w8a16,dtype=auto \
  --tasks arc_challenge \
  --num_fewshot 0 \
  --apply_chat_template \
  --batch_size 1 \
  --log_samples \
  --output_path runs/gptq_w8a16/arc_challenge

Intended Use

Research on GPTQ INT8 weight-only quantization.
Software quality comparison against the pinned BF16 base model and the RTN W8A16 checkpoint.
Input to a separately validated deployment compiler or hardware-runtime workflow.

Limitations

This is W8A16/weight-only, not end-to-end W8A8 or integer-only inference.
lm_head, activations, Attention operations, and KV Cache are not INT8-quantized by this checkpoint.
The small calibration set is suitable for pipeline validation but may not be optimal for model quality.
Runtime speed depends on backend support; reduced weight storage does not guarantee faster inference on every device.
The model inherits the capabilities, limitations, and potential biases of the base Qwen2 model.
No formal accuracy, perplexity, latency, power, or hardware-resource result is bundled yet.

License

The base model is released under the Apache 2.0 license. This quantized derivative follows the same license; users remain responsible for complying with the base model terms.

Citation

bibtex
@article{qwen2,
  title={Qwen2 Technical Report},
  author={Yang, An and others},
  journal={arXiv preprint arXiv:2407.10671},
  year={2024}
}