GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Rainy-W-cy
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Rainy-W-cy
Model Tree
Qwen/Qwen2-0.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Field | Value |
|---|---|
| Base model | Qwen/Qwen2-0.5B-Instruct |
| Base revision | c540970f9e29518b1d8f06ab8b24cba66ad77b6d |
| Offline algorithm | RTN (round-to-nearest weight quantization) |
| Quantized targets | Linear |
| Weight format | Signed INT8, symmetric |
| Weight granularity | Per-output-channel (strategy=channel, group_size=null) |
| Activation/KV format | Floating point; not quantized by this checkpoint |
| Ignored module | lm_head |
| Resolved model dtype | BF16 |
| Serialization | compressed-tensors, pack-quantized |
| Export provider | llm-compressor==0.12.0 |
| Compression library | compressed-tensors==0.17.1 |
RTN was executed without a calibration dataset. No fine-tuning or additional training was performed.
Use a Transformers environment that supports the serialized compressed-tensors format. The export environment used Transformers 5.10.1 and compressed-tensors==0.17.1.
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel_id = "<your-hf-account>/Qwen2-0.5B-Instruct-RTN-W8A16"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id,torch_dtype="auto",device_map="auto",)messages = [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Explain INT8 weight-only quantization briefly."},]text = tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True,)inputs = tokenizer([text], return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:],skip_special_tokens=True,)print(response)
config.json contains the effective compressed-tensors quantization configuration.provider_recipe.json records the requested and resolved provider parameters.recipe.yaml contains the effective llm-compressor recipe.model.safetensors contains the compressed model weights.No formal benchmark results are claimed in this model card yet. The local artifact manifest records software evaluation as not_checked. Before publishing results, evaluate this checkpoint and the BF16 base model with identical task versions, prompts, sample selections, chat templates, generation parameters, and random seeds.
Example with LM Evaluation Harness:
bash
lm-eval run \--model hf \--model_args pretrained=<your-hf-account>/Qwen2-0.5B-Instruct-RTN-W8A16,dtype=auto \--tasks arc_challenge \--num_fewshot 0 \--apply_chat_template \--batch_size 1 \--log_samples \--output_path runs/rtn_w8a16/arc_challenge
lm_head, activations, Attention operations, and KV Cache are not INT8-quantized by this checkpoint.The base model is released under the Apache 2.0 license. This quantized derivative follows the same license; users remain responsible for complying with the base model terms.
bibtex
@article{qwen2,title={Qwen2 Technical Report},author={Yang, An and others},journal={arXiv preprint arXiv:2407.10671},year={2024}}