GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
SAIFIINDUSTRIES
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
SAIFIINDUSTRIES
Model Tree
Qwen/Qwen2.5-VL-72B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Quantized version of Qwen/Qwen2.5-VL-72B-Instruct.
This model was obtained by quantizing the weights of Qwen/Qwen2.5-VL-72B-Instruct to INT4 data type, ready for inference with vLLM >= 0.5.2.
This model can be deployed efficiently using the vLLM backend, as shown in the example below.
python
from vllm.assets.image import ImageAssetfrom vllm import LLM, SamplingParams# prepare modelllm = LLM(model="neuralmagic/Qwen2.5-VL-72B-Instruct-quantized.w4a16",trust_remote_code=True,max_model_len=4096,max_num_seqs=2,)# prepare inputsquestion = "What is the content of this image?"inputs = {"prompt": f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n","multi_modal_data": {"image": ImageAsset("cherry_blossom").pil_image.convert("RGB")},}# generate responseprint("========== SAMPLE GENERATION ==============")outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))print(f"PROMPT : {outputs[0].prompt}")print(f"RESPONSE: {outputs[0].outputs[0].text}")print("==========================================")
vLLM also supports OpenAI-compatible serving. See the documentation for more details.
This model was created with llm-compressor by running the code snippet below as part a multimodal announcement blog.
python
import base64from io import BytesIOimport torchfrom datasets import load_datasetfrom qwen_vl_utils import process_vision_infofrom transformers import AutoProcessorfrom llmcompressor.modifiers.quantization import GPTQModifierfrom llmcompressor.transformers import oneshotfrom llmcompressor.transformers.tracing import (TraceableQwen2_5_VLForConditionalGeneration,)from compressed_tensors.quantization import QuantizationArgs, QuantizationType, QuantizationStrategy, ActivationOrdering, QuantizationScheme# Load model.model_id = "Qwen/Qwen2.5-VL-72B-Instruct"model = TraceableQwen2_5_VLForConditionalGeneration.from_pretrained(model_id,device_map="auto",torch_dtype="auto",)processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)# Oneshot argumentsDATASET_ID = "lmms-lab/flickr30k"DATASET_SPLIT = {"calibration": "test[:512]"}NUM_CALIBRATION_SAMPLES = 512MAX_SEQUENCE_LENGTH = 2048# Load dataset and preprocess.ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)ds = ds.shuffle(seed=42)dampening_frac=0.01# Apply chat template and tokenize inputs.def preprocess_and_tokenize(example):# preprocessbuffered = BytesIO()example["image"].save(buffered, format="PNG")encoded_image = base64.b64encode(buffered.getvalue())encoded_image_text = encoded_image.decode("utf-8")base64_qwen = f"data:image;base64,{encoded_image_text}"messages = [{"role": "user","content": [{"type": "image", "image": base64_qwen},{"type": "text", "text": "What does the image show?"},],}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)# tokenizereturn processor(text=[text],images=image_inputs,videos=video_inputs,padding=False,max_length=MAX_SEQUENCE_LENGTH,truncation=True,)ds = ds.map(preprocess_and_tokenize, remove_columns=ds["calibration"].column_names)# Define a oneshot data collator for multimodal inputs.def data_collator(batch):assert len(batch) == 1return {key: torch.tensor(value) for key, value in batch[0].items()}recipe = GPTQModifier(targets="Linear",config_groups={"config_group": QuantizationScheme(targets=["Linear"],weights=QuantizationArgs(num_bits=4,type=QuantizationType.INT,strategy=QuantizationStrategy.GROUP,group_size=128,symmetric=True,dynamic=False,actorder=ActivationOrdering.WEIGHT,),),},sequential_targets=["Qwen2_5_VLDecoderLayer"],ignore=["lm_head", "re:visual.*"],update_size=NUM_CALIBRATION_SAMPLES,dampening_frac=dampening_frac)SAVE_DIR=f"{model_id.split('/')[1]}-quantized.w4a16"# Perform oneshotoneshot(model=model,tokenizer=model_id,dataset=ds,recipe=recipe,max_seq_length=MAX_SEQUENCE_LENGTH,num_calibration_samples=NUM_CALIBRATION_SAMPLES,trust_remote_code_model=True,data_collator=data_collator,output_dir=SAVE_DIR)
The model was evaluated using mistral-evals for vision-related tasks and using lm_evaluation_harness for select text-based benchmarks. The evaluations were conducted using the following commands:
markdown
vllm serve RedHatAI/Qwen2.5-VL-72B-Instruct-quantized.w4a16 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7python -m eval.run eval_vllm \--model_name RedHatAI/Qwen2.5-VL-72B-Instruct-quantized.w4a16 \--url http://0.0.0.0:8000 \--output_dir ~/tmp \--eval_name <vision_task_name>
markdown
lm_eval \--model vllm \--model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \--tasks mmlu \--num_fewshot 5 \--batch_size auto \--output_path output_dir
markdown
lm_eval \--model vllm \--model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \--tasks mgsm_cot_native \--apply_chat_template \--num_fewshot 0 \--batch_size auto \--output_path output_dir
This model achieves up to 3.95x speedup in single-stream deployment and up to 6.6x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario. The following performance benchmarks were conducted with vLLM version 0.7.2, and GuideLLM.
**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens
**QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).
**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens
**QPS: Queries per second.
**QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).