GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Vishva007
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Vishva007
Model Tree
Qwen/Qwen3-VL-8B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This model is the AWQ (Activation-aware Weight Quantization) export of Qwen/Qwen3-VL-8B-Instruct.
It combines the speed of AWQ with the accuracy of AutoRound. The weights were fine-tuned for 800 steps to ensure the 4-bit degradation is negligible. The vision tower remains in full precision (FP16) to maintain top-tier performance on OCR and visual tasks.
vLLM serving and Transformers.bash
pip install vllmpython
from vllm import LLM, SamplingParams model_id = "Vishva007/Qwen3-VL-8B-Instruct-W4A16-AutoRound-AWQ" llm = LLM( model=model_id, quantization="awq", trust_remote_code=True, max_model_len=4096) # ... (Standard vLLM inference code)bash
pip install autoawq transformerspython
from transformers import Qwen2VLForConditionalGeneration, AutoProcessorimport torch model_id = "Vishva007/Qwen3-VL-8B-Instruct-W4A16-AutoRound-AWQ" # Load with Flash Attention 2 for best performancemodel = Qwen2VLForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.float16, device_map="auto", attn_implementation="flash_attention_2")processor = AutoProcessor.from_pretrained(model_id) # Inference Examplemessages = [ { "role": "user", "content": [ {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"}, {"type": "text", "text": "What does this image show?"}, ], }] text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor( text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt",).to("cuda") generated_ids = model.generate(**inputs, max_new_tokens=128)print(processor.batch_decode(generated_ids, skip_special_tokens=True))One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get 5–500 in free credits when you add your first $10.
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.14 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.14-runpod | d7lxsa4w9m |  |
| PyTorch 2.14 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.14-runpod |
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.13 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.13-runpod | gmlupxnxfk | |
| PyTorch 2.13 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.13-runpod |
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.12 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.12-runpod | ctmz86zmf0 | |
| PyTorch 2.12 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.12-runpod |
bibtex
@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}}yk0y6j6rpg |
|
| PyTorch 2.14 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.14-runpod | gsp4gwx0nw | |
y3j8xvk4f4 |
|
| PyTorch 2.13 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.13-runpod | vigpissn5w | |
qjko5yiwzi |
|
| PyTorch 2.12 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.12-runpod | ifg6xmye0f | |