GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Vishva007
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Vishva007
Model Tree
Qwen/Qwen3-4B-Instruct-2507
Input Modalities
Output Modalities
Supported Functionality
This is the AWQ (Activation-aware Weight Quantization) version of Qwen/Qwen3-4B-Instruct-2507.
It was generated using Intel's AutoRound algorithm, which optimizes the weight rounding to minimize quantization loss. This ensures superior accuracy compared to standard AWQ conversion methods.
vLLM, TGI, and Transformers.This model is optimized for high-throughput serving with vLLM.
bash
pip install vllmpython
from vllm import LLM, SamplingParams model_id = "Vishva007/Qwen3-4B-Instruct-2507-W4A16-AutoRound-AWQ" llm = LLM( model=model_id, quantization="awq", dtype="half", max_model_len=8192, gpu_memory_utilization=0.90) prompts = ["What is the capital of France?"]sampling_params = SamplingParams(temperature=0.7, top_p=0.8) outputs = llm.generate(prompts, sampling_params)for output in outputs: print(output.outputs.text)bash
pip install autoawq transformerspython
from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Vishva007/Qwen3-4B-Instruct-2507-W4A16-AutoRound-AWQ" model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto")tokenizer = AutoTokenizer.from_pretrained(model_id) prompt = "Write a python function to reverse a string."messages = [{"role": "user", "content": prompt}] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=256)print(tokenizer.decode(outputs, skip_special_tokens=True))This model maintains the strong performance of the Qwen3-4B-Instruct-2507 base model, including its updated reasoning and coding capabilities.
| Model | Format | VRAM (Est.) |
|---|---|---|
| Qwen3-4B-Instruct (BF16) | Original | ~9 GB |
| Qwen3-4B-Instruct (AWQ) | 4-bit | ~3.5 GB |
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get 5–500 in free credits when you add your first $10.
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.15 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.15-runpod | 0rsw686010 |  |
| PyTorch 2.15 (CUDA 13.4) | 13.4 | vishva123/cuda-13.4-pytorch-2.15-runpod | 0s0rhdfaqy | |
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.14 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.14-runpod | d7lxsa4w9m | |
| PyTorch 2.14 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.14-runpod | yk0y6j6rpg | |
| PyTorch 2.14 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.14-runpod | gsp4gwx0nw | |
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.13 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.13-runpod | gmlupxnxfk | |
| PyTorch 2.13 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.13-runpod | y3j8xvk4f4 | |
| PyTorch 2.13 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.13-runpod | vigpissn5w | |
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.12 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.12-runpod | ctmz86zmf0 | |
| PyTorch 2.12 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.12-runpod | qjko5yiwzi | |
| PyTorch 2.12 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.12-runpod | ifg6xmye0f | |
bibtex
@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}, eprint={2505.09388}, archivePrefix={arXiv}, primaryClass={cs.CL}}from vllm import LLM, SamplingParams model_id = "Vishva007/Qwen3-4B-Instruct-2507-W4A16-AutoRound-AWQ" llm = LLM( model=model_id, quantization="awq", dtype="half", max_model_len=8192, gpu_memory_utilization=0.90) prompts = ["What is the capital of France?"]sampling_params = SamplingParams(temperature=0.7, top_p=0.8) outputs = llm.generate(prompts, sampling_params)for output in outputs: print(output.outputs.text)from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Vishva007/Qwen3-4B-Instruct-2507-W4A16-AutoRound-AWQ" model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto")tokenizer = AutoTokenizer.from_pretrained(model_id) prompt = "Write a python function to reverse a string."messages = [{"role": "user", "content": prompt}] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=256)print(tokenizer.decode(outputs, skip_special_tokens=True))@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}, eprint={2505.09388}, archivePrefix={arXiv}, primaryClass={cs.CL}}