GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
pitcany
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
pitcany
Model Tree
cerebras/GLM-4.5-Air-REAP-82B-A12B
Input Modalities
Output Modalities
Supported Functionality
bash
sudo docker run --runtime nvidia --gpus all --detach \--name "vLLM" \-v ~/.cache/huggingface:/root/.cache/huggingface \-p 10.1.1.10:80:80 \--ipc=host \-e "CUDA_VISIBLE_DEVICES=0,1,2,3" \-e "VLLM_TARGET_DEVICE=cuda" \-e "CUDA_DEVICE_ORDER=PCI_BUS_ID" \-e "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \-e "VLLM_USE_FLASHINFER_SAMPLER=1" \-e "VLLM_ATTENTION_BACKEND=FLASHINFER" \vllm/vllm-openai:latest \--model "MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit" \--max-model-len 131072 \--dtype bfloat16 \--kv-cache-dtype fp8 \--gpu-memory-utilization 0.92 \--tensor-parallel-size 4 \--enable-expert-parallel \--swap-space 0 \--max-num-seqs 9 \--enable-auto-tool-choice \--tool-call-parser glm45 \--host 0.0.0.0 \--port 80 \--served-model-name "GLM-4.5-Air" \--enable-prefix-caching \--enable-chunked-prefillbash
sudo docker run --runtime nvidia --gpus all --detach \--name "vLLM" \-v ~/.cache/huggingface:/root/.cache/huggingface \-p 10.1.1.10:80:80 \--ipc=host \-e "CUDA_VISIBLE_DEVICES=0,1,2,3" \-e "VLLM_TARGET_DEVICE=cuda" \-e "CUDA_DEVICE_ORDER=PCI_BUS_ID" \-e "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \-e "VLLM_USE_FLASHINFER_SAMPLER=1" \-e "VLLM_ATTENTION_BACKEND=FLASHINFER" \vllm/vllm-openai:latest \--model "MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit" \--max-model-len 131072 \--dtype bfloat16 \--kv-cache-dtype fp8 \--gpu-memory-utilization 0.92 \--tensor-parallel-size 4 \--enable-expert-parallel \--swap-space 0 \--max-num-seqs 9 \--enable-auto-tool-choice \--tool-call-parser glm45 \--reasoning-parser glm45 \--host 0.0.0.0 \--port 80 \--served-model-name "GLM-4.5-Air" \--enable-prefix-caching \--enable-chunked-prefillThis model was quantized using the following configuration:
yaml
- 4-bit integer symmetric quantization- Group size: 32- MSE observer- Duo scaling: enabled- Custom smoothing mappings for MoE architecture- Calibration dataset: HuggingFaceH4/ultrachat_200k (256 samples)The following layers were not quantized to preserve model quality:
The following layers were quantized to 4-bit:
This model is optimized for inference with vLLM.
bash
pip install vllmpython
from vllm import LLM, SamplingParams # Load the quantized modelmodel = LLM("MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit") # Configure sampling parameterssampling_params = SamplingParams( temperature=0.7, top_p=0.9, max_tokens=512) # Generate textprompt = "Hello, I am an AI assistant. How can I help you today?"outputs = model.generate([prompt], sampling_params) print(outputs[0].outputs[0].text)python
from vllm import LLM, SamplingParams model = LLM("MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit") messages = [ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What is the capital of France?"}] # The model will automatically apply the chat templateoutputs = model.chat(messages, sampling_params=SamplingParams(max_tokens=256))print(outputs[0].outputs[0].text)Minimum:
Recommended:
This model was quantized on 2x RTX 6000 Blackwell GPUs (96GB each) using:
This quantized model is suitable for:
If you use this model, please cite the original GLM-4.5 paper and the llm-compressor library:
bibtex
@software{llm-compressor, title = {LLM Compressor}, author = {vLLM Team}, url = {https://github.com/vllm-project/llm-compressor}, year = {2024}}This model inherits the Apache 2.0 license from the base model cerebras/GLM-4.5-Air-REAP-82B-A12B.
For issues or questions about this quantized model, please open an issue on the model repository.
For questions about the original model, see cerebras/GLM-4.5-Air-REAP-82B-A12B.
sudo docker run --runtime nvidia --gpus all --detach \--name "vLLM" \-v ~/.cache/huggingface:/root/.cache/huggingface \-p 10.1.1.10:80:80 \--ipc=host \-e "CUDA_VISIBLE_DEVICES=0,1,2,3" \-e "VLLM_TARGET_DEVICE=cuda" \-e "CUDA_DEVICE_ORDER=PCI_BUS_ID" \-e "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \-e "VLLM_USE_FLASHINFER_SAMPLER=1" \-e "VLLM_ATTENTION_BACKEND=FLASHINFER" \vllm/vllm-openai:latest \--model "MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit" \--max-model-len 131072 \--dtype bfloat16 \--kv-cache-dtype fp8 \--gpu-memory-utilization 0.92 \--tensor-parallel-size 4 \--enable-expert-parallel \--swap-space 0 \--max-num-seqs 9 \--enable-auto-tool-choice \--tool-call-parser glm45 \--host 0.0.0.0 \--port 80 \--served-model-name "GLM-4.5-Air" \--enable-prefix-caching \--enable-chunked-prefillsudo docker run --runtime nvidia --gpus all --detach \--name "vLLM" \-v ~/.cache/huggingface:/root/.cache/huggingface \-p 10.1.1.10:80:80 \--ipc=host \-e "CUDA_VISIBLE_DEVICES=0,1,2,3" \-e "VLLM_TARGET_DEVICE=cuda" \-e "CUDA_DEVICE_ORDER=PCI_BUS_ID" \-e "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \-e "VLLM_USE_FLASHINFER_SAMPLER=1" \-e "VLLM_ATTENTION_BACKEND=FLASHINFER" \vllm/vllm-openai:latest \--model "MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit" \--max-model-len 131072 \--dtype bfloat16 \--kv-cache-dtype fp8 \--gpu-memory-utilization 0.92 \--tensor-parallel-size 4 \--enable-expert-parallel \--swap-space 0 \--max-num-seqs 9 \--enable-auto-tool-choice \--tool-call-parser glm45 \--reasoning-parser glm45 \--host 0.0.0.0 \--port 80 \--served-model-name "GLM-4.5-Air" \--enable-prefix-caching \--enable-chunked-prefill- 4-bit integer symmetric quantization- Group size: 32- MSE observer- Duo scaling: enabled- Custom smoothing mappings for MoE architecture- Calibration dataset: HuggingFaceH4/ultrachat_200k (256 samples)from vllm import LLM, SamplingParams # Load the quantized modelmodel = LLM("MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit") # Configure sampling parameterssampling_params = SamplingParams( temperature=0.7, top_p=0.9, max_tokens=512) # Generate textprompt = "Hello, I am an AI assistant. How can I help you today?"outputs = model.generate([prompt], sampling_params) print(outputs[0].outputs[0].text)from vllm import LLM, SamplingParams model = LLM("MidnightPhreaker/GLM-4.5-Air-REAP-82B-A12B-AWQ-4bit") messages = [ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What is the capital of France?"}] # The model will automatically apply the chat templateoutputs = model.chat(messages, sampling_params=SamplingParams(max_tokens=256))print(outputs[0].outputs[0].text)@software{llm-compressor, title = {LLM Compressor}, author = {vLLM Team}, url = {https://github.com/vllm-project/llm-compressor}, year = {2024}}