Why the Dockerfile is required
This package currently depends on a patched vLLM runtime.
The included Dockerfile applies a loader patch needed for this checkpoint layout, specifically to support loading split Q/K/V vision weights into combined qkv_proj parameters used by the runtime.
Use the provided Docker image unless you have independently applied and validated the same patch in your own vLLM environment.
Recommended serving path
From the root of this model repo:
docker build -t vllm-glm46v-bf16vision .
docker run --rm -it \
--gpus all \
--ipc=host \
-p 8001:8000 \
-v "$(pwd):/model" \
vllm-glm46v-bf16vision \
--model /model \
--served-model-name GLM-4.6V-Flash-W4A16-BF16Vision \
--trust-remote-code \
--dtype bfloat16 \
--quantization compressed-tensors \
--max-model-len 8000 \
--limit-mm-per-prompt '{"image":1}' \
--allowed-local-media-path /
The --max-model-len 8000 value is the recommended local serving default for this package. It is not a statement about the upstream model’s full trained context length.
Smoke test
After the server is running:
python examples/smoke_image_data_url.py \
--url http://localhost:8001/v1/chat/completions \
--model GLM-4.6V-Flash-W4A16-BF16Vision
The smoke test sends an image as a data URL to the OpenAI-compatible /v1/chat/completions endpoint and checks that the model returns a text response.
Expected use
Best fit:
- RTX 30xx / 40xx GPUs
- local multimodal inference
- OpenAI-compatible vLLM serving
- one image per prompt
- moderate context serving defaults
Not the best fit:
- CPU inference
- Apple Silicon inference
- unpatched vLLM installs
- generic Transformers pipeline usage without testing
- long-context stress testing without adjusting memory settings
Notes
- The base model is
zai-org/GLM-4.6V-Flash.
- This repo is a quantized serving package derived from the base model.
- Text / MoE weights use W4A16 compressed-tensors.
- Vision weights remain BF16.
- Runtime validation is tied to the supplied Dockerfile.
- If you use a different serving stack, treat it as unsupported until independently smoke-tested.
Files
Expected repo layout:
.
├── README.md
├── LICENSE
├── Dockerfile
├── patches/
│ └── vllm_glm46v_qkv_loader.patch
├── examples/
│ └── smoke_image_data_url.py
├── config.json
├── generation_config.json
├── tokenizer.json
├── tokenizer_config.json
├── preprocessor_config.json
├── chat_template.jinja
├── model.safetensors.index.json
└── *.safetensors
Attribution
This package is derived from zai-org/GLM-4.6V-Flash.