useful-quants

GLM-4.6V-Flash-W4A16-BF16Vision

Why the Dockerfile is required

This package currently depends on a patched vLLM runtime.

The included Dockerfile applies a loader patch needed for this checkpoint layout, specifically to support loading split Q/K/V vision weights into combined qkv_proj parameters used by the runtime.

Use the provided Docker image unless you have independently applied and validated the same patch in your own vLLM environment.

Recommended serving path

From the root of this model repo:

bash
docker build -t vllm-glm46v-bf16vision .

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v "$(pwd):/model" \
  vllm-glm46v-bf16vision \
  --model /model \
  --served-model-name GLM-4.6V-Flash-W4A16-BF16Vision \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization compressed-tensors \
  --max-model-len 8000 \
  --limit-mm-per-prompt '{"image":1}' \
  --allowed-local-media-path /

The --max-model-len 8000 value is the recommended local serving default for this package. It is not a statement about the upstream model’s full trained context length.

Smoke test

After the server is running:

bash
python examples/smoke_image_data_url.py \
  --url http://localhost:8001/v1/chat/completions \
  --model GLM-4.6V-Flash-W4A16-BF16Vision

The smoke test sends an image as a data URL to the OpenAI-compatible /v1/chat/completions endpoint and checks that the model returns a text response.

Expected use

Best fit:

RTX 30xx / 40xx GPUs
local multimodal inference
OpenAI-compatible vLLM serving
one image per prompt
moderate context serving defaults

Not the best fit:

CPU inference
Apple Silicon inference
unpatched vLLM installs
generic Transformers pipeline usage without testing
long-context stress testing without adjusting memory settings

Notes

The base model is zai-org/GLM-4.6V-Flash.
This repo is a quantized serving package derived from the base model.
Text / MoE weights use W4A16 compressed-tensors.
Vision weights remain BF16.
Runtime validation is tied to the supplied Dockerfile.
If you use a different serving stack, treat it as unsupported until independently smoke-tested.

Files

Expected repo layout:

text
.
├── README.md
├── LICENSE
├── Dockerfile
├── patches/
│   └── vllm_glm46v_qkv_loader.patch
├── examples/
│   └── smoke_image_data_url.py
├── config.json
├── generation_config.json
├── tokenizer.json
├── tokenizer_config.json
├── preprocessor_config.json
├── chat_template.jinja
├── model.safetensors.index.json
└── *.safetensors

Attribution

This package is derived from zai-org/GLM-4.6V-Flash.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

useful-quants

Model Tree

Base

zai-org/GLM-4.6V-Flash

Quantized

this model

Input Modalities

Text

Output Modalities