GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
moonzerokevin
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
moonzerokevin
Model Tree
Qwen/Qwen3-VL-32B-Instruct
Input Modalities
Output Modalities
Supported Functionality
In bf16 the encoder is 66.7 GB, which does not fit on a 5090/4090-class card. The usual
workaround is accelerate.cpu_offload, which streams the whole model across PCIe on every
request. In a three-stage H3 deployment that made text encoding the pipeline bottleneck.
| bf16 + cpu_offload | this artifact (NF4 resident) | |
|---|---|---|
| text encode per request | 7.4 – 8.5 s | 0.09 s (text-only) / 0.5 – 1.5 s (with a keyframe) |
| weights on disk | 66.7 GB | 19 GB |
| VRAM while resident | 1.5 GB (weights keep crossing PCIe) | 18.2 GB |
| load time | ~16 s (read 66.7 GB + quantise) | ~6 s |
Measured on an RTX 5090 (31.36 GiB), transformers 5.15.0, bitsandbytes 0.50.1.
prompt_embeds versus the bf16 reference: cosine 1.00003, relative L2 1.6 %.
End-to-end, generating video with the same prompt, spec and seed — only the encoder differs — frame PSNR against the bf16 run:
| comparison | PSNR |
|---|---|
| same file against itself | ∞ |
| bf16 ↔ NF4, first+last-frame conditioning | 24 – 29 dB |
| bf16 ↔ NF4, first-frame conditioning | 18 – 28 dB |
| bf16 ↔ NF4, text-to-video | 15 – 17 dB |
| same config, different seed | 8.6 – 10.2 dB |
The encoder swap sits far from the "different seed" floor, i.e. it perturbs the same clip rather than resampling a new one — and image-conditioned modes are affected least, because the reference frame anchors the composition.
python
from transformers import Qwen3VLForConditionalGenerationtext_encoder = Qwen3VLForConditionalGeneration.from_pretrained("moonzerokevin/qwen3vl-32b-minimax-h3-nf4", device_map="cuda")
Then hand it to the H3 modular pipeline in place of the bf16 encoder:
python
pipe = ModularPipeline.from_pretrained(h3_path, workflow="fl2va")pipe.load_components(names=["tokenizer", "processor", "vae", ...]) # skip text_encoderpipe.text_encoder = text_encoder
Requires bitsandbytes (tested with 0.50.1 on sm120).
RedHatAI/Qwen3-VL-32B-Instruct-NVFP4 (compressed-tensors, ~16 GB) loads under
transformers but cannot run inference there — the NVFP4 kernels live in vLLM; the
forward pass fails with 'Linear' object has no attribute 'weight'. That is why this
artifact uses bitsandbytes NF4 instead.bnb_4bit_use_double_quant=True), compute dtype bfloat16.