GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
saidutta69
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
saidutta69
Model Tree
Qwen/Qwen3-VL-4B-Instruct
Input Modalities
Output Modalities
Supported Functionality
GGUF ladder included (text-only) — pick the quant that fits your card:
| Your GPU | Recommended quant | Weights |
|---|---|---|
| RTX 3090 / 4090 / 5090 (24 GB) | Q8_0 | 3.99 GB |
| RTX 4080 / 5080 / 4060 Ti 16G (16 GB) | Q6_K | 3.08 GB |
| RTX 3060 / 4070 / 5070 (12 GB) | Q5_K_M | 2.69 GB |
| RTX 4060 / 3070 (8 GB) | Q4_K_M | 2.33 GB |
| GTX 1660 Super / 2060 / 3050 laptop (6 GB) | Q4_K_M | 2.33 GB |
| CPU-only / Apple Silicon | Q4_K_M | 2.33 GB, fits in system RAM |
Weights only, at this model's 4.44B native size; add ~1 GB per 32K of context. OOM? Drop one quant level. Headroom to spare? Go one up.
Trial 120 of a 200-trial Heretic run (seed 3485018052).
| Parameter | Value |
|---|---|
| direction_index | 19.43 |
| attn.o_proj.max_weight | 1.49 |
| attn.o_proj.max_weight_position | 21.65 |
| attn.o_proj.min_weight | 0.92 |
| attn.o_proj.min_weight_distance | 9.30 |
| mlp.down_proj.max_weight | 1.12 |
| mlp.down_proj.max_weight_position | 25.61 |
| mlp.down_proj.min_weight | 0.11 |
| Metric | This model | Original model (Qwen/Qwen3-VL-4B-Instruct) |
|---|---|---|
| KL divergence | 0.0396 | 0 (by definition) |
| Refusals | 8/100 | 97/100 |
Refusals on the harmful evaluation set drop from 97/100 to 8/100 while the vision tower and grounding behaviour stay close to the original (KL 0.0396).
Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the Heretic repo and the original abliteration writeup for the mechanism.
Made with ❤️ by RACER IS OP — follow for more uncensored models
| File | Size |
|---|---|
model-00001-of-00002.safetensors | 4.65 GB |
model-00002-of-00002.safetensors | 3.62 GB |
BF16, 4.44B total (text backbone + vision encoder). Use these for image/video/OCR — the vision encoder is not in the GGUFs. The reproduce/ directory carries the full Heretic recipe — config.toml, requirements.txt, the Optuna study journal, and SHA-256 sums — so this exact model can be regenerated bit-for-bit. Reproduce it with heretic --reproduce reproduce/reproduce.json.
Text-only ladder (F16 + 4 quants) produced with llama.cpp.
| File | Format | Size |
|---|---|---|
Qwen3-VL-4B-Instruct-heretic-F16.gguf | GGUF F16 | 7.50 GB |
Qwen3-VL-4B-Instruct-heretic-Q4_K_M.gguf | GGUF Q4_K_M | 2.33 GB |
Qwen3-VL-4B-Instruct-heretic-Q5_K_M.gguf | GGUF Q5_K_M | 2.69 GB |
Qwen3-VL-4B-Instruct-heretic-Q6_K.gguf | GGUF Q6_K | 3.08 GB |
Qwen3-VL text backbone (arch=qwen3vl) - loads natively in llama.cpp / LM Studio / Jan for text-only
use. Run llama serve -hf saidutta69/Qwen3-VL-4B-Instruct-heretic:Q4_K_M to pull the default quant.
bash
# llama.cpp - TEXT ONLY (the GGUF has no vision encoder)llama serve -hf saidutta69/Qwen3-VL-4B-Instruct-heretic:Q4_K_Mpython
# transformers - full vision-language (use the safetensors, not the GGUF)from transformers import Qwen3VLForConditionalGeneration, AutoProcessor model_name = "saidutta69/Qwen3-VL-4B-Instruct-heretic"model = Qwen3VLForConditionalGeneration.from_pretrained(model_name, dtype="auto", device_map="auto")processor = AutoProcessor.from_pretrained(model_name) messages = [{"role": "user", "content": [ {"type": "image", "image": "https://example.com/screenshot.png"}, {"type": "text", "text": "Transcribe all text in this image and describe the layout."},]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))Also runnable via vLLM, SGLang, Ollama, LM Studio, and Jan.
Qwen3-VL emits bounding boxes in a normalised coordinate space, so you can ask for element positions directly and get coordinates back rather than prose descriptions — the practical path for UI automation and for building training data from screenshots.
| Architecture | Qwen3VLForConditionalGeneration (vision-language decoder) |
| Parameters | 4.44B total; 4.02B text backbone in the GGUF |
| Layers / heads | 36 layers, 32 attention heads, 8 KV heads, head dim 128 |
| Hidden / intermediate | 2560 / 9728 |
| Position embedding | mRoPE (interleaved, sections 24/20/20), theta = 5,000,000 |
| Context length | 262,144 |
| Vocab | 151,936 |
Qwen3-VL is the vision-language branch of the Qwen series, available in dense and MoE sizes with Instruct and Thinking editions. This generation brings deeper visual perception and reasoning, extended context, stronger spatial and video understanding, and improved agent interaction — OCR, grounding, and video temporal reasoning are the headline capabilities over the earlier Qwen2.5-VL.
Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it — don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Qwen3-VL's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.
Inherits the apache-2.0 license from the base model.
| mlp.down_proj.min_weight_distance | 15.41 |
Qwen3-VL-4B-Instruct-heretic-Q8_0.gguf | GGUF Q8_0 | 3.99 GB |
| bfloat16 |
| Modalities | Text, image, and video in; text out (safetensors). GGUF is text-only |
| Base model | Qwen/Qwen3-VL-4B-Instruct |