GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
CCSSNE
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
CCSSNE
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
19cdf7e)text
model.safetensors (~16 GB)config.jsontokenizer.jsontokenizer_config.jsongeneration_config.jsonchat_template.jinjatext
comfyui/qwen3-8b-heretic.safetensors # bf16, 16GBcomfyui/qwen3-8b-heretic_fp8_e4m3fn.safetensors # fp8, 8.8GBcomfyui/qwen3-8b-heretic_nvfp4.safetensors # nvfp4, 6.0GB| Quant | Size | Notes |
|---|---|---|
| F16 | 16GB | Lossless reference |
| Q8_0 | 8.2GB | Excellent quality |
| Q6_K | 6.3GB | Very good quality |
| Q5_K_M | 5.5GB | Good quality |
| Q5_K_S | 5.4GB | Slightly smaller Q5 |
| Q4_K_M | 5.0GB | Recommended balance |
| Q4_K_S | 4.8GB | Smaller Q4 variant |
| Q3_K_M | 3.9GB | For low VRAM only |
The NVFP4 (4-bit floating point, E2M1) variants use ComfyUI's native quantization format. They are ~3x smaller than bf16 and load natively in ComfyUI without any plugins. Blackwell GPUs (RTX 5090/5080, SM100+) can use native FP4 tensor cores for best performance, but ComfyUI also supports software dequantization on older GPUs (tested working on RTX 4090).
Download a ComfyUI format file:
comfyui/qwen3-8b-heretic_fp8_e4m3fn.safetensors (8.8GB)comfyui/qwen3-8b-heretic_nvfp4.safetensors (6.0GB)comfyui/qwen3-8b-heretic.safetensors (16GB)Place in ComfyUI/models/text_encoders/
In your Klein 9B workflow, use the ClipLoader node and select the heretic file
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model = AutoModelForCausalLM.from_pretrained( "DreamFast/qwen3-8b-heretic", device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)tokenizer = AutoTokenizer.from_pretrained("DreamFast/qwen3-8b-heretic") prompt = "Describe a dramatic sunset over a cyberpunk city"inputs = tokenizer(prompt, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0], skip_special_tokens=True))bash
llama-server -m qwen3-8b-heretic-Q4_K_M.ggufCreated using Heretic v1.2.0 (git master) with 3000 optimization trials:
text
? Which trial do you want to use? [Trial 2732] Refusals: 10/100, KL divergence: 0.1001> [Trial 2681] Refusals: 13/100, KL divergence: 0.0838 <-- selected [Trial 2337] Refusals: 18/100, KL divergence: 0.0643 [Trial 2419] Refusals: 19/100, KL divergence: 0.0600 [Trial 2195] Refusals: 21/100, KL divergence: 0.0534 ...Trial 2681 was selected for its balance of low refusals (13/100) and reasonable KL divergence (0.0838), indicating minimal model damage while achieving 87% of previously-refused prompts now working.
This model is released under the Apache 2.0 License, following the base Qwen3-8B model license.
model.safetensors (~16 GB)config.jsontokenizer.jsontokenizer_config.jsongeneration_config.jsonchat_template.jinjacomfyui/qwen3-8b-heretic.safetensors # bf16, 16GBcomfyui/qwen3-8b-heretic_fp8_e4m3fn.safetensors # fp8, 8.8GBcomfyui/qwen3-8b-heretic_nvfp4.safetensors # nvfp4, 6.0GBfrom transformers import AutoModelForCausalLM, AutoTokenizerimport torch model = AutoModelForCausalLM.from_pretrained( "DreamFast/qwen3-8b-heretic", device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)tokenizer = AutoTokenizer.from_pretrained("DreamFast/qwen3-8b-heretic") prompt = "Describe a dramatic sunset over a cyberpunk city"inputs = tokenizer(prompt, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0], skip_special_tokens=True))? Which trial do you want to use? [Trial 2732] Refusals: 10/100, KL divergence: 0.1001> [Trial 2681] Refusals: 13/100, KL divergence: 0.0838 <-- selected [Trial 2337] Refusals: 18/100, KL divergence: 0.0643 [Trial 2419] Refusals: 19/100, KL divergence: 0.0600 [Trial 2195] Refusals: 21/100, KL divergence: 0.0534 ...