GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
darrellbest
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
darrellbest
Model Tree
darrellbest/Qwen3-0.6B-Heretic
Input Modalities
Output Modalities
Supported Functionality
| Part | Precision |
|---|---|
| All attention and MLP linear layers (196) | FP8 E4M3 weights with per-channel scales, dynamic per-token FP8 activations (W8A8) |
Embeddings (tied to lm_head), norms | bf16 (unchanged) |
The 151,936 x 1,024 embedding table stays in bf16 and is about 0.31 GB of the file, which is why FP8 is not half the bf16 size on a model this small.
Made with llm-compressor 0.13.0 (scheme="FP8_DYNAMIC", no
calibration data needed). Format: compressed-tensors.
sh
vllm serve darrellbest/Qwen3-0.6B-Heretic-FP8Loaded in vLLM 0.30.0 on an RTX PRO 6000 Blackwell: ordinary prompts answered correctly and fluently (greedy output
matches the bf16 model almost word for word), and thinking mode reasons to the right answer on arithmetic and closes its
<think> block. Throughput: ~360 tok/s single-stream, ~7,500 tok/s aggregate at batch 32 (512-token generations,
measured on a shared GPU; about the same as bf16, since a model this small is not bandwidth-bound).
3/100 refusals against 54/100 for the original, KL divergence 0.0027. Method and parameters are on the parent card. The FP8 weights were not re-measured for refusals.
Reduced safety guardrails by design. You are responsible for what you do with it.
| Repository | Format | Size | Use it with |
|---|---|---|---|
| Qwen3-0.6B-Heretic | bf16 safetensors | 1.19 GB | transformers, vLLM, SGLang |
| Qwen3-0.6B-Heretic-GGUF | GGUF BF16 / Q8_0 / Q4_K_M | 1.20 / 0.64 / 0.40 GB | llama.cpp, Ollama |
| Qwen3-0.6B-Heretic-FP8 | FP8 W8A8, compressed-tensors | 0.76 GB | vLLM |
| Qwen3-0.6B-Heretic-NVFP4 | NVFP4, compressed-tensors | 0.57 GB | vLLM on Blackwell |