GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Hob-forge
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Hob-forge
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
Serve (tested — vLLM 0.19.0, RTX 5070 12 GB; vLLM downloads the weights automatically):
bash
vllm serve Hob-forge/Qwen3.5-4B-FP8 --max-model-len 4096 --gpu-memory-utilization 0.75
Or in Python:
python
from vllm import LLM, SamplingParamsllm = LLM(model="Hob-forge/Qwen3.5-4B-FP8", max_model_len=4096, gpu_memory_utilization=0.75)out = llm.chat([{"role": "user", "content": "Hello!"}],SamplingParams(temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5, max_tokens=512),)print(out[0].outputs[0].text)
To just fetch the weights:
bash
hf download Hob-forge/Qwen3.5-4B-FP8
Hardware: FP8 GEMM needs a recent NVIDIA GPU (Ada / Hopper / Blackwell, sm89+); ~6 GB VRAM for weights plus KV cache — a 12 GB card runs it comfortably at 4K context. On an 8 GB card, CPU, or modest RAM? FP8 is not the right artifact for you — use our GGUF edition instead: Hob-forge/Qwen3.5-4B-Instruct-GGUF.
Mirrors Qwen/Qwen3.5-27B-FP8 exactly:
| modules | |
|---|---|
| FP8 block-128 | language self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj, linear_attn.{in_proj_qkv, in_proj_z, out_proj}, MTP-block attention + MLP projections |
| kept BF16 | embeddings (tied lm_head), linear_attn.{conv1d, in_proj_a, in_proj_b}, the entire vision tower, mtp.fc, all norms |
207 weight tensors quantized, 531 kept bit-identical to upstream.
A/B against BF16 (same box, same inputs, vLLM 0.19.0, teacher-forced):
| measurement | BF16 | FP8 | delta |
|---|---|---|---|
| PPL, prose (Austen excerpt, 4776 tok) | 15.141 | 15.174 | +0.22% |
| PPL, prose (Shelley excerpt, 3741 tok) | 10.173 | 10.256 | +0.81% |
| PPL, code (Python, 2800 tok) | 2.859 | 2.864 | +0.19% |
| 12-item exact-answer QA (greedy) | 12/12 | 12/12 | parity |
Greedy 64-token continuations match BF16 token-for-token 61% of the time (7/20 prompts identical throughout) — normal for any quantization: near-tie top-2 tokens flip and paths diverge. Every diverged continuation was manually inspected and is coherent. This is a small probe, not a benchmark suite — for production decisions run your own evals.
Use upstream's recommendations (from the Qwen3.5-4B card):
temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5temperature=0.6, top_p=0.95, top_k=20temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5Tested only on Blackwell (RTX 5070) under vLLM; other stacks (SGLang, transformers fine-grained-FP8) load this format but are untested by us — reports welcome.
Also requested in #27 — under evaluation. NVFP4 needs a calibration pass and currently serves only on Blackwell-class hardware; if we can produce and validate one honestly, it will appear as a sibling repo.
Hob Forge — small household spirits for small machines. We set up small models correctly for small GPUs and modest RAM.
Quantized with the included script; Apache-2.0, upstream LICENSE retained.