GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
megafetis
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
megafetis
Model Tree
Qwen/Qwen3-Reranker-4B
Input Modalities
Output Modalities
Supported Functionality
W8A8:
weights int8 symmetric per-channel (static), activations int8 symmetric per-token (dynamic).lm_head is not quantized — vLLM builds the classifier from its yes/no rows. The last 4 decoder layers
(32–35) are kept in bf16 (max |ΔP(yes)| to bf16: 0.022 → 0.008).| Metric | Value |
|---|---|
| |P(yes) − P_bf16(yes)|, mean / max (80 pairs) | 0.0003 / 0.0081 |
| Nonsense query, max P(yes): W8A8 / bf16 | 0.0002 / 0.0002 |
The numbers come from the quantized model in transformers (simulated quantization) right after calibration.
bash
vllm serve megafetis/Qwen3-Reranker-4B-W8A8 --runner pooling --served-model-name rr4 \ --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \ --chat-template qwen3_reranker.jinja --max-model-len 8192 --enable-prefix-caching# RTX 2080 Ti (no bf16): add --dtype halfqwen3_reranker.jinja — from vLLM examples/pooling/score/template/. Without the template scores are meaningless.
Endpoint — /v1/rerank (Jina format) or /score.
Recipe — recipe.yaml; the scripts — ai-models/ of the project that produced it. License — Apache 2.0, as the base model.