GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
hoja-solutions
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
hoja-solutions
Model Tree
jialinyyzz/humanizer
Input Modalities
Output Modalities
Supported Functionality
The comparison is against bf16 in vLLM 0.30.0 on the authors' held-out evaluation set: 209 English and 102 Chinese drafts, each followed by the authors' first bf16 rewrite, with every draft and rewrite token scored (187,551 positions in English, 99,166 in Chinese).
| Build | Top-1 vs. bf16, EN / ZH (avg) | KL to bf16, EN / ZH |
|---|---|---|
| bf16, second run in vLLM | 99.20% / 99.10% (99.15%) | 0.0000 / 0.0001 |
| NVFP4A16 | 95.03% / 94.51% (94.77%) | 0.0120 / 0.0121 |
| NVFP4 W4A4 (this build) | 90.86% / 89.25% (90.05%) | 0.0427 / 0.0500 |
KL is computed on bf16's 20 most likely tokens plus one bucket for the rest of the vocabulary, so it reads slightly lower than a full-vocabulary KL. On the authors' own scale (llama.cpp, same evaluation set, different tokens), 90% top-1 is where a standard 3-bit IQ3_M GGUF sits, and their quantization-aware Q4_K_M reaches 95.2%.
Over one rewrite of each of the 311 drafts, this build left a number out of 37 of the 102 Chinese rewrites. Two bf16 samples left one out of 28 and 31. In English it was within the bf16 range. Many of these are dates written another way, not wrong facts.
Most of the loss comes from the 4-bit activations. Simulating them on top of bf16 weights reproduces most of the drift (full-vocabulary KL 0.038 on 24 of the drafts). It comes from every projection type (attention 0.017, gate/up 0.014, down 0.009) and from every layer, with the later layers worst, so keeping a few layers at 16 bits does not fix it. Gemma 4's hidden states include a few values near 3,000 at the inputs of the first layer. A likely cause is that a group of 16 activations holding one of them shares its single scale with that value, so the other 15 round to zero.
All builds measured the same way, by top-1 average and KL (EN / ZH):
| Build | Top-1 | KL |
|---|---|---|
| W4A4, round to nearest | 86.89% | 0.0751 / 0.0950 |
| W4A4, importance-weighted scales (imatrix) | 88.25% | 0.0598 / 0.0739 |
| W4A4, GPTQ (this build) | 90.05% | 0.0427 / 0.0500 |
| W4A4, GPTQ, down_proj activations in bf16 | 90.70% | 0.0367 / 0.0421 |
| W4A4, GPTQ, attention activations in bf16 | 91.96% | 0.0275 / 0.0325 |
| W4A16, GPTQ with imatrix scales | 94.30% | 0.0146 / 0.0142 |
| W4A16, GPTQ (the NVFP4A16 build) | 94.77% | 0.0120 / 0.0121 |
RTX 5090, vLLM 0.30.0, 311 drafts, about 144,000 generated tokens:
| Build | One request at a time | 311 requests, up to 32 at once |
|---|---|---|
| bf16 | 59 tok/s | 501 tok/s |
| NVFP4A16 | 136 tok/s | 1,585 tok/s |
| NVFP4 W4A4 (this build) | 134 tok/s | 1,930 tok/s |
The usage is the same as for the NVFP4A16 build and the original model. Use the text-completion prompt from prompt_format.json on /v1/completions, or one draft per message on /v1/chat/completions with the bundled chat_template.jinja, which adds BOS to the authors' GGUF template. Sample with temperature 1.0, top-p 0.95 and top-k off, and no stop strings. generation_config.json sets those defaults (top_k: 0 instead of the original's 64).
bash
vllm serve hoja-solutions/humanizer-NVFP4 --served-model-name humanizer --max-model-len 8192 \ --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}'GPTQ with llm-compressor 0.14.0 (static activation ordering, block size 128, dampening 0.01) and the NVFP4 scheme on all 328 decoder projections. The activation group scales are computed at run time, under a per-tensor global scale fixed from the calibration data. The token embeddings (tied to the output layer) and the vision and audio projections stay in bf16. The calibration set (384 sequences, 365,486 tokens) is described in the NVFP4A16 card. It comes from train rows of jialinyyzz/humanizer-data, whose DATA.md lists source licenses and the terms of the models that wrote the drafts. The run took 8 minutes on an RTX 5090 and peaked at 20.9 GB of RSS.
Only tested on an RTX 5090 with vLLM 0.30.0, for text input.
The model, its training data and the evaluation set are by the humanizer authors (jialinyyzz/humanizer, sgaofen/humanizer-local-model), released under Apache-2.0. It is a finetune of google/gemma-4-12B.
vllm serve hoja-solutions/humanizer-NVFP4 --served-model-name humanizer --max-model-len 8192 \ --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}'