GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
grimlee
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
grimlee
Model Tree
huihui-ai/Huihui-Qwen3.8-27B-abliterated
Input Modalities
Output Modalities
Supported Functionality
The checkpoint uses the standard EXL3 format with a target body bitrate of 3.0 bits per weight.
The target bitrate applies to the quantized body; not every tensor in the checkpoint is stored at exactly 3 bits.
3.0 bpwlm_head: 6-bit6-bit4-bit2502048mul1always4096 MBsafetensors shardsThe four shards are the expected result of the 4096 MB shard-size setting
and do not indicate a missing or incomplete model.
The checkpoint metadata reports:
quant_method=exl3version=1.5.0bits=3.0head_bits=6vision_bits=6mtp_bits=4The conversion configuration was approximately:
bash
python convert.py \ -i "$SRC" \ -o "$OUT" \ -w "$WORK" \ -b 3.0 \ -hb 6 \ -mb 4 \ -vb 6 \ -cr 250 \ -cc 2048 \ -cb mul1 \ --out_scales always \ -ss 4096 \ -cpi 120 \ -d 0This is a standard EXL3 checkpoint.
The model itself does not require the SM89 fork described below. The fork is an optional runtime optimization for Ada SM89 GPUs and does not modify the EXL3 checkpoint format.
Upstream ExLlamaV3:
https://github.com/turboderp-org/exllamav3
Experimental Ada / SM89 fork:
https://github.com/grimlee/exllamav3
SM89 optimization branch:
https://github.com/grimlee/exllamav3/tree/sm89-ada-optimizations
I profiled ExLlamaV3 1.5.0 prefill performance on an NVIDIA RTX 4060 Ti 16 GB (Ada SM89) and tested two small SM89-specific runtime specializations.
These results are specific to the tested GPU, model shapes and runtime configuration. Other Ada / SM89 GPUs and model shapes have not yet been characterized to the same extent.
The tested SM89 F16ACC layout is:
text
BM=128BN=64BK=644 warps (2x2)STAGES=2GROUP_M=16PAD=0Matched cold-prefill server A/B results:
| Context | Control | SM89 F16ACC | Uplift |
|---|---|---|---|
| 16K | 727.38 tok/s | 761.64 tok/s | +4.710% |
| 32K | 652.37 tok/s | 679.31 tok/s | +4.129% |
| 64K | 535.73 tok/s | 554.60 tok/s | +3.522% |
Correctness was checked on all 16 F16ACC shapes observed in the real workload using three deterministic seeds per shape.
Results:
max_abs = 0mean_abs = 0max_relative = 0A per-shape oracle / dispatcher was also evaluated. It projected only about
1.8 tok/s beyond the simpler global SM89 configuration, so the global
configuration was retained.
The oracle number is a projection, not an additional end-to-end server measurement.
The second specialization targets the staged/cache-backed FP16 long-query prefill path.
The direct packed quantized-cache path keeps the upstream behavior.
Tested configuration:
text
Control:BM64 / BN32 / W8 / S2SM89 candidate:BM64 / BN32 / W4 / S2The key change is therefore:
text
8 warps -> 4 warpsThe F16ACC specialization above was enabled in both A/B arms, so the following experiment isolates the paged-attention change.
Formal cold-request server results:
| Context | Control | SM89 optimized | Uplift |
|---|---|---|---|
| 16K | 763.530 tok/s | 787.156 tok/s | +3.094% |
| 32K | 680.912 tok/s | 723.184 tok/s | +6.208% |
| 64K | 555.397 tok/s | 613.711 tok/s | +10.500% |
The formal requests used:
text
cached_tokens=0cached_pages=0so these are cold-prefill measurements rather than prefix-cache reuse results.
Correctness gates at 16K, 32K and 64K all passed with:
text
max_abs=0No measured VRAM increase was observed.
Peak VRAM:
| Context | Peak VRAM |
|---|---|
| 16K | 14,488 MiB |
| 32K | 14,488 MiB |
| 64K | 14,608 MiB |
During the formal A/B runs there was no observed:
The reported runtime measurements used:
| Setting | Value |
|---|---|
| GPU | NVIDIA RTX 4060 Ti 16 GB |
| Architecture | Ada SM89 / compute capability 8.9 |
| ExLlamaV3 | 1.5.0 |
| Upstream base commit | 02aef45cd681b960a00afcd0749a4ab99e6c1bfe |
| Model | Qwen3.8-27B EXL3 3.0 bpw |
| KV cache | int4 |
| QC staging | EXL3_QC_STAGING=1 |
| Maximum prefill chunk |
Prefill throughput in the tables is the server's measured prefill timing, rather than a client-side prompt-token / TTFT approximation.
These measurements should not be interpreted as a guarantee of identical performance gains across every Ada GPU, model shape, batch size, cache mode or context length.
The checkpoint retains the multimodal vision structure and its quantization
metadata reports vision_bits=6. Vision was disabled during these performance
benchmarks, so the results above should not be interpreted as a complete
vision-runtime compatibility test.
SM89 branch:
https://github.com/grimlee/exllamav3/tree/sm89-ada-optimizations
Detailed benchmark notes:
https://github.com/grimlee/exllamav3/blob/sm89-ada-optimizations/doc/sm89_ada_optimizations.md
Full GitHub Actions build matrix:
https://github.com/grimlee/exllamav3/actions/runs/35473907070
Upstream discussion:
https://github.com/turboderp-org/exllamav3/issues/384
The referenced GitHub Actions run completed successfully with:
text
target=allrelease=0This validates the source and wheel build matrix. It does not replace the GPU correctness testing and server A/B measurements described above.
For normal use, upstream ExLlamaV3 can be used directly:
https://github.com/turboderp-org/exllamav3
For the optional RTX 4060 Ti / SM89 optimized runtime:
bash
git clone https://github.com/grimlee/exllamav3.gitcd exllamav3git checkout sm89-ada-optimizationsThe SM89 branch is an experimental runtime optimization.
It is not required to load this model and does not introduce a different model format.
This repository only provides an EXL3 quantization and the runtime benchmark work documented above.
For information about:
please refer to:
huihui-ai/Huihui-Qwen3.8-27B-abliterated
https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated
This quantization does not introduce separate safety tuning beyond the source model.
| 4096 tokens |
| MTP | enabled |
| Vision | disabled for the performance A/B |
| Request type | cold requests |
| Prefix cache | cached_tokens=0 |