GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
zenmagnets
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
zenmagnets
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
| Profile | Correct | Accuracy | ECE | Mean NLL | Mean HTTP latency |
|---|---|---|---|---|---|
| FP8 SM120 | 19,012/23,900 | 79.548% | 0.069079 | 0.699128 | 0.353 s |
| BF16 original | 19,035/23,900 | 79.644% | 0.069088 | 0.693885 | 0.555 s |
Both runs completed all 23,900 rows with no failures, exclusions, or truncation. FP8 SM120's mean request latency was 36.4% lower at concurrency 1. ECE is nearly identical; the BF16 original had slightly better mean NLL. The two profiles selected different answers on 324 rows; BF16 was correct on 148 rows where FP8 SM120 was not, and FP8 SM120 was correct on 125 where BF16 was not.
| Input tokens | Rows | FP8 SM120 | BF16 original |
|---|---|---|---|
| <=256 | 5,667 | 46.3 ms | 65.8 ms |
| 257-1,024 | 12,247 | 87.3 ms | 139.9 ms |
| 1,025-4,096 | 4,284 | 364.4 ms | 609.1 ms |
| 4,097-16,384 | 1,183 | 1.551 s | 2.510 s |
The suite used four option rotations and one request at a time. These are end-to-end HTTP latencies, not concurrent throughput. The full report includes matched row and request verification, per-task results, and latency distributions in benchmarks/decisionbench-full.json.
The top-choice accuracy and aggregate calibration metrics do not describe every probability equally closely. Against BF16, candidate probabilities had median absolute difference 0.00000462, p99 0.02511, and maximum 0.36081. In 218/23,900 rows (0.91%), at least one candidate probability differed by 0.10 or more. Probability-difference counts are in the full report.
This package is optimized and benchmarked on one RTX PRO 6000 Blackwell (SM120, 96 GB). NVIDIA's compute_120f family target covers compute capability 12.0 and 12.1, so the kernels are expected to be portable and perform well on SM121 too. SM121 runtime performance has not been measured here. See NVIDIA's compute-capability compatibility table.
The measured process-memory peak was 19,968 MiB, sampled once per second during warmup and the full suite. The GPU power limit was 395 W and was not changed by the launcher. Allow at least 32 GiB of free VRAM for a comfortable margin; larger image inputs may need more.
No new model training or calibration fitting was performed. The adapter is not merged into the FP8 base. This package is independent of the original model authors.
Use Docker, NVIDIA Container Toolkit, and a driver compatible with CUDA 13.0. Python 3.11 or newer is needed for the included verification and client scripts.
Download the repository, including adapter/ and runtime/:
bash
hf download zenmagnets/Imajev-4B-FP8-SM120 --local-dir ./imajev-4b-fp8-sm120cd imajev-4b-fp8-sm120python3 runtime/verify_release.py .docker build -t imajev-fp8-sm120:local .docker run --rm --gpus device=0 --cpus 12 --shm-size 4g \ -p 127.0.0.1:8765:8765 \ -v "$PWD:/models:ro" \ -v imajev-fp8-cache:/root/.cache \ imajev-fp8-sm120:localThe first startup downloads a pinned Hugging Face kernel into the cache and compiles GPU kernels. Once that cache is populated, add -e HF_HUB_OFFLINE=1 to run offline. The model itself is loaded from the mounted local repository; no Hugging Face token is needed after download.
The optimized worker passed a native DecisionBench smoke, the packaged functional suite including vision, and the full DecisionBench run. See benchmarks/validation.json and benchmarks/release-validation.json for the checks and runtime versions.
Readiness is GET /v1/models; decisions use POST /v1/systemone. This typed-decision API does not expose /v1/chat/completions.
bash
curl -s http://localhost:8765/v1/modelscurl -s http://localhost:8765/v1/systemone \ -H 'Content-Type: application/json' \ -d '{ "state": {"ticket": "The same invoice was charged twice. Please refund the duplicate."}, "questions": { "department": { "type": "choice", "instructions": "Which department should handle this ticket?", "criteria": { "billing": "Payments and refunds", "technical_support": "Product failures", "sales": "New purchases" } } } }'Answers contain the selected value, probabilities, unknown_probability, and abstained. Supported question types are choice, noul, and score. Up to two images can be supplied as data URLs in the images list. See Imajev's API specification for the complete contract.
The included Python client sends text, one-image, or two-image requests:
bash
python3 examples/client.py --base-url http://localhost:8765python3 examples/client.py --base-url http://localhost:8765 --image ./product.jpgpython3 examples/client.py --base-url http://localhost:8765 --image ./first.jpg --image ./second.jpgThe upstream server handles GPU requests serially; simultaneous callers queue. The configured input limit is 65,536 tokens. CUDA graphs cover inputs up to 4,096 tokens; the longest request in the evaluation contained 39,111 tokens.
The tuned path combines FlashInfer's SM120 CUTLASS block-FP8 GEMM, a fused Triton activation quantizer, activation-quantization reuse within a decoder layer, BF16 LoRA arithmetic, FP32 decision readout, and vision-feature reuse across option rotations and questions.
The original MTP weights are preserved in the checkpoint but are not loaded or used by the decision-serving path.
| Component | Source and pinned revision |
|---|---|
| BF16 base | Qwen/Qwen3.5-4B |
| Imajev adapter, head, calibration | mohit67890/imajev-4b |
| Decision server | mohit67890/imajev |
| FP8 conversion recipe | Hob-forge/Qwen3.5-4B-FP8 |
| Fine-grained FP8 fallback kernel |
The base, Imajev code, and conversion recipe are Apache-2.0 licensed. See LICENSE, NOTICE, and MODEL_PROVENANCE.json. The adapter configuration's original training-machine path was replaced with its public base-model ID and pinned revision; the adapter, readout, and calibration tensor bytes are unchanged.
| >16,384 | 519 | 7.125 s | 10.772 s |
| kernels-community/finegrained-fp8 |