GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
InfiniCloud
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
InfiniCloud
Model Tree
llm-jp/llm-jp-4.1-33b-thinking
Input Modalities
Output Modalities
Supported Functionality
de581ee4353568c480d47129948d1be99f02f4ff[2,13] is preserved.Japanese MGSM, 250 questions, 5-shot, low reasoning effort, temperature 0, seed 42, maximum 4096 generated tokens. Official flexible-extract scoring is applied only to complete final answers; incomplete answers score zero. FP8 was measured on 2026-10-03 and the other rows on 2026-09-29, with the same procedure and environment (vLLM 0.29.0, the same GPU).
| Precision | MGSM correct / 250 | Accuracy | Complete final | Smoke low / medium (3 each) | Weights GiB |
|---|---|---|---|---|---|
| BF16 | 203 | 81.2% | 248 | 2/3・2/3 | 61.876 |
| FP8-Dynamic | 212 | 84.8% | 250 | 2/3・2/3 | 32.823 |
| GPTQ-INT4-G64 | 203 | 81.2% | 249 | 2/3・3/3 | 19.191 |
| NVFP4 | 203 | 81.2% | 248 | 2/3・2/3 | 20.099 |
FP8 answered 9 more questions than BF16 (it fixed 10 BF16 errors and missed 1 BF16 successes). This comes from one fixed-condition run and does not establish an advantage over BF16; small numerical differences can change greedy generation paths. One arithmetic benchmark does not establish general quality equivalence or superiority. See methodology and limitations.
RTX PRO 6000 Blackwell Max-Q 96GB; vLLM 0.29.0; BF16 KV cache 8 GiB; context 16K; max_num_seqs 4; CUDA Graphs and prefix caching disabled. Random inputs and 256 output tokens, ignoring EOS; medians of three runs of 32 requests per condition. Percentiles are also medians of per-run percentiles.
| Input tokens | Concurrency | Output tok/s | TTFT p50 / p95 ms | TPOT p50 / p95 ms |
|---|---|---|---|---|
| 512 | 1 | 35.6 | 105.0 / 106.4 | 26.89 / 31.06 |
| 512 | 4 | 135.8 | 360.6 / 365.3 | 27.61 / 31.10 |
| 4096 | 1 | 31.8 | 728.8 / 733.5 | 28.00 / 31.50 |
| 4096 | 4 | 94.4 | 2137.5 / 2811.5 | 35.27 / 39.45 |
Output tok/s of all four formats under the same eager settings (FP8 measured on 2026-10-03):
| Input tokens | Concurrency | BF16 tok/s | FP8 tok/s | GPTQ tok/s | NVFP4 tok/s |
|---|---|---|---|---|---|
| 512 | 1 | 19.8 | 35.6 | 41.1 | 15.5 |
| 512 | 4 | 78.0 | 135.8 | 149.4 | 60.8 |
| 4096 | 1 | 18.0 | 31.8 | 34.8 | 15.1 |
| 4096 | 4 | 57.7 | 94.4 | 92.1 | 56.8 |
These are not maximum-performance claims for Blackwell or FP8.
Same GPU and inference packages; BF16 KV cache 40 GiB, max_num_seqs 16, context 16K, max_num_batched_tokens 4096, chunked prefill enabled, prefix caching disabled. Both torch.compile and FULL_AND_PIECEWISE CUDA Graphs are enabled. Each condition uses 32 requests x 3 runs, random inputs, 256 output tokens, seed 42, and at least max(4, concurrency) warmups per run. Values are medians; p50/p95 are medians of the per-run percentiles.
| Input tokens | Concurrency | Output tok/s | TTFT p50 / p95 ms | TPOT p50 / p95 ms |
|---|---|---|---|---|
| 512 | 1 | 38.5 | 107.2 / 109.0 | 25.71 / 25.76 |
| 512 | 4 | 146.9 | 307.9 / 403.9 | 26.19 / 26.89 |
| 512 | 16 | 489.9 | 968.5 / 1350.1 | 29.24 / 31.89 |
| 4096 | 1 | 34.3 | 730.2 / 736.3 | 26.45 / 26.48 |
| 4096 | 4 | 101.2 | 2123.3 / 2454.0 | 31.37 / 36.60 |
| 4096 | 16 | 194.0 | 2892.2 / 10974.1 | 70.64 / 75.65 |
| 8192 | 1 | 30.1 | 1567.0 / 1582.8 | 27.22 / 27.26 |
| 8192 | 4 | 72.7 | 3835.8 / 5283.1 | 39.90 / 46.30 |
| 8192 | 16 | 107.9 | 4040.3 / 23328.6 | 130.39 / 136.34 |
Output tok/s of all four formats with graph + compile. BF16 used a 24 GiB KV cache because 40 GiB does not fit next to its weights on one 96 GB GPU; no run had KV preemption, so the smaller pool does not change the computation (BF16 measured on 2026-10-04). — marks input 8192 x concurrency 16, which does not fit in BF16 (about 33 GiB of KV).
| Input tokens | Concurrency | BF16 tok/s | FP8 tok/s | GPTQ tok/s | NVFP4 tok/s |
|---|---|---|---|---|---|
| 512 | 1 | 20.0 | 38.5 | 60.6 | 59.3 |
| 512 | 4 | 78.8 | 146.9 | 216.7 | 229.3 |
| 512 | 16 | 264.3 | 489.9 | 556.0 | 762.9 |
| 4096 | 1 | 18.2 | 34.3 | 45.8 | 53.7 |
| 4096 | 4 | 58.0 | 101.2 | 101.0 | 159.0 |
| 4096 | 16 | 119.4 | 194.0 | 140.5 | 298.5 |
| 8192 | 1 | 16.4 | 30.1 | 35.4 | 46.7 |
| 8192 | 4 | 43.9 | 72.7 | 61.3 | 111.6 |
| 8192 | 16 | — | 107.9 | 72.9 | 160.4 |
FP8 is faster than BF16 in every measured condition and falls between GPTQ and NVFP4 for long inputs at high concurrency. At concurrency 1 it is the slowest of the three quantized formats: its weights are about 1.7x the 4-bit releases, which matters when generation is memory-bandwidth bound. This does not rank the formats for every input length or concurrency.
Matched eager controls use the same 40 GiB KV cache and max_num_seqs 16:
| Input tokens | Concurrency | Eager tok/s | Graph + compile tok/s |
|---|---|---|---|
| 512 | 1 | 35.5 | 38.5 |
| 512 | 4 | 132.4 | 146.9 |
| 4096 | 1 | 31.5 | 34.3 |
| 4096 | 4 | 95.4 | 101.2 |
Across both FP8 models, all 78 runs / 2,496 measured requests passed validation, with zero KV preemptions. Graph capture completion was confirmed for both FP8 graph servers; per-request replay traces were not collected. MGSM was not rerun with graph execution. See follow-up methodology, all model results, and audit. The comparison combines compilation and CUDA Graph effects and is not a claim of maximum performance or a separate measurement of either effect.
bash
bash reproduce/serve-cudagraph.sh InfiniCloud/llm-jp-4.1-33b-thinking-FP8-DynamicPass eager as the second argument to reproduce the matched eager server settings.
Tested inference: vLLM 0.29.0, compressed-tensors 0.17.0, Transformers 5.14.1, PyTorch 2.13.0, CUDA 13.0, driver 580.178.04. Native CUTLASS FP8 (W8A8) linear kernels were verified on Blackwell. Other GPUs, ordinary Transformers model loading, 64K context, and application-specific JSON schema compliance were not validated.
bash
bash reproduce/serve.sh InfiniCloud/llm-jp-4.1-33b-thinking-FP8-DynamicThis command reproduces the eager settings above (KV 8 GiB, max_num_seqs 4).
trust_remote_code is optional. Without it, Transformers and vLLM load the
tokenizer from tokenizer.json with the same token IDs; with it, the supplied
Llmjp4Tokenizer is used, so review its code first. reproduce/serve.sh keeps
the flag used for the published measurements; vLLM 0.29.0 was also checked to
start and answer without it. Applications must separate
Harmony analysis/final channels and use the supplied tokenizer, not direct
openai-harmony tokenization. See the upstream cookbook.
Full numerical verification checked finite tensors, positive scales, unchanged retained BF16 weights, the FP8 dequantization error (relative, mean 2.66%, max 2.68%) and tokenizer files identical to the source. See verification.json, release inventory, and quantization reproduction instructions.
Distributed under Apache License 2.0, as declared by the source model card. LLM-jp 4.1 was developed by the Research and Development Center for Large Language Models at Japan's National Institute of Informatics. See NOTICE for changes and attribution, and the source model card for training data, risks, acknowledgements and contact information.