GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
InfiniCloud
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
InfiniCloud
Model Tree
llm-jp/llm-jp-4.1-33b-thinking
Input Modalities
Output Modalities
Supported Functionality
de581ee4353568c480d47129948d1be99f02f4fftrain_sft conversations, maximum 2048 tokens,
seed 42, reasoning effort medium, date 2026-09-29, dataset revision
8049631c405ae6576f93f445c6b8166f76f5505a.
Calibration was not specialized for Japanese or reasoning tasks.[2,13] is preserved.
No legacy LLM-jp 4 tokenizer compatibility patch is applied.Japanese MGSM, 250 questions, 5-shot, low reasoning effort, temperature 0, seed 42, maximum 4096 generated tokens. Official flexible-extract scoring is applied only to complete final answers; incomplete answers score zero.
| Precision | MGSM correct / 250 | Accuracy | Complete final | Smoke low / medium (3 each) | Weights GiB |
|---|---|---|---|---|---|
| BF16 | 203 | 81.2% | 248 | 2/3・2/3 | 61.876 |
| FP8-Dynamic | 212 | 84.8% | 250 | 2/3・2/3 | 32.823 |
| GPTQ-INT4-G64 | 203 | 81.2% | 249 | 2/3・3/3 | 19.191 |
| NVFP4 | 203 | 81.2% | 248 | 2/3・2/3 | 20.099 |
The FP8-Dynamic row was measured on 2026-10-03 with the same procedure and environment (separate release). Its higher count comes from one fixed-condition run and does not establish an advantage over BF16.
GPTQ and NVFP4 have equal correct-answer counts but differ on individual questions. One arithmetic benchmark does not establish general quality equivalence or superiority. See methodology and limitations for smoke-test failures and generation-limit terminations.
RTX PRO 6000 Blackwell Max-Q 96GB; vLLM 0.29.0; BF16 KV cache 8 GiB; context 16K; max_num_seqs 4; CUDA Graphs and prefix caching disabled. Random inputs and 256 output tokens, ignoring EOS; medians of three runs of 32 requests per condition. Percentiles are also medians of per-run percentiles.
| Input tokens | Concurrency | Output tok/s | TTFT p50 / p95 ms | TPOT p50 / p95 ms |
|---|---|---|---|---|
| 512 | 1 | 15.5 | 135.2 / 170.8 | 62.37 / 68.97 |
| 512 | 4 | 60.8 | 230.0 / 251.6 | 66.11 / 68.90 |
| 4096 | 1 | 15.1 | 419.0 / 424.1 | 64.91 / 69.37 |
| 4096 | 4 | 56.8 | 1176.8 / 1399.4 | 66.18 / 72.28 |
Output tok/s of all four formats under the same eager settings (FP8 measured on 2026-10-03):
| Input tokens | Concurrency | BF16 tok/s | FP8 tok/s | GPTQ tok/s | NVFP4 tok/s |
|---|---|---|---|---|---|
| 512 | 1 | 19.8 | 35.6 | 41.1 | 15.5 |
| 512 | 4 | 78.0 | 135.8 | 149.4 | 60.8 |
| 4096 | 1 | 18.0 | 31.8 | 34.8 | 15.1 |
| 4096 | 4 | 57.7 | 94.4 | 92.1 | 56.8 |
GPTQ had higher generation throughput in this setup; NVFP4 had lower TTFT for longer inputs. The completed CUDA Graph follow-up is reported below. These are not maximum-performance claims for Blackwell or NVFP4.
Memory bandwidth, kernel implementations, and overhead from disabling CUDA Graphs may contribute to the observed speed differences. Their individual contributions have not been isolated.
Same GPU and inference packages; BF16 KV cache 40 GiB, max_num_seqs 16, context 16K, max_num_batched_tokens 4096, chunked prefill enabled, prefix caching disabled. Both torch.compile and FULL_AND_PIECEWISE CUDA Graphs are enabled. Each condition uses 32 requests x 3 runs, random inputs, 256 output tokens, seed 42, and at least max(4, concurrency) warmups per run. Values are medians; p50/p95 are medians of the per-run percentiles.
| Input tokens | Concurrency | Output tok/s | TTFT p50 / p95 ms | TPOT p50 / p95 ms |
|---|---|---|---|---|
| 512 | 1 | 59.3 | 154.3 / 163.1 | 16.34 / 16.39 |
| 512 | 4 | 229.3 | 285.7 / 301.1 | 16.40 / 16.48 |
| 512 | 16 | 762.9 | 563.6 / 774.0 | 18.74 / 19.55 |
| 4096 | 1 | 53.7 | 418.4 / 424.7 | 17.06 / 17.11 |
| 4096 | 4 | 159.0 | 1190.6 / 1560.9 | 20.46 / 23.54 |
| 4096 | 16 | 298.5 | 1642.6 / 6157.0 | 46.94 / 49.41 |
| 8192 | 1 | 46.7 | 940.3 / 955.3 | 17.84 / 17.86 |
| 8192 | 4 | 111.6 | 2462.8 / 3210.4 | 26.15 / 30.73 |
| 8192 | 16 | 160.4 | 2467.5 / 13970.8 | 88.50 / 92.13 |
Output tok/s of all four formats with graph + compile. BF16 used a 24 GiB KV cache because 40 GiB does not fit next to its weights on one 96 GB GPU; no run had KV preemption, so the smaller pool does not change the computation (BF16 measured on 2026-10-04). — marks input 8192 x concurrency 16, which does not fit in BF16 (about 33 GiB of KV).
| Input tokens | Concurrency | BF16 tok/s | FP8 tok/s | GPTQ tok/s | NVFP4 tok/s |
|---|---|---|---|---|---|
| 512 | 1 | 20.0 | 38.5 | 60.6 | 59.3 |
| 512 | 4 | 78.8 | 146.9 | 216.7 | 229.3 |
| 512 | 16 | 264.3 | 489.9 | 556.0 | 762.9 |
| 4096 | 1 | 18.2 | 34.3 | 45.8 | 53.7 |
| 4096 | 4 | 58.0 | 101.2 | 101.0 | 159.0 |
| 4096 | 16 | 119.4 | 194.0 | 140.5 | 298.5 |
| 8192 | 1 | 16.4 | 30.1 | 35.4 | 46.7 |
| 8192 | 4 | 43.9 | 72.7 | 61.3 | 111.6 |
| 8192 | 16 | — | 107.9 | 72.9 | 160.4 |
For input 8192 and concurrency 16, NVFP4 achieved 2.20x GPTQ's aggregate output throughput in this model family (160.4 vs 72.9 tok/s). This result does not imply a speed advantage for every input length or concurrency.
Matched eager controls use the same 40 GiB KV cache and max_num_seqs 16:
| Input tokens | Concurrency | Eager tok/s | Graph + compile tok/s |
|---|---|---|---|
| 512 | 1 | 15.3 | 59.3 |
| 512 | 4 | 59.6 | 229.3 |
| 4096 | 1 | 15.0 | 53.7 |
| 4096 | 4 | 56.7 | 159.0 |
Across all four models, all 156 runs / 4,992 measured requests passed validation, with zero KV preemptions. Graph capture completion was confirmed for all four graph servers; per-request replay traces were not collected. MGSM was not rerun with graph execution. See follow-up methodology, all model results, and audit. The comparison combines compilation and CUDA Graph effects and is not a claim of maximum performance or a separate measurement of either effect.
bash
bash reproduce/serve-cudagraph.sh InfiniCloud/llm-jp-4.1-33b-thinking-NVFP4Pass eager as the second argument to reproduce the matched eager server settings.
Tested inference: vLLM 0.29.0, compressed-tensors 0.17.0, Transformers 5.14.1, PyTorch 2.13.0, CUDA 13.0, driver 580.178.04. Native FlashInfer CUTLASS NVFP4 kernels were verified on Blackwell. Other GPUs, ordinary Transformers model loading, 64K context, and application-specific JSON schema compliance were not validated.
bash
bash reproduce/serve.sh InfiniCloud/llm-jp-4.1-33b-thinking-NVFP4This command reproduces the original eager settings (KV 8 GiB, max_num_seqs 4).
trust_remote_code is optional. Without it, Transformers and vLLM load the
tokenizer from tokenizer.json with the same token IDs; with it, the supplied
Llmjp4Tokenizer is used, so review its code first. reproduce/serve.sh keeps
the flag used for the published measurements. Applications must separate
Harmony analysis/final channels and use the supplied tokenizer, not direct
openai-harmony tokenization. See the upstream cookbook.
Full numerical verification checked finite tensors, positive scales, unchanged retained BF16 weights and preserved tokenizer behavior. See verification.json, release inventory, and quantization reproduction instructions.
tokenizer_config.json, tokenizer.json, and the added
special_tokens_map.json; the tokenizer code and chat template were
already identical). The 2026-09-30 upload carried a re-saved
tokenizer_config.json whose tokenizer_class named the custom
Llmjp4Tokenizer, so loaders that do not run remote code (for example vLLM without
--trust-remote-code) could not load the 33B model.
Weights, token IDs, the chat template and all published results are unchanged.Distributed under Apache License 2.0, as declared by the source model card. LLM-jp 4.1 was developed by the Research and Development Center for Large Language Models at Japan's National Institute of Informatics. See NOTICE for changes and attribution, and the source model card for training data, risks, acknowledgements and contact information.