GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sakamakismile
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sakamakismile
Model Tree
Nanbeige/Nanbeige4.3-33B
Input Modalities
Output Modalities
Supported Functionality
On eight RTX PRO 2000 Blackwell GPUs with vLLM 0.32.0 and TP8, three English prose prompts produced pooled end-to-end output throughput of 64.78 tok/s with plain NVFP4 and 53.66 tok/s with optional DFlash2 (0.83x across these three requests). These are capped response prefixes with one concurrent request, temperature 0 and thinking off, one response per case. See the per-case table under English prose throughput.
For English prose on this tested setup, use plain NVFP4 decoding as the default: DFlash2 was slower in all three cases.
The English coding example is a separate workload: 63.82 to 120.20 end-to-end output tok/s (1.88x) in one 512-token example. DFlash2's benefit depends on the prompt and concurrency. The six reasoning sanity checks use Japanese prompts and are reported separately; the English prose measurements evaluate speed, not accuracy or general reasoning quality.
Heretic, Arbitrary-Rank Ablation (ARA), rank 50, 5 outer L-BFGS steps, row magnitude preservation. Twenty search slots were used, including interrupted and pruned slots and one duplicate caused by a restart; this is not twenty unique completed candidates.
Selected Optuna trial 5 (display trial 6): layers [12,36), preserve weight 0.9, steering weight 0.0003, relative overcorrection 0.8, 7 neighbors. Modification data: 400 ordinary and 400 harmful behavior prompts, evaluated on separate 100-prompt splits of each pinned dataset.
The selected candidate had 12/100 keyword-detected refusals and first-token KL divergence 0.006947 on the search evaluation prompts. The official source had 99/100 keyword-detected refusals. The zero-refusal candidates changed the ordinary-response distribution much more, so they were not selected. Keyword rates and first-token KL do not establish general reasoning quality or correctness.
Dataset revisions: mlabonne/harmless_alpaca at 02c6a92cfcf11bb0c387334f8146d149d65b587f; mlabonne/harmful_behaviors at 01cead01398926d81f7c52bdb790ee8cf77ebba7.
AutoRound 0.10.2 through llmcompressor 0.11.0, NVFP4 W4A4, 200 iterations per decoder block, effective optimization batch size 8 (micro-batch 2 with 4 gradient accumulation steps). Embeddings, normalization and lm_head remain BF16.
Calibration uses neuralmagic/calibration, LLM train split, revision fb6bc2f8c66543876fb31613f5872b9030220e15. Seed 20261026, 512 contiguous sequences of 2048 real tokens (1,048,576 tokens, no padding), packed from 2,713 shuffled conversations. Packed token digest: 6e47f66a2c414012ea9d8b363b5e8b03e6f045f66fd5198d3717f83c7dda578b. Calibration text is not distributed with the weights.
Calibration decoder-input captures, reference outputs and best-parameter caches were offloaded to CPU to fit the eight-GPU environment (about 15.5 GiB visible memory per GPU). Tuning used micro-batches of 2 with 4 gradient accumulation steps, preserving effective batch size 8 and 200 optimizer updates per decoder block.
recipe.yaml records the llmcompressor modifier settings. recipe.json also records the CPU-cache policy and native AutoRound gradient-accumulation overrides applied by the quantization wrapper.
Quantization toolchain: Python 3.14.4, Torch 2.11.0+cu128, Transformers 5.9.0, AutoRound 0.10.2, llmcompressor 0.11.0, compressed-tensors 0.16.0. These versions were inspected in the quantization environment using package metadata without importing model libraries or accessing GPUs. The Heretic/BF16-export package versions are recorded separately in recipe.json.
Official BF16 and locally processed BF16 both answered six small, deterministic sanity tasks correctly, with a 16,384-token generation ceiling. These simple checks are not a standardized reasoning benchmark. AIME was not repeated for this local artifact.
All three models use the same vLLM TP8 speed-test settings, including FP8 KV cache; BF16 labels describe weight precision. NCCL_P2P_DISABLE=1 and NCCL_CUMEM_ENABLE=0 are set, and custom all-reduce is disabled. MTP/speculative decoding was not enabled. Prefix caching is enabled and reset before each measured condition. Speed tests disable thinking and use one warmup request for decode, with no warmup requests for the one-output-token prefill measurement. The six thinking-enabled sanity tasks use a separate 16,384-token generation ceiling. Full server and request settings are recorded in recipe.json.
BF16 measurements on eight RTX PRO 2000 Blackwell GPUs, vLLM 0.32.0 TP8, 32 requests per concurrency, mean actual input 2081.375 tokens and output 128 tokens:
| Concurrency | Local BF16 decode tokens/s | Local BF16 aggregate output tokens/s |
|---|---|---|
| 1 | 27.05 | 22.82 |
| 2 | 23.71 | 37.92 |
| 4 | 22.63 | 60.63 |
| 8 | 16.78 | 85.71 |
Decode tokens/s is 1000 divided by mean time per output token after the first token. Aggregate throughput includes request processing and prefill time. Each condition was measured once. End-to-end input throughput with 1 output token and concurrency 8 was 2369.07 input tokens/s for local BF16; it is not a kernel-only prefill metric.
Sanity checks:
| Model | Correct |
|---|---|
| source | 6/6 |
| bf16 | 6/6 |
| nvfp4 | 6/6 |
NVFP4 measurements (same conditions):
| Concurrency | Decode tokens/s | Aggregate output tokens/s |
|---|---|---|
| 1 | 66.61 | 53.71 |
| 2 | 59.62 | 89.96 |
| 4 | 53.41 | 130.96 |
| 8 | 36.12 | 174.62 |
NVFP4 end-to-end input throughput with one output token: 4324.79 input tokens/s.
The final shard digests are in SHA256SUMS. The upstream model declares Apache-2.0; retain upstream attribution when redistributing.
The license text is included in LICENSE.
The model retains the upstream tokenizer and chat template. trust_remote_code=False is used. The modified model intentionally changes refusal behavior and may produce inaccurate or harmful outputs. Inference results depend on the runtime, hardware, context and quantization.
This NVFP4 target also ran with the separate ProCreations/Nanbeige4.3-33B-DFlash2 draft using vLLM 0.32.0. The target weight file is unchanged. Draft weights are downloaded separately from their author and are not included here. See the author's model card and license notices.
Three original English prose prompts were measured with one concurrent request (C1): an explanation, an original story opening, and a balanced essay. Each arm generated one response per prompt, after a 16-token same-prompt warmup and prefix reset. Temperature 0, thinking off, 512-token generation cap. The prompts request long-form prose, so these measurements time the beginning of each response, not complete essays. The same prompts and runtime settings are used for both arms.
| English prose case | Plain end-to-end output tok/s | DFlash2 end-to-end output tok/s | Ratio | Mean accepted length incl. bonus |
|---|---|---|---|---|
| Expository prose | 64.96 | 53.21 | 0.82x | 1.35 |
| Fiction | 64.62 | 55.25 | 0.86x | 1.40 |
| Argumentative prose | 64.75 | 52.59 | 0.81x | 1.33 |
| Pooled across these three requests | 64.78 | 53.66 | 0.83x | — |
For English prose on this tested setup, use plain NVFP4 decoding as the default: DFlash2 was slower in all three cases.
Pooled throughput is total generated tokens divided by the sum of API response times. The main rate includes prompt processing, first-token wait and streaming. This is one response per case, with no confidence interval; it does not establish an English-wide speedup or writing-quality result. English prose is reported separately from the coding example and random-input tests. english-prose-summary.json records the prompts, input/output token counts, finish reasons, TTFT, token-ID-based decode rates and acceptance counters.
For TP8, the original draft's 28 query heads cannot be divided across eight ranks. The supplied CPU helper inserts one zero query head per KV group and the corresponding zero output-projection columns (28 to 32 heads, KV4, head dimension 128). It verifies all 45 unchanged tensors and the retained bytes of the six projection tensors, plus every inserted zero. This preserves the GQA-to-O mapping in exact arithmetic; floating point output identity is not guaranteed. The helper does not retrain the draft or modify the target.
bash
hf download ProCreations/Nanbeige4.3-33B-DFlash2 \ --revision 22a73c4d099e61a34b6a26251b0a3a98a7b41746 --local-dir ../Nanbeige4.3-33B-DFlash2# Run in an environment containing torch and safetensors. CUDA is disabled by the helper.python dflash2/pad_query_heads.py \ --source ../Nanbeige4.3-33B-DFlash2 --output ../Nanbeige4.3-33B-DFlash2-TP8 NCCL_P2P_DISABLE=1 NCCL_CUMEM_ENABLE=0 vllm serve . \ --tensor-parallel-size 8 --max-model-len 32768 --max-num-seqs 8 \ --max-num-batched-tokens 8192 --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 --disable-custom-all-reduce --enable-prefix-caching \ --speculative-config '{"method":"dflash","model":"../Nanbeige4.3-33B-DFlash2-TP8","num_speculative_tokens":7,"draft_tensor_parallel_size":8,"quantization":null,"kv_cache_dtype":"bfloat16"}'Use a new output directory for the helper, separate from the target repository. The source is pinned by SHA256. The original author uses a block of eight positions: one verified anchor plus seven proposed tokens, so vLLM's num_speculative_tokens is 7. In the tested run, the target used native FlashInfer CUTLASS NVFP4/FLASHINFER attention, and the draft used BF16/FLASH_ATTN attention with BF16 KV. The plain target works without this optional setting.
On eight RTX PRO 2000 Blackwell cards, same random-prompt conditions as the plain run, 32 requests per condition, one sweep:
| Concurrency | Plain decode tok/s | DFlash2 decode tok/s | Plain aggregate tok/s | DFlash2 aggregate tok/s |
|---|---|---|---|---|
| 1 | 66.61 | 60.04 | 53.71 | 48.82 |
| 2 | 59.62 | 43.35 | 89.96 | 73.17 |
| 4 | 53.41 | 29.46 | 130.96 | 98.48 |
| 8 | 36.12 | 18.54 | 174.62 | 117.88 |
DFlash2 end-to-end input throughput with one output token: 3962.43 input tokens/s. The six thinking-enabled sanity cases use Japanese prompts and passed 6/6 with a 16,384-token cap. This is a small check, not evidence of general capability equivalence. Decode tests use 128 output tokens and thinking disabled. Aggregate output throughput includes prefill and request processing. The input rate includes HTTP, queuing and the first output token. DFlash2 was slower at every concurrency in this random-prompt bank; use plain decoding for those conditions. Workload, concurrency and target changes affect acceptance and speed. These results do not predict other machines or prompts. See measurement-summary.json for acceptance counters and exact conditions.
Four mixed-language natural prompts were also measured at C1, with one response per arm and case after a short same-prompt warmup and prefix reset. Temperature 0 and thinking off. All eight responses reached the 512-token cap: these are generation-prefix timings, not the time to complete the requested task. Outputs can differ. These are exploratory latency measurements, not graded accuracy tests or domain-wide speed claims. The main metric below is output tokens divided by the full API response time. Streaming decode measurements use actual delta token IDs and exclude every token in the first chunk, rather than assuming one token per chunk.
| Case | Plain end-to-end output tok/s | DFlash2 end-to-end output tok/s | Ratio | Mean accepted length incl. bonus |
|---|---|---|---|---|
| Coding (English prompt) | 63.82 | 120.20 | 1.88x | 2.63 |
| Math (Japanese prompt) | 63.75 | 65.14 | 1.02x | 1.42 |
| Japanese explanation | 63.81 | 62.61 | 0.98x | 1.36 |
| API troubleshooting (English prompt) | 64.01 | 70.64 | 1.10x | 1.53 |
See natural-prompt-summary.json for the prompts, token counts, decode measurements, finish reasons and acceptance.
hf download ProCreations/Nanbeige4.3-33B-DFlash2 \ --revision 22a73c4d099e61a34b6a26251b0a3a98a7b41746 --local-dir ../Nanbeige4.3-33B-DFlash2# Run in an environment containing torch and safetensors. CUDA is disabled by the helper.python dflash2/pad_query_heads.py \ --source ../Nanbeige4.3-33B-DFlash2 --output ../Nanbeige4.3-33B-DFlash2-TP8 NCCL_P2P_DISABLE=1 NCCL_CUMEM_ENABLE=0 vllm serve . \ --tensor-parallel-size 8 --max-model-len 32768 --max-num-seqs 8 \ --max-num-batched-tokens 8192 --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 --disable-custom-all-reduce --enable-prefix-caching \ --speculative-config '{"method":"dflash","model":"../Nanbeige4.3-33B-DFlash2-TP8","num_speculative_tokens":7,"draft_tensor_parallel_size":8,"quantization":null,"kv_cache_dtype":"bfloat16"}'