GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
fleetml
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
fleetml
Model Tree
orcarouter/Qwen3.8-27B-Uncensored
Input Modalities
Output Modalities
Supported Functionality
| Component | Disk size | GPU weight allocation |
|---|---|---|
| NVFP4 target | 20.59 GB | 18.80 GB |
| NVFP4 DSpark drafter | 1.64 GB | 1.70 GB |
| Combined pair | 22.24 GB | 20.50 GB |
The validated Hermes serving profile launches with a 122,880-token context limit. SGLang automatically allocated an 87,798-token active pool and left 2.83 GB of GPU memory free after cache allocation and CUDA graph capture on one RTX 5090.
| Context field | Tokens |
|---|---|
| Architecture maximum | 262,144 |
| Validated launch context | 122,880 |
| Automatically allocated active pool | 87,798 |
The current NVFP4 target and NVFP4 drafter profile reached 217.73 tokens per second median decode across 18 successful requests on one RTX 5090.
| Metric | Result |
|---|---|
| Requests | 18 of 18 successful |
| Median decode | 217.73 tokens per second |
| Median time to first token | 0.113 seconds |
| P95 time to first token | 0.149 seconds |
| Generated tokens per request | 26 to 640 |
| Total generated tokens | 3,633 |
Test conditions were concurrency 1, temperature 0, two warmups, three repeats across six prompts, FP8 E4M3 KV cache, and a fixed request seed. Decode throughput excludes prompt processing and time to first token. The 16,000-token request setting was a ceiling. This result is not a sustained 16,000-token generation measurement.
The earlier deterministic BF16 drafter profile measured 139.32 tokens per second with a 16,384-token active pool. Its full standard decoding, native MTP, public DSpark, and trained DSpark comparison remains available in benchmarks/benchmark-summary.json.
| Field | Value |
|---|---|
| Source revision | 404ea47aaa5d8a8b00049c9e9750089aca011ab2 |
| Format | ModelOpt NVFP4 safetensors |
| Export size | 20,593,147,464 bytes |
| Architecture | Qwen3_5ForConditionalGeneration |
| Native MTP tensors | 15 |
| Recommended runtime | SGLang Qwen3.8 image |
| Recommended hardware | NVIDIA RTX 5090 32 GB |
bash
docker run --rm --gpus all --ipc=host -p 30000:30000 \lmsysorg/sglang:qwen38-27b \sglang serve \--model-path fleetml/Qwen3.8-27B-Uncensored-NVFP4-RTX5090 \--quantization modelopt_fp4 \--trust-remote-code \--speculative-algorithm DSPARK \--speculative-draft-model-path fleetml/Qwen3.8-27B-Uncensored-DSpark-RTX5090 \--speculative-draft-model-quantization modelopt_fp4 \--speculative-dspark-block-size 7 \--speculative-num-steps 1 \--speculative-eagle-topk 1 \--context-length 122880 \--max-mamba-cache-size 8 \--kv-cache-dtype fp8_e4m3 \--attention-backend flashinfer \--chunked-prefill-size 1024 \--mamba-radix-cache-strategy extra_buffer_lazy \--mamba-ssm-dtype bfloat16 \--mm-feature-transport cpu \--cuda-graph-max-bs-decode 1 \--mem-fraction-static 0.86 \--max-running-requests 1 \--reasoning-parser qwen3 \--tool-call-parser qwen3_coder \--host 0.0.0.0 \--port 30000
The active token pool is allocated automatically. Available capacity can vary slightly with the runtime build and GPU state.
0.47.0.dev70, commit c4129b6eThis checkpoint is intended for controlled local research, evaluation, and agent development. Use it responsibly and comply with the Apache 2.0 license and applicable law.