fleetml

Qwen3.8-27B-Uncensored-NVFP4-RTX5090

README

License: apache-2.0

Combined release

Table with columns: Component, Disk size, GPU weight allocation
Component	Disk size	GPU weight allocation
NVFP4 target	20.59 GB	18.80 GB
NVFP4 DSpark drafter	1.64 GB	1.70 GB
Combined pair	22.24 GB	20.50 GB

The validated Hermes serving profile launches with a 122,880-token context limit. SGLang automatically allocated an 87,798-token active pool and left 2.83 GB of GPU memory free after cache allocation and CUDA graph capture on one RTX 5090.

Table with columns: Context field, Tokens
Context field	Tokens
Architecture maximum	262,144
Validated launch context	122,880
Automatically allocated active pool	87,798

Measured performance

The current NVFP4 target and NVFP4 drafter profile reached 217.73 tokens per second median decode across 18 successful requests on one RTX 5090.

Table with columns: Metric, Result
Metric	Result
Requests	18 of 18 successful
Median decode	217.73 tokens per second
Median time to first token	0.113 seconds
P95 time to first token	0.149 seconds
Generated tokens per request	26 to 640
Total generated tokens	3,633

Test conditions were concurrency 1, temperature 0, two warmups, three repeats across six prompts, FP8 E4M3 KV cache, and a fixed request seed. Decode throughput excludes prompt processing and time to first token. The 16,000-token request setting was a ceiling. This result is not a sustained 16,000-token generation measurement.

The earlier deterministic BF16 drafter profile measured 139.32 tokens per second with a 16,384-token active pool. Its full standard decoding, native MTP, public DSpark, and trained DSpark comparison remains available in benchmarks/benchmark-summary.json.

Checkpoint

Table with columns: Field, Value
Field	Value
Source revision	`404ea47aaa5d8a8b00049c9e9750089aca011ab2`
Format	ModelOpt NVFP4 safetensors
Export size	20,593,147,464 bytes
Architecture	`Qwen3_5ForConditionalGeneration`
Native MTP tensors	15
Recommended runtime	SGLang Qwen3.8 image
Recommended hardware	NVIDIA RTX 5090 32 GB

Run with the matched NVFP4 DSpark drafter

bash

docker run --rm --gpus all --ipc=host -p 30000:30000 \  lmsysorg/sglang:qwen38-27b \  sglang serve \  --model-path fleetml/Qwen3.8-27B-Uncensored-NVFP4-RTX5090 \  --quantization modelopt_fp4 \  --trust-remote-code \  --speculative-algorithm DSPARK \  --speculative-draft-model-path fleetml/Qwen3.8-27B-Uncensored-DSpark-RTX5090 \  --speculative-draft-model-quantization modelopt_fp4 \  --speculative-dspark-block-size 7 \  --speculative-num-steps 1 \  --speculative-eagle-topk 1 \  --context-length 122880 \  --max-mamba-cache-size 8 \  --kv-cache-dtype fp8_e4m3 \  --attention-backend flashinfer \  --chunked-prefill-size 1024 \  --mamba-radix-cache-strategy extra_buffer_lazy \  --mamba-ssm-dtype bfloat16 \  --mm-feature-transport cpu \  --cuda-graph-max-bs-decode 1 \  --mem-fraction-static 0.86 \  --max-running-requests 1 \  --reasoning-parser qwen3 \  --tool-call-parser qwen3_coder \  --host 0.0.0.0 \  --port 30000

The active token pool is allocated automatically. Available capacity can vary slightly with the runtime build and GPU state.

Provenance

Base model: Qwen/Qwen3.8-27B
BF16 source: orcarouter/Qwen3.8-27B-Uncensored
Quantization: NVIDIA ModelOpt 0.47.0.dev70, commit c4129b6e
Calibration: 126 tokenized UltraChat conversations
Validation: complete shard, tensor, dtype, MTP, load, API, Hermes, and benchmark checks

Use

This checkpoint is intended for controlled local research, evaluation, and agent development. Use it responsibly and comply with the Apache 2.0 license and applicable law.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

fleetml

Model Tree

Base

orcarouter/Qwen3.8-27B-Uncensored

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality