gittensor-model-hub

Qwen3.8-27B-NVFP4

README

License: apache-2.0

Performance profile

Same method on two Blackwell GPUs, 2026-08-14. vLLM 0.27.1 (V1), FlashInfer SM120 NVFP4 GEMM, torch 2.13.0+cu130, CUDA 13.0.

Table with columns: Item, RTX PRO 6000 (96 GB), GeForce RTX 5090 (32 GB)
Item	RTX PRO 6000 (96 GB)	GeForce RTX 5090 (32 GB)
Driver	580.126.09	580.173.02
Weights in VRAM	18.76 GiB	18.77 GiB
`--gpu-memory-utilization`	0.90	0.97 (needed for 256k)
FP8 KV pool	2,032,765 tokens	275,941 tokens
Conc. at 262,144	7.75x	1.05x
TTFT p50 @ 508 tok	80 ms	106 ms
TTFT p50 @ ~62k	6.70 s	8.52 s
TTFT @ ~238k	62.1 s	87.1 s
Decode @ conc 1 (OSL 256)	67.5 tok/s	80.6 tok/s
Aggregate @ conc 16	862 tok/s	1,029 tok/s
Decode @ ~61k / OSL 128	63.1 tok/s	74.3 tok/s
64k each x 4	4/4 OK	4/4 OK
Longest completed prompt	242,686	242,686
Tool calls	5/5	5/5

Sampling: temperature=0, enable_thinking=false, streaming /v1/chat/completions. Decode tok/s uses ignore_eos and 256 output tokens unless noted. --max-num-seqs 16 on both. 32 GB cannot hold a 256k KV at 0.90 util (vLLM estimated max ~205k); 0.97 leaves 275,941 KV tokens, enough for one 256k request or four ~62k.

text
--max-model-len 262144
--max-num-seqs 16
--kv-cache-dtype fp8
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
# 96 GB:  --gpu-memory-utilization 0.90
# 32 GB:  --gpu-memory-utilization 0.97

RTX PRO 6000 Blackwell 96 GB

Time to first token

Prefill-bound past a few thousand tokens; cost per token rises at long context on this hybrid (Gated DeltaNet + Gated Attention) model.

Table with columns: Input tokens, TTFT, Notes
Input tokens	TTFT	Notes
19	71 ms	warmup
508	80 ms p50 / 83 ms p95	n=8
1,967	128 ms p50	n=4
61,775	6.70 s p50	~64k class
121,625	19.5 s
182,425	39.1 s

Decode throughput

Table with columns: Concurrency, Per-request decode, Aggregate output, TTFT p50, Wall
Concurrency	Per-request decode	Aggregate output	TTFT p50	Wall
1	67.5 tok/s	66 tok/s	77 ms	7.7 s
2	61.8 tok/s	119 tok/s	146 ms	8.6 s
4	60.2 tok/s	232 tok/s	152 ms	8.8 s

At ~61k input / 128 output: 63.1 tok/s (TTFT 6.56 s).

Long context

Table with columns: Workload, Prompt tokens, Conc, TTFT, Result
Workload	Prompt tokens	Conc	TTFT	Result
64k x 1	61,775	1	6.70 s p50	OK
64k each x 4	61,775 x 4	4	7.2 s min / 20.5 s p50	4/4 OK
~128k	121,625	1	19.5 s	OK

Tool calling

Table with columns: Test, Finish reason, Result, Latency
Test	Finish reason	Result	Latency
`tool_choice=auto` get_weather	`tool_calls`	`{"city": "Paris"}`	0.46 s
`tool_choice=required`	`tool_calls`	get_weather (New York)	1.31 s
Parallel two cities	`tool_calls`

GeForce RTX 5090 32 GB

Same flags except --gpu-memory-utilization 0.97.

Time to first token

Table with columns: Input tokens, TTFT, Notes
Input tokens	TTFT	Notes
19	228 ms	warmup
508	106 ms p50 / 106 ms p95	n=8
1,967	134 ms p50	n=4
61,775	8.52 s p50	~64k class
121,625	26.3 s
182,425	53.8 s

Decode throughput

Table with columns: Concurrency, Per-request decode, Aggregate output, TTFT p50, Wall
Concurrency	Per-request decode	Aggregate output	TTFT p50	Wall
1	80.6 tok/s	78 tok/s	126 ms	6.6 s
2	73.1 tok/s	137 tok/s	220 ms	7.5 s
4	71.0 tok/s	267 tok/s	227 ms	7.7 s

At ~61k input / 128 output: 74.3 tok/s (TTFT 8.36 s).

Long context

Table with columns: Workload, Prompt tokens, Conc, TTFT, Result
Workload	Prompt tokens	Conc	TTFT	Result
64k x 1	61,775	1	8.52 s p50	OK
64k each x 4	61,775 x 4	4	8.8 s min / 25.7 s p50	4/4 OK
~128k	121,625	1	26.3 s	OK

KV headroom is tight: four ~62k prompts is ~247k tokens vs a 276k pool.

Tool calling

Table with columns: Test, Finish reason, Result, Latency
Test	Finish reason	Result	Latency
`tool_choice=auto` get_weather	`tool_calls`	`{"city": "Paris"}`	0.42 s
`tool_choice=required`	`tool_calls`	get_weather (New York)	1.24 s
Parallel two cities	`tool_calls`

Coherence smoke test

PRO 6000: 2+2 -> 4, sky color, JSON primary-color list, one-sentence cat prompt, thinking-on speed word problem (60 miles / 1.5 h -> 40). 5/5. Not a substitute for MMLU/GPQA; this card does not report accuracy vs BF16.

Recipe

Table with columns: Item, Value
Item	Value
Tool	Model Optimizer git `c4129b6` (`quant_method: modelopt`)
Weights / activations	NVFP4 W4A4, group size 16
KV cache	FP8 (`fp8_cast` at PTQ; serve with `--kv-cache-dtype fp8`)
Calibration	128 image-text samples (`--calib_with_images`)
PTQ peak VRAM	65.8 GB (on the 96 GB card)

Left in BF16 on purpose: vision tower, lm_head, embeddings, MTP, and Gated-DeltaNet conv1d / in_proj_a / in_proj_b. MLP and remaining Linear layers are NVFP4.

Serve (Blackwell)

bash
vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.90 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

On 32 GB (RTX 5090), use --gpu-memory-utilization 0.97 or drop --max-model-len to ~205k. vLLM 0.24+ (0.27.x recommended). First boot JITs FlashInfer SM120 FP4 GEMM (needs nvcc plus CUDA 13 headers matching the compiler). Limit parallel JIT on smaller host RAM (MAX_JOBS=2).

SGLang: --quantization modelopt_fp4.

Qwen3.8 is a thinking model. For short answers, pass "chat_template_kwargs": {"enable_thinking": false}.

License

Apache 2.0, same as the Qwen3.8-27B base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

gittensor-model-hub

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Performance profile

Same method on two Blackwell GPUs, 2026-08-14. vLLM 0.27.1 (V1), FlashInfer SM120 NVFP4 GEMM, torch 2.13.0+cu130, CUDA 13.0.

Table with columns: Item, RTX PRO 6000 (96 GB), GeForce RTX 5090 (32 GB)
Item	RTX PRO 6000 (96 GB)	GeForce RTX 5090 (32 GB)
Driver	580.126.09	580.173.02
Weights in VRAM	18.76 GiB	18.77 GiB
`--gpu-memory-utilization`	0.90	0.97 (needed for 256k)
FP8 KV pool	2,032,765 tokens	275,941 tokens
Conc. at 262,144	7.75x	1.05x
TTFT p50 @ 508 tok	80 ms	106 ms
TTFT p50 @ ~62k	6.70 s	8.52 s
TTFT @ ~238k	62.1 s	87.1 s
Decode @ conc 1 (OSL 256)	67.5 tok/s	80.6 tok/s
Aggregate @ conc 16	862 tok/s	1,029 tok/s
Decode @ ~61k / OSL 128	63.1 tok/s	74.3 tok/s
64k each x 4	4/4 OK	4/4 OK
Longest completed prompt	242,686	242,686
Tool calls	5/5	5/5

text
--max-model-len 262144
--max-num-seqs 16
--kv-cache-dtype fp8
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
# 96 GB:  --gpu-memory-utilization 0.90
# 32 GB:  --gpu-memory-utilization 0.97

RTX PRO 6000 Blackwell 96 GB

Time to first token

Prefill-bound past a few thousand tokens; cost per token rises at long context on this hybrid (Gated DeltaNet + Gated Attention) model.

Table with columns: Input tokens, TTFT, Notes
Input tokens	TTFT	Notes
19	71 ms	warmup
508	80 ms p50 / 83 ms p95	n=8
1,967	128 ms p50	n=4
61,775	6.70 s p50	~64k class
121,625	19.5 s
182,425	39.1 s

Decode throughput

Table with columns: Concurrency, Per-request decode, Aggregate output, TTFT p50, Wall
Concurrency	Per-request decode	Aggregate output	TTFT p50	Wall
1	67.5 tok/s	66 tok/s	77 ms	7.7 s
2	61.8 tok/s	119 tok/s	146 ms	8.6 s
4	60.2 tok/s	232 tok/s	152 ms	8.8 s

At ~61k input / 128 output: 63.1 tok/s (TTFT 6.56 s).

Long context

Table with columns: Workload, Prompt tokens, Conc, TTFT, Result
Workload	Prompt tokens	Conc	TTFT	Result
64k x 1	61,775	1	6.70 s p50	OK
64k each x 4	61,775 x 4	4	7.2 s min / 20.5 s p50	4/4 OK
~128k	121,625	1	19.5 s	OK

Tool calling

Table with columns: Test, Finish reason, Result, Latency
Test	Finish reason	Result	Latency
`tool_choice=auto` get_weather	`tool_calls`	`{"city": "Paris"}`	0.46 s
`tool_choice=required`	`tool_calls`	get_weather (New York)	1.31 s
Parallel two cities	`tool_calls`

GeForce RTX 5090 32 GB

Same flags except --gpu-memory-utilization 0.97.

Time to first token

Table with columns: Input tokens, TTFT, Notes
Input tokens	TTFT	Notes
19	228 ms	warmup
508	106 ms p50 / 106 ms p95	n=8
1,967	134 ms p50	n=4
61,775	8.52 s p50	~64k class
121,625	26.3 s
182,425	53.8 s

Decode throughput

Table with columns: Concurrency, Per-request decode, Aggregate output, TTFT p50, Wall
Concurrency	Per-request decode	Aggregate output	TTFT p50	Wall
1	80.6 tok/s	78 tok/s	126 ms	6.6 s
2	73.1 tok/s	137 tok/s	220 ms	7.5 s
4	71.0 tok/s	267 tok/s	227 ms	7.7 s

At ~61k input / 128 output: 74.3 tok/s (TTFT 8.36 s).

Long context

Table with columns: Workload, Prompt tokens, Conc, TTFT, Result
Workload	Prompt tokens	Conc	TTFT	Result
64k x 1	61,775	1	8.52 s p50	OK
64k each x 4	61,775 x 4	4	8.8 s min / 25.7 s p50	4/4 OK
~128k	121,625	1	26.3 s	OK

KV headroom is tight: four ~62k prompts is ~247k tokens vs a 276k pool.

Tool calling

Table with columns: Test, Finish reason, Result, Latency
Test	Finish reason	Result	Latency
`tool_choice=auto` get_weather	`tool_calls`	`{"city": "Paris"}`	0.42 s
`tool_choice=required`	`tool_calls`	get_weather (New York)	1.24 s
Parallel two cities	`tool_calls`

Coherence smoke test

Recipe

Table with columns: Item, Value
Item	Value
Tool	Model Optimizer git `c4129b6` (`quant_method: modelopt`)
Weights / activations	NVFP4 W4A4, group size 16
KV cache	FP8 (`fp8_cast` at PTQ; serve with `--kv-cache-dtype fp8`)
Calibration	128 image-text samples (`--calib_with_images`)
PTQ peak VRAM	65.8 GB (on the 96 GB card)

Left in BF16 on purpose: vision tower, lm_head, embeddings, MTP, and Gated-DeltaNet conv1d / in_proj_a / in_proj_b. MLP and remaining Linear layers are NVFP4.

Serve (Blackwell)

bash
vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.90 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

SGLang: --quantization modelopt_fp4.

Qwen3.8 is a thinking model. For short answers, pass "chat_template_kwargs": {"enable_thinking": false}.

License

Apache 2.0, same as the Qwen3.8-27B base model.