sakamakismile

fable-traces-NVFP4

Deploy Dedicated

README

License: apache-2.0

Practical Warning

Not a serious instruction model
Not reliable for Japanese reasoning
Not reliable for arithmetic, logic, coding, or factual answers
Useful only for testing NVFP4 loading, serving, KV-cache behavior, and raw TPS

Quantization

Source precision: BF16 safetensors
Output format: compressed-tensors
NVFP4 format: nvfp4-pack-quantized
Scheme: NVFP4 W4A4
Targets: Linear
Ignored modules: lm_head
Calibration: neuralmagic/calibration, LLM, 512 samples, max sequence 4096
Tooling: llmcompressor, compressed-tensors

The resulting model.safetensors is about 2.82 GB.

vLLM Usage

For a 128K context run with FP8 KV cache:

bash

vllm serve sakamakismile/fable-traces-NVFP4 \  --max-model-len 131072 \  --kv-cache-dtype fp8 \  --gpu-memory-utilization 0.92

For Python:

python

from vllm import LLM, SamplingParams llm = LLM(    model="sakamakismile/fable-traces-NVFP4",    max_model_len=131072,    kv_cache_dtype="fp8",    gpu_memory_utilization=0.92,) messages = [[{"role": "user", "content": "Tell me something interesting."}]]outputs = llm.chat(messages, SamplingParams(max_tokens=128, temperature=0.0))print(outputs[0].outputs[0].text)

Benchmark

Measured locally with vLLM 0.21.0+cu129 on NVIDIA RTX PRO 2000 Blackwell 16 GB GPUs, driver 595.71.05.

Common settings:

max_model_len=131072
kv_cache_dtype=fp8
max_num_seqs=8
max_tokens=256
repeats=3, warmup=1
enforce_eager=True

TP=1

vLLM reported 156,576 GPU KV-cache tokens with FP8 KV cache.

Table with columns: TP, Parallel, Mean output tok/s, Best output tok/s, Mean total tok/s
TP	Parallel	Mean output tok/s	Best output tok/s	Mean total tok/s
1	1	69.03	71.45	118.07
1	2	129.30	131.40	219.80
1	4	238.04	244.81	404.32
1

TP=2

vLLM reported 359,488 GPU KV-cache tokens with FP8 KV cache.

Table with columns: TP, Parallel, Mean output tok/s, Best output tok/s, Mean total tok/s
TP	Parallel	Mean output tok/s	Best output tok/s	Mean total tok/s
2	1	60.37	60.72	103.26
2	2	100.86	122.85	166.75
2	4	234.43	236.21	388.02
2

For this short-output benchmark, TP=1 is faster; TP=2 mainly buys more KV-cache headroom for long-context use.

Raw benchmark files are included under benchmarks/.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

sakamakismile

Model Tree

Base

AliesTaha/fable-traces

Quantized

this model

Input Modalities

Text

Output Modalities