mph

qwen3.8-27b-mxfp8

README

License: apache-2.0

Quantization details

This checkpoint was produced with TorchAO using MXDynamicActivationMXWeightConfig:

Table with columns: Component, Precision
Component	Precision
`model.language_model` (text decoder `Linear` layers)	MXFP8 weights; activations quantized dynamically at inference
`visual` (ViT + merger), `lm_head`	bfloat16
Hybrid Gated DeltaNet `Conv1d` / `A_log` / `dt_bias` / `in_proj_b` / `in_proj_a`	bfloat16

Format: torchao-flattened safetensors (MXTensor qdata/scale + metadata)
Block size: 32
Dtypes: float8_e4m3fn for weights and activations
Scaling: RCEIL
Base dtype: bfloat16

Weights were quantized once on GPU, exported to CPU, flattened with flatten_tensor_state_dict, and saved with a TorchAoConfig in config.json. Reload does not re-run weight quantization; the language model still applies dynamic activation quantization during forward passes.

Hardware requirements

MXFP8 inference requires a Blackwell-class NVIDIA GPU (compute capability SM100+, i.e. major version ≥ 10). Examples include B200, GB200, and RTX Pro 6000. Older architectures (Ampere, Hopper, etc.) are not supported for this checkpoint.

CUDA GPU with SM100+
Sufficient VRAM for a 27B multimodal model (peak usage depends on sequence length and vision inputs)

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True if you hit fragmentation during load or generation.

Software requirements

bash

pip install "transformers>=5.5.4" torch torchao safetensors

You need a recent torchao build with MXFP8 inference support. For serving, use a recent vllm with TorchAO MXFP8 support (nightly or a source build is typical).

Load the processor from the same directory as the weights (vLLM does this automatically; there is no --processor flag):

python

from transformers import AutoProcessor, Qwen3_5ForConditionalGenerationimport torch QUANTIZED_MODEL = "YOUR_USERNAME/qwen3.8-27b-mxfp8"  # or local path processor = AutoProcessor.from_pretrained(QUANTIZED_MODEL)model = Qwen3_5ForConditionalGeneration.from_pretrained(    QUANTIZED_MODEL,    torch_dtype=torch.bfloat16,)model.to("cuda")model.eval()

Qwen3.8 reuses the Qwen3.5 Transformers class (Qwen3_5ForConditionalGeneration).

Usage

Thinking mode is on by default. Pass enable_thinking=False to apply_chat_template for instruct (non-thinking) mode.

Text-only

python

messages = [    {"role": "user", "content": "Explain MXFP8 in one sentence."},]inputs = processor.apply_chat_template(    messages,    tokenize=True,    return_dict=True,    return_tensors="pt",    add_generation_prompt=True,    enable_thinking=True,)inputs = inputs.to(model.device) with torch.inference_mode():    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False) response = processor.decode(    output_ids[0, inputs["input_ids"].shape[-1]:],    skip_special_tokens=False,)print(response)

Image + text

python

from PIL import Image image = Image.open("example.png").convert("RGB")messages = [    {        "role": "user",        "content": [            {"type": "image", "image": image},            {"type": "text", "text": "What is shown in this image?"},        ],    }]# Same apply_chat_template → generate → decode flow as above.

Serving with vLLM

This checkpoint can be served with vllm serve. vLLM loads the Qwen3 VL image/video processor from the model directory, so preprocessor_config.json and video_preprocessor_config.json must sit next to the weights. --tokenizer is optional once those sidecars are present.

Save the following as serve.sh (or run it inline):

bash

#!/usr/bin/env bashset -euo pipefail export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True# TorchAO MX kernels and vLLM compile cache currently compose poorly.export VLLM_DISABLE_COMPILE_CACHE=1 MODEL=mph/qwen3.8-27b-mxfp8TOKENIZER=Qwen/Qwen3.8-27BSERVED_NAME=qwen3.8-27b-mxfp8PORT=8000MAX_MODEL_LEN=5000  # native context is 262144 vllm serve "$MODEL" \  --tokenizer "$TOKENIZER" \  --served-model-name "$SERVED_NAME" \  --host 0.0.0.0 \  --port "$PORT" \  --max-model-len "$MAX_MODEL_LEN" \  --max-num-seqs 115 \  --gpu-memory-utilization 0.94 \  --attention-backend FLASHINFER \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --mm-encoder-tp-mode data

bash

chmod +x serve.sh./serve.sh

The server exposes an OpenAI-compatible API at http://localhost:8000/v1.

Recommended sampling (from the base model card):

Table with columns: Mode, temperature, top_p, top_k, presence_penalty
Mode	`temperature`	`top_p`	`top_k`	`presence_penalty`
Thinking (default)	1.0	0.95	20	0.0
Instruct / non-thinking	0.7	0.80	20	1.5

bash

curl http://localhost:8000/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{    "model": "qwen3.8-27b-mxfp8",    "messages": [      {"role": "user", "content": "Explain MXFP8 in one sentence."}    ],    "temperature": 1.0,    "top_p": 0.95,    "max_tokens": 1024,    "chat_template_kwargs": {      "enable_thinking": true,      "preserve_thinking": true    }  }'

python

from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")resp = client.chat.completions.create(    model="qwen3.8-27b-mxfp8",    messages=[{"role": "user", "content": "Explain MXFP8 in one sentence."}],    temperature=1.0,    top_p=0.95,    extra_body={        "top_k": 20,        "chat_template_kwargs": {"enable_thinking": True},        "reasoning_effort": "low",  # xhigh (default), medium, or low    },)print(resp.choices[0].message.content)

Optional flags:

--language-model-only — skip the vision encoder (more KV cache for text-only serving)
--tensor-parallel-size N — split across GPUs if one card is not enough
--default-chat-template-kwargs '{"enable_thinking": false}' — disable thinking server-wide
YaRN to 1M context (from the base model card):

bash

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 ./serve.sh# then add to the vllm serve invocation:#   --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'#   --max-model-len 1000000

Files

Table with columns: File, Description
File	Description
`model.safetensors`	Quantized weights
`config.json`	Model config + `quantization_config` (`TorchAoConfig`)
`generation_config.json`	Generation defaults from the base model
`preprocessor_config.json`	Image processor (required by vLLM)
`video_preprocessor_config.json`

Limitations

Quantization quality has not been formally benchmarked against the full-precision base model; validate on your tasks before production use.
MXFP8 kernels and TorchAO MX support are still evolving; pin compatible torch / torchao / vllm versions for reproducibility.
Vision and lm_head paths run in bf16, so memory savings are concentrated in the text decoder.

License

Apache 2.0. Follow the license terms of Qwen/Qwen3.8-27B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

mph

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quantization details

This checkpoint was produced with TorchAO using MXDynamicActivationMXWeightConfig:

Table with columns: Component, Precision
Component	Precision
`model.language_model` (text decoder `Linear` layers)	MXFP8 weights; activations quantized dynamically at inference
`visual` (ViT + merger), `lm_head`	bfloat16
Hybrid Gated DeltaNet `Conv1d` / `A_log` / `dt_bias` / `in_proj_b` / `in_proj_a`	bfloat16

Format: torchao-flattened safetensors (MXTensor qdata/scale + metadata)
Block size: 32
Dtypes: float8_e4m3fn for weights and activations
Scaling: RCEIL
Base dtype: bfloat16

Hardware requirements

CUDA GPU with SM100+
Sufficient VRAM for a 27B multimodal model (peak usage depends on sequence length and vision inputs)

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True if you hit fragmentation during load or generation.

Software requirements

bash

pip install "transformers>=5.5.4" torch torchao safetensors

You need a recent torchao build with MXFP8 inference support. For serving, use a recent vllm with TorchAO MXFP8 support (nightly or a source build is typical).

Load the processor from the same directory as the weights (vLLM does this automatically; there is no --processor flag):

python

from transformers import AutoProcessor, Qwen3_5ForConditionalGenerationimport torch QUANTIZED_MODEL = "YOUR_USERNAME/qwen3.8-27b-mxfp8"  # or local path processor = AutoProcessor.from_pretrained(QUANTIZED_MODEL)model = Qwen3_5ForConditionalGeneration.from_pretrained(    QUANTIZED_MODEL,    torch_dtype=torch.bfloat16,)model.to("cuda")model.eval()

Qwen3.8 reuses the Qwen3.5 Transformers class (Qwen3_5ForConditionalGeneration).

Usage

Thinking mode is on by default. Pass enable_thinking=False to apply_chat_template for instruct (non-thinking) mode.

Text-only

python

messages = [    {"role": "user", "content": "Explain MXFP8 in one sentence."},]inputs = processor.apply_chat_template(    messages,    tokenize=True,    return_dict=True,    return_tensors="pt",    add_generation_prompt=True,    enable_thinking=True,)inputs = inputs.to(model.device) with torch.inference_mode():    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False) response = processor.decode(    output_ids[0, inputs["input_ids"].shape[-1]:],    skip_special_tokens=False,)print(response)

Image + text

python

from PIL import Image image = Image.open("example.png").convert("RGB")messages = [    {        "role": "user",        "content": [            {"type": "image", "image": image},            {"type": "text", "text": "What is shown in this image?"},        ],    }]# Same apply_chat_template → generate → decode flow as above.

Serving with vLLM

Save the following as serve.sh (or run it inline):

bash

#!/usr/bin/env bashset -euo pipefail export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True# TorchAO MX kernels and vLLM compile cache currently compose poorly.export VLLM_DISABLE_COMPILE_CACHE=1 MODEL=mph/qwen3.8-27b-mxfp8TOKENIZER=Qwen/Qwen3.8-27BSERVED_NAME=qwen3.8-27b-mxfp8PORT=8000MAX_MODEL_LEN=5000  # native context is 262144 vllm serve "$MODEL" \  --tokenizer "$TOKENIZER" \  --served-model-name "$SERVED_NAME" \  --host 0.0.0.0 \  --port "$PORT" \  --max-model-len "$MAX_MODEL_LEN" \  --max-num-seqs 115 \  --gpu-memory-utilization 0.94 \  --attention-backend FLASHINFER \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --mm-encoder-tp-mode data

bash

chmod +x serve.sh./serve.sh

The server exposes an OpenAI-compatible API at http://localhost:8000/v1.

Recommended sampling (from the base model card):

Table with columns: Mode, temperature, top_p, top_k, presence_penalty
Mode	`temperature`	`top_p`	`top_k`	`presence_penalty`
Thinking (default)	1.0	0.95	20	0.0
Instruct / non-thinking	0.7	0.80	20	1.5

bash

curl http://localhost:8000/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{    "model": "qwen3.8-27b-mxfp8",    "messages": [      {"role": "user", "content": "Explain MXFP8 in one sentence."}    ],    "temperature": 1.0,    "top_p": 0.95,    "max_tokens": 1024,    "chat_template_kwargs": {      "enable_thinking": true,      "preserve_thinking": true    }  }'

python

from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")resp = client.chat.completions.create(    model="qwen3.8-27b-mxfp8",    messages=[{"role": "user", "content": "Explain MXFP8 in one sentence."}],    temperature=1.0,    top_p=0.95,    extra_body={        "top_k": 20,        "chat_template_kwargs": {"enable_thinking": True},        "reasoning_effort": "low",  # xhigh (default), medium, or low    },)print(resp.choices[0].message.content)

Optional flags:

--language-model-only — skip the vision encoder (more KV cache for text-only serving)
--tensor-parallel-size N — split across GPUs if one card is not enough
--default-chat-template-kwargs '{"enable_thinking": false}' — disable thinking server-wide
YaRN to 1M context (from the base model card):

bash

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 ./serve.sh# then add to the vllm serve invocation:#   --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'#   --max-model-len 1000000

Files

Table with columns: File, Description
File	Description
`model.safetensors`	Quantized weights
`config.json`	Model config + `quantization_config` (`TorchAoConfig`)
`generation_config.json`	Generation defaults from the base model
`preprocessor_config.json`	Image processor (required by vLLM)
`video_preprocessor_config.json`

Limitations

Quantization quality has not been formally benchmarked against the full-precision base model; validate on your tasks before production use.
MXFP8 kernels and TorchAO MX support are still evolving; pin compatible torch / torchao / vllm versions for reproducibility.
Vision and lm_head paths run in bf16, so memory savings are concentrated in the text decoder.

License

Apache 2.0. Follow the license terms of Qwen/Qwen3.8-27B.