mph

ornith-1.5-35b-a3b-mxfp8

Quantization details

This checkpoint was produced with TorchAO using MXDynamicActivationMXWeightConfig:

Table with columns: Component, Precision
Component	Precision
`model.language_model` `Linear` layers (attention, GDN `in_proj_qkv` / `in_proj_z` / `out_proj`, shared-expert `gate_proj` / `up_proj` / `down_proj`)	MXFP8 weights; activations quantized dynamically at inference
Packed MoE experts (`Qwen3_5MoeExperts.gate_up_proj`, `down_proj`)	MXFP8 weights (`nn.Parameter` of shape `[num_experts, out, in]`, not `Linear`)
`visual` (ViT + merger), `lm_head`	bfloat16
Hybrid Gated DeltaNet `Conv1d` / `A_log` / `dt_bias` / `in_proj_b` / `in_proj_a`	bfloat16
MoE routers (`mlp.gate` TopK router, `shared_expert_gate`)	bfloat16

Format: torchao-flattened safetensors (MXTensor qdata/scale + metadata)
Block size: 32
Dtypes: float8_e4m3fn for weights and activations
Scaling: RCEIL
Base dtype: bfloat16

Packed experts are 3D parameters (gate_up_proj: [256, 1024, 2048], down_proj: [256, 2048, 512]). TorchAO's Linear-only quantize_ path skips them, so they are converted with FqnToConfig on those parameter FQNs. Shared-expert Linear projections are MXFP8; that is distinct from the packed routed experts.

Gated DeltaNet in_proj_b / in_proj_a stay bf16 because vLLM fuses them into in_proj_ba with output sizes [48, 48]; swizzled MXFP8 scales cannot be row-sliced at 48 (they need 128-alignment). mlp.gate is a Qwen3_5MoeTopKRouter (nn.Parameter, not Linear), so TorchAO skips it. shared_expert_gate is a Linear and is explicitly left in bf16.

Weights were quantized once on GPU, exported to CPU, flattened with flatten_tensor_state_dict, and saved with a TorchAoConfig in config.json. Reload does not re-run weight quantization; the language model still applies dynamic activation quantization during forward passes.

Hardware requirements

MXFP8 inference requires a Blackwell-class NVIDIA GPU (compute capability SM100+, i.e. major version ≥ 10). Examples include B200, GB200, and RTX Pro 6000. Older architectures (Ampere, Hopper, etc.) are not supported for this checkpoint.

CUDA GPU with SM100+
Sufficient VRAM for a 35B-A3B multimodal MoE (peak usage depends on sequence length, runtime, and vision inputs). The bf16 base is ≈70 GB; this MXFP8 checkpoint is ≈37 GB on disk (packed routed experts are the bulk of that saving). Transformers with experts_implementation="grouped_mm" keeps those packs as MXTensors in memory. vLLM's FusedMoE loader cannot chunk/unbind swizzled MXFP8 experts, so it dequantizes them to bf16 on load (~64 GB for experts plus MXFP8 Linears) unless you patch the loader. The official card recommends 2× 80GB GPUs for the full 256K context.

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True if you hit fragmentation during load or generation.

Software requirements

bash
pip install "transformers>=5.8.1" torch torchao safetensors

You need a recent torchao build with MXFP8 inference support. For serving, use a recent vllm with TorchAO MXFP8 support (nightly or a source build is typical). The base model card asks for vLLM ≥ 0.19.1.

Load the processor from the same directory as the weights (vLLM does this automatically; there is no --processor flag):

python
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
import torch

QUANTIZED_MODEL = "YOUR_USERNAME/ornith-1.5-35b-a3b-mxfp8"  # or local path

processor = AutoProcessor.from_pretrained(QUANTIZED_MODEL)
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    QUANTIZED_MODEL,
    torch_dtype=torch.bfloat16,
    experts_implementation="grouped_mm",
)
model.to("cuda")
model.eval()

Ornith-1.5-35B-A3B is a Qwen3.5-MoE VLM and uses Qwen3_5MoeForConditionalGeneration. Packed expert weights are swizzled MXTensors; the default eager expert loop indexes gate_up_proj[expert_idx] / down_proj[expert_idx], which cannot slice swizzled scales at a single expert row (128-alignment). Pass experts_implementation="grouped_mm" (or "batched_mm") so the 3D tensors are used whole.

Usage

This is a reasoning model: by default the assistant turn opens with a <think> … </think> block before the final answer. Pass enable_thinking=False to apply_chat_template to skip the thinking block.

Text-only

python
messages = [
    {"role": "user", "content": "Explain MXFP8 in one sentence."},
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = inputs.to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)

response = processor.decode(
    output_ids[0, inputs["input_ids"].shape[-1]:],
    skip_special_tokens=False,
)
print(response)

Image + text

python
from PIL import Image

image = Image.open("example.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    }
]
# Same apply_chat_template → generate → decode flow as above.

Serving with vLLM

vLLM applies TorchAO only to LinearBase layers. Packed routed experts restore as 3D MXTensors from flattened safetensors metadata, then RoutedExperts.load_weights identifies fused packs by dim() == 3 and runs chunk(2, dim=1) / unbind(). MXTensor does not implement aten.split, and swizzled scales cannot slice a single expert row (128-alignment). Stock vllm serve therefore dies at engine init:

markdown
NotImplementedError: MXTensor dispatch: ... aten.split

A load-time dequant of those 3D packs (this repo: scripts/vllm_ornith.sh via scripts/vllm_ornith_patch/sitecustomize.py) lets FusedMoE finish loading. Experts then run on vLLM's unquantized MoE backend in bf16; attention / GDN / shared-expert Linears stay MXFP8. Size --max-num-seqs / --gpu-memory-utilization for ~bf16 expert weights, not the 37 GB file.

vLLM still loads the Qwen3 VL image/video processor from the model directory, so processor_config.json must sit next to the weights. --tokenizer is optional once those sidecars are present.

Save the following as serve.sh (or run it inline). Put a dequant-on-load shim on PYTHONPATH first (see scripts/vllm_ornith.sh in this repo) or engine core init will fail as above:

bash
#!/usr/bin/env bash
set -euo pipefail

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
# TorchAO MX kernels and vLLM compile cache currently compose poorly.
export VLLM_DISABLE_COMPILE_CACHE=1
# Required: dequantize 3D MXTensor expert packs before FusedMoE .chunk/.unbind.
# export PYTHONPATH=/path/to/quant/scripts/vllm_ornith_patch${PYTHONPATH:+:$PYTHONPATH}

MODEL=mph/ornith-1.5-35b-a3b-mxfp8
TOKENIZER=ornith-ai/Ornith-1.5-35B-A3B
SERVED_NAME=ornith-1.5-35b-a3b-mxfp8
PORT=8000
MAX_MODEL_LEN=5000

vllm serve "$MODEL" \
  --tokenizer "$TOKENIZER" \
  --served-model-name "$SERVED_NAME" \
  --host 0.0.0.0 \
  --port "$PORT" \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs 125 \
  --gpu-memory-utilization 0.94 \
  --attention-backend FLASHINFER \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --mm-encoder-tp-mode data

bash
chmod +x serve.sh
./serve.sh

The server exposes an OpenAI-compatible API at http://localhost:8000/v1. With --reasoning-parser qwen3 and --tool-call-parser qwen3_xml, chain-of-thought is returned in reasoning_content and <tool_call> blocks are surfaced as OpenAI-style tool_calls.

Recommended sampling (from the base model card):

Table with columns: Mode, temperature, top_p, top_k
Mode	`temperature`	`top_p`	`top_k`
General tasks	0.6	0.95	20
Reproduce reported benchmarks	1.0	0.95	20

bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ornith-1.5-35b-a3b-mxfp8",
    "messages": [
      {"role": "user", "content": "Explain MXFP8 in one sentence."}
    ],
    "temperature": 0.6,
    "top_p": 0.95,
    "max_tokens": 1024,
    "chat_template_kwargs": {
      "enable_thinking": true
    }
  }'

python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="ornith-1.5-35b-a3b-mxfp8",
    messages=[{"role": "user", "content": "Explain MXFP8 in one sentence."}],
    temperature=0.6,
    top_p=0.95,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": True},
    },
)
message = resp.choices[0].message
print(getattr(message, "reasoning_content", None))
print(message.content)

Optional flags:

--language-model-only — skip the vision encoder (more KV cache for text-only serving)
--tensor-parallel-size N — split across GPUs if one card is not enough
--default-chat-template-kwargs '{"enable_thinking": false}' — disable thinking server-wide
YaRN to ~1M context (from the base model card):

bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 ./serve.sh
# then add to the vllm serve invocation:
#   --hf-overrides '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144}}'
#   --max-model-len 1000000

Open-source runtimes apply YaRN statically to every request. Only enable it when the workload needs the longer window; size factor so that factor × 262144 covers the target length (use factor: 2.0 around 524K tokens).

Files

Table with columns: File, Description
File	Description
`model.safetensors`	Quantized weights
`config.json`	Model config + `quantization_config` (`TorchAoConfig`)
`generation_config.json`	Generation defaults from the base model
`processor_config.json`	Image/video processor (required by vLLM)
`tokenizer.json` / /

Limitations

Quantization quality has not been formally benchmarked against the full-precision base model; validate on your tasks before production use.
MXFP8 kernels and TorchAO MX support are still evolving; pin compatible torch / torchao / vllm versions for reproducibility.
Vision, lm_head, GDN fused projections, and MoE routers run in bf16. On-disk savings are concentrated in packed routed experts (gate_up_proj / down_proj) plus attention and shared-expert Linear layers.
Transformers eager MoE (experts_implementation="eager") is incompatible with these packed MXFP8 experts; use grouped_mm or batched_mm as above so the 3D tensors are used whole.
vLLM cannot keep packed experts as MXFP8: FusedMoE splits fused with (), which does not implement. Serve only after dequantizing those 3D packs on load; expert VRAM is then ~bf16. Linear MXFP8 weights are unaffected.

License

MIT. Follow the license terms of ornith-ai/Ornith-1.5-35B-A3B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

mph

Model Tree

Base

ornith-ai/Ornith-1.5-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Quantization details

This checkpoint was produced with TorchAO using MXDynamicActivationMXWeightConfig:

Table with columns: Component, Precision
Component	Precision
`model.language_model` `Linear` layers (attention, GDN `in_proj_qkv` / `in_proj_z` / `out_proj`, shared-expert `gate_proj` / `up_proj` / `down_proj`)	MXFP8 weights; activations quantized dynamically at inference
Packed MoE experts (`Qwen3_5MoeExperts.gate_up_proj`, `down_proj`)	MXFP8 weights (`nn.Parameter` of shape `[num_experts, out, in]`, not `Linear`)
`visual` (ViT + merger), `lm_head`	bfloat16
Hybrid Gated DeltaNet `Conv1d` / `A_log` / `dt_bias` / `in_proj_b` / `in_proj_a`	bfloat16
MoE routers (`mlp.gate` TopK router, `shared_expert_gate`)	bfloat16

Format: torchao-flattened safetensors (MXTensor qdata/scale + metadata)
Block size: 32
Dtypes: float8_e4m3fn for weights and activations
Scaling: RCEIL
Base dtype: bfloat16

Hardware requirements

CUDA GPU with SM100+
Sufficient VRAM for a 35B-A3B multimodal MoE (peak usage depends on sequence length, runtime, and vision inputs). The bf16 base is ≈70 GB; this MXFP8 checkpoint is ≈37 GB on disk (packed routed experts are the bulk of that saving). Transformers with experts_implementation="grouped_mm" keeps those packs as MXTensors in memory. vLLM's FusedMoE loader cannot chunk/unbind swizzled MXFP8 experts, so it dequantizes them to bf16 on load (~64 GB for experts plus MXFP8 Linears) unless you patch the loader. The official card recommends 2× 80GB GPUs for the full 256K context.

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True if you hit fragmentation during load or generation.

Software requirements

bash
pip install "transformers>=5.8.1" torch torchao safetensors

Load the processor from the same directory as the weights (vLLM does this automatically; there is no --processor flag):

python
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
import torch

QUANTIZED_MODEL = "YOUR_USERNAME/ornith-1.5-35b-a3b-mxfp8"  # or local path

processor = AutoProcessor.from_pretrained(QUANTIZED_MODEL)
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    QUANTIZED_MODEL,
    torch_dtype=torch.bfloat16,
    experts_implementation="grouped_mm",
)
model.to("cuda")
model.eval()

Usage

Text-only

python
messages = [
    {"role": "user", "content": "Explain MXFP8 in one sentence."},
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = inputs.to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)

response = processor.decode(
    output_ids[0, inputs["input_ids"].shape[-1]:],
    skip_special_tokens=False,
)
print(response)

Image + text

python
from PIL import Image

image = Image.open("example.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    }
]
# Same apply_chat_template → generate → decode flow as above.

Serving with vLLM

markdown
NotImplementedError: MXTensor dispatch: ... aten.split

vLLM still loads the Qwen3 VL image/video processor from the model directory, so processor_config.json must sit next to the weights. --tokenizer is optional once those sidecars are present.

Save the following as serve.sh (or run it inline). Put a dequant-on-load shim on PYTHONPATH first (see scripts/vllm_ornith.sh in this repo) or engine core init will fail as above:

bash
#!/usr/bin/env bash
set -euo pipefail

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
# TorchAO MX kernels and vLLM compile cache currently compose poorly.
export VLLM_DISABLE_COMPILE_CACHE=1
# Required: dequantize 3D MXTensor expert packs before FusedMoE .chunk/.unbind.
# export PYTHONPATH=/path/to/quant/scripts/vllm_ornith_patch${PYTHONPATH:+:$PYTHONPATH}

MODEL=mph/ornith-1.5-35b-a3b-mxfp8
TOKENIZER=ornith-ai/Ornith-1.5-35B-A3B
SERVED_NAME=ornith-1.5-35b-a3b-mxfp8
PORT=8000
MAX_MODEL_LEN=5000

vllm serve "$MODEL" \
  --tokenizer "$TOKENIZER" \
  --served-model-name "$SERVED_NAME" \
  --host 0.0.0.0 \
  --port "$PORT" \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs 125 \
  --gpu-memory-utilization 0.94 \
  --attention-backend FLASHINFER \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --mm-encoder-tp-mode data

bash
chmod +x serve.sh
./serve.sh

Recommended sampling (from the base model card):

Table with columns: Mode, temperature, top_p, top_k
Mode	`temperature`	`top_p`	`top_k`
General tasks	0.6	0.95	20
Reproduce reported benchmarks	1.0	0.95	20

bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ornith-1.5-35b-a3b-mxfp8",
    "messages": [
      {"role": "user", "content": "Explain MXFP8 in one sentence."}
    ],
    "temperature": 0.6,
    "top_p": 0.95,
    "max_tokens": 1024,
    "chat_template_kwargs": {
      "enable_thinking": true
    }
  }'

python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="ornith-1.5-35b-a3b-mxfp8",
    messages=[{"role": "user", "content": "Explain MXFP8 in one sentence."}],
    temperature=0.6,
    top_p=0.95,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": True},
    },
)
message = resp.choices[0].message
print(getattr(message, "reasoning_content", None))
print(message.content)

Optional flags:

--language-model-only — skip the vision encoder (more KV cache for text-only serving)
--tensor-parallel-size N — split across GPUs if one card is not enough
--default-chat-template-kwargs '{"enable_thinking": false}' — disable thinking server-wide
YaRN to ~1M context (from the base model card):

bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 ./serve.sh
# then add to the vllm serve invocation:
#   --hf-overrides '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144}}'
#   --max-model-len 1000000

Files

Table with columns: File, Description
File	Description
`model.safetensors`	Quantized weights
`config.json`	Model config + `quantization_config` (`TorchAoConfig`)
`generation_config.json`	Generation defaults from the base model
`processor_config.json`	Image/video processor (required by vLLM)
`tokenizer.json` / /

Limitations

Quantization quality has not been formally benchmarked against the full-precision base model; validate on your tasks before production use.
MXFP8 kernels and TorchAO MX support are still evolving; pin compatible torch / torchao / vllm versions for reproducibility.
Vision, lm_head, GDN fused projections, and MoE routers run in bf16. On-disk savings are concentrated in packed routed experts (gate_up_proj / down_proj) plus attention and shared-expert Linear layers.
Transformers eager MoE (experts_implementation="eager") is incompatible with these packed MXFP8 experts; use grouped_mm or batched_mm as above so the 3D tensors are used whole.
vLLM cannot keep packed experts as MXFP8: FusedMoE splits fused with (), which does not implement. Serve only after dequantizing those 3D packs on load; expert VRAM is then ~bf16. Linear MXFP8 weights are unaffected.

License

MIT. Follow the license terms of ornith-ai/Ornith-1.5-35B-A3B.

ornith-1.5-35b-a3b-mxfp8

README

Quantization details

Hardware requirements

Software requirements

Usage

Text-only

Image + text

Serving with vLLM

Files

Limitations

License

Explore FriendliAI today

README

Quantization details

Hardware requirements

Software requirements

Usage

Text-only

Image + text

Serving with vLLM

Files

Limitations

License