Quantization details
This checkpoint was produced with TorchAO using MXDynamicActivationMXWeightConfig:
Table with columns: Component, Precision| Component | Precision |
|---|
model.language_model Linear layers (attention, GDN in_proj_qkv / in_proj_z / out_proj, shared-expert gate_proj / up_proj / down_proj) | MXFP8 weights; activations quantized dynamically at inference |
Packed MoE experts (Qwen3_5MoeExperts.gate_up_proj, down_proj) | MXFP8 weights (nn.Parameter of shape [num_experts, out, in], not Linear) |
visual (ViT + merger), lm_head | bfloat16 |
Hybrid Gated DeltaNet Conv1d / A_log / dt_bias / in_proj_b / in_proj_a | bfloat16 |
MoE routers (mlp.gate TopK router, shared_expert_gate) | bfloat16 |
- Format: torchao-flattened
safetensors (MXTensor qdata/scale + metadata)
- Block size: 32
- Dtypes:
float8_e4m3fn for weights and activations
- Scaling: RCEIL
- Base dtype: bfloat16
Packed experts are 3D parameters (gate_up_proj: [256, 1024, 2048], down_proj: [256, 2048, 512]). TorchAO's Linear-only quantize_ path skips them, so they are converted with FqnToConfig on those parameter FQNs. Shared-expert Linear projections are MXFP8; that is distinct from the packed routed experts.
Gated DeltaNet in_proj_b / in_proj_a stay bf16 because vLLM fuses them into in_proj_ba with output sizes [48, 48]; swizzled MXFP8 scales cannot be row-sliced at 48 (they need 128-alignment). mlp.gate is a Qwen3_5MoeTopKRouter (nn.Parameter, not Linear), so TorchAO skips it. shared_expert_gate is a Linear and is explicitly left in bf16.
Weights were quantized once on GPU, exported to CPU, flattened with flatten_tensor_state_dict, and saved with a TorchAoConfig in config.json. Reload does not re-run weight quantization; the language model still applies dynamic activation quantization during forward passes.
Hardware requirements
MXFP8 inference requires a Blackwell-class NVIDIA GPU (compute capability SM100+, i.e. major version ≥ 10). Examples include B200, GB200, and RTX Pro 6000. Older architectures (Ampere, Hopper, etc.) are not supported for this checkpoint.
- CUDA GPU with SM100+
- Sufficient VRAM for a 35B-A3B multimodal MoE (peak usage depends on sequence length, runtime, and vision inputs). The bf16 base is ≈70 GB; this MXFP8 checkpoint is ≈37 GB on disk (packed routed experts are the bulk of that saving). Transformers with
experts_implementation="grouped_mm" keeps those packs as MXTensors in memory. vLLM's FusedMoE loader cannot chunk/unbind swizzled MXFP8 experts, so it dequantizes them to bf16 on load (~64 GB for experts plus MXFP8 Linears) unless you patch the loader. The official card recommends 2× 80GB GPUs for the full 256K context.
Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True if you hit fragmentation during load or generation.
Software requirements
pip install "transformers>=5.8.1" torch torchao safetensors
You need a recent torchao build with MXFP8 inference support. For serving, use a recent vllm with TorchAO MXFP8 support (nightly or a source build is typical). The base model card asks for vLLM ≥ 0.19.1.
Load the processor from the same directory as the weights (vLLM does this automatically; there is no --processor flag):
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGenerationimport torch QUANTIZED_MODEL = "YOUR_USERNAME/ornith-1.5-35b-a3b-mxfp8" # or local path processor = AutoProcessor.from_pretrained(QUANTIZED_MODEL)model = Qwen3_5MoeForConditionalGeneration.from_pretrained( QUANTIZED_MODEL, torch_dtype=torch.bfloat16, experts_implementation="grouped_mm",)model.to("cuda")model.eval()
Ornith-1.5-35B-A3B is a Qwen3.5-MoE VLM and uses Qwen3_5MoeForConditionalGeneration. Packed expert weights are swizzled MXTensors; the default eager expert loop indexes gate_up_proj[expert_idx] / down_proj[expert_idx], which cannot slice swizzled scales at a single expert row (128-alignment). Pass experts_implementation="grouped_mm" (or "batched_mm") so the 3D tensors are used whole.
Usage
This is a reasoning model: by default the assistant turn opens with a <think> … </think> block before the final answer. Pass enable_thinking=False to apply_chat_template to skip the thinking block.
Text-only
messages = [ {"role": "user", "content": "Explain MXFP8 in one sentence."},]inputs = processor.apply_chat_template( messages, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True, enable_thinking=True,)inputs = inputs.to(model.device) with torch.inference_mode(): output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False) response = processor.decode( output_ids[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=False,)print(response)
Image + text
from PIL import Image image = Image.open("example.png").convert("RGB")messages = [ { "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": "What is shown in this image?"}, ], }]# Same apply_chat_template → generate → decode flow as above.
Serving with vLLM
vLLM applies TorchAO only to LinearBase layers. Packed routed experts restore as 3D MXTensors from flattened safetensors metadata, then RoutedExperts.load_weights identifies fused packs by dim() == 3 and runs chunk(2, dim=1) / unbind(). MXTensor does not implement aten.split, and swizzled scales cannot slice a single expert row (128-alignment). Stock vllm serve therefore dies at engine init:
NotImplementedError: MXTensor dispatch: ... aten.split
A load-time dequant of those 3D packs (this repo: scripts/vllm_ornith.sh via scripts/vllm_ornith_patch/sitecustomize.py) lets FusedMoE finish loading. Experts then run on vLLM's unquantized MoE backend in bf16; attention / GDN / shared-expert Linears stay MXFP8. Size --max-num-seqs / --gpu-memory-utilization for ~bf16 expert weights, not the 37 GB file.
vLLM still loads the Qwen3 VL image/video processor from the model directory, so processor_config.json must sit next to the weights. --tokenizer is optional once those sidecars are present.
Save the following as serve.sh (or run it inline). Put a dequant-on-load shim on PYTHONPATH first (see scripts/vllm_ornith.sh in this repo) or engine core init will fail as above:
#!/usr/bin/env bashset -euo pipefail export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True# TorchAO MX kernels and vLLM compile cache currently compose poorly.export VLLM_DISABLE_COMPILE_CACHE=1# Required: dequantize 3D MXTensor expert packs before FusedMoE .chunk/.unbind.# export PYTHONPATH=/path/to/quant/scripts/vllm_ornith_patch${PYTHONPATH:+:$PYTHONPATH} MODEL=mph/ornith-1.5-35b-a3b-mxfp8TOKENIZER=ornith-ai/Ornith-1.5-35B-A3BSERVED_NAME=ornith-1.5-35b-a3b-mxfp8PORT=8000MAX_MODEL_LEN=5000 vllm serve "$MODEL" \ --tokenizer "$TOKENIZER" \ --served-model-name "$SERVED_NAME" \ --host 0.0.0.0 \ --port "$PORT" \ --max-model-len "$MAX_MODEL_LEN" \ --max-num-seqs 125 \ --gpu-memory-utilization 0.94 \ --attention-backend FLASHINFER \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --mm-encoder-tp-mode data
chmod +x serve.sh./serve.sh
The server exposes an OpenAI-compatible API at http://localhost:8000/v1. With --reasoning-parser qwen3 and --tool-call-parser qwen3_xml, chain-of-thought is returned in reasoning_content and <tool_call> blocks are surfaced as OpenAI-style tool_calls.
Recommended sampling (from the base model card):
Table with columns: Mode, temperature, top_p, top_k| Mode | temperature | top_p | top_k |
|---|
| General tasks | 0.6 | 0.95 | 20 |
| Reproduce reported benchmarks | 1.0 | 0.95 | 20 |
curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "ornith-1.5-35b-a3b-mxfp8", "messages": [ {"role": "user", "content": "Explain MXFP8 in one sentence."} ], "temperature": 0.6, "top_p": 0.95, "max_tokens": 1024, "chat_template_kwargs": { "enable_thinking": true } }'
from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")resp = client.chat.completions.create( model="ornith-1.5-35b-a3b-mxfp8", messages=[{"role": "user", "content": "Explain MXFP8 in one sentence."}], temperature=0.6, top_p=0.95, extra_body={ "top_k": 20, "chat_template_kwargs": {"enable_thinking": True}, },)message = resp.choices[0].messageprint(getattr(message, "reasoning_content", None))print(message.content)
Optional flags:
--language-model-only — skip the vision encoder (more KV cache for text-only serving)
--tensor-parallel-size N — split across GPUs if one card is not enough
--default-chat-template-kwargs '{"enable_thinking": false}' — disable thinking server-wide
- YaRN to ~1M context (from the base model card):
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 ./serve.sh# then add to the vllm serve invocation:# --hf-overrides '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144}}'# --max-model-len 1000000
Open-source runtimes apply YaRN statically to every request. Only enable it when the workload needs the longer window; size factor so that factor × 262144 covers the target length (use factor: 2.0 around 524K tokens).
Files
Table with columns: File, Description| File | Description |
|---|
model.safetensors | Quantized weights |
config.json | Model config + quantization_config (TorchAoConfig) |
generation_config.json | Generation defaults from the base model |
processor_config.json | Image/video processor (required by vLLM) |
tokenizer.json / tokenizer_config.json / chat_template.jinja | Tokenizer + chat template |
manifest.json | Provenance and quantization summary |
Limitations
- Quantization quality has not been formally benchmarked against the full-precision base model; validate on your tasks before production use.
- MXFP8 kernels and TorchAO MX support are still evolving; pin compatible
torch / torchao / vllm versions for reproducibility.
- Vision,
lm_head, GDN fused projections, and MoE routers run in bf16. On-disk savings are concentrated in packed routed experts (gate_up_proj / down_proj) plus attention and shared-expert Linear layers.
- Transformers eager MoE (
experts_implementation="eager") is incompatible with these packed MXFP8 experts; use grouped_mm or batched_mm as above so the 3D tensors are used whole.
- vLLM cannot keep packed experts as MXFP8: FusedMoE splits fused
gate_up_proj with .chunk() (aten.split), which MXTensor does not implement. Serve only after dequantizing those 3D packs on load; expert VRAM is then ~bf16. Linear MXFP8 weights are unaffected.
License
MIT. Follow the license terms of ornith-ai/Ornith-1.5-35B-A3B.