Tiel-Coder-35B-A3B-Genesis-Hermes-MXFP4 API & Inference Endpoint

Trunk: all 30,720 routed-expert projections (source stores fused experts.gate_up_proj / stacked down_proj — unfused per expert here) + 150 linear-attention projections, quantized with the inference kernel's own quantizer (torch.ops.vllm.xpu_mxfp4_quantize — format match by construction). Everything else (self-attn, norms, shared experts, embed, lm_head, vision) is unchanged BF16.
MTP draft block (mtp.*): 768 experts unfused and MXFP4-quantized, 17 support tensors BF16 — ready for --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. (Saves ~1.1 GiB vs the BF16 draft: 20.81 → 19.71 GiB resident.)

64,089 tensors / 32 shards / 20.51 GiB. SHA256SUMS included.

Measured quality of the quantization

In-conversion spot checks (kernel dequantizer, 1-in-64): max mean rel-err ~0.12.
Independent post-hoc CPU audit: mean rel-err ~0.12, max 0.145.

Measured serving (Intel Arc 140V, vLLM 0.30.0+xpu, eager)

Table with columns: Config, 1024/128/1 decode, 2048/128/1 decode
Config	1024/128/1 decode	2048/128/1 decode
Eager	19.15 tok/s	19.51 tok/s
MTP K=1 (MXFP4 draft)	24.85–25.41 tok/s	24.87 tok/s
MTP K=2 (MXFP4 draft)	27.69 tok/s	26.72 tok/s

Golden-suite prompts clean, 0 errors. Example launch:

bash

vllm serve Tiel-Coder-35B-A3B-Genesis-Hermes-MXFP4 --served-model-name Tiel \  --max-model-len 4096 --max-num-seqs 1 --gpu-memory-utilization 0.68 \  --language-model-only --enforce-eager \  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \  --kv-cache-memory-bytes 500000000

Provenance

Built with the quantize_tiel_mxfp4.py / unfuse_tiel_mtp.py / quantize_tiel_mtp_mxfp4.py / finalize_tiel_mxfp4.py / audit_tiel_mxfp4.py pipeline (see alpha2beta/vllm.xpu). Base weights: symrex/Tiel-Coder-35B-A3B-Genesis-Hermes-GGUF-dequantized (BF16). Reference schema: pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4.

Trunk: all 30,720 routed-expert projections (source stores fused experts.gate_up_proj / stacked down_proj — unfused per expert here) + 150 linear-attention projections, quantized with the inference kernel's own quantizer (torch.ops.vllm.xpu_mxfp4_quantize — format match by construction). Everything else (self-attn, norms, shared experts, embed, lm_head, vision) is unchanged BF16.
MTP draft block (mtp.*): 768 experts unfused and MXFP4-quantized, 17 support tensors BF16 — ready for --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. (Saves ~1.1 GiB vs the BF16 draft: 20.81 → 19.71 GiB resident.)

64,089 tensors / 32 shards / 20.51 GiB. SHA256SUMS included.

Measured quality of the quantization

In-conversion spot checks (kernel dequantizer, 1-in-64): max mean rel-err ~0.12.
Independent post-hoc CPU audit: mean rel-err ~0.12, max 0.145.

Measured serving (Intel Arc 140V, vLLM 0.30.0+xpu, eager)

Table with columns: Config, 1024/128/1 decode, 2048/128/1 decode
Config	1024/128/1 decode	2048/128/1 decode
Eager	19.15 tok/s	19.51 tok/s
MTP K=1 (MXFP4 draft)	24.85–25.41 tok/s	24.87 tok/s
MTP K=2 (MXFP4 draft)	27.69 tok/s	26.72 tok/s

Golden-suite prompts clean, 0 errors. Example launch:

bash

vllm serve Tiel-Coder-35B-A3B-Genesis-Hermes-MXFP4 --served-model-name Tiel \  --max-model-len 4096 --max-num-seqs 1 --gpu-memory-utilization 0.68 \  --language-model-only --enforce-eager \  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \  --kv-cache-memory-bytes 500000000

Tiel-Coder-35B-A3B-Genesis-Hermes-MXFP4

README

Contents

Measured quality of the quantization

Measured serving (Intel Arc 140V, vLLM 0.30.0+xpu, eager)

Provenance

Explore FriendliAI today

README

Contents

Measured quality of the quantization

Measured serving (Intel Arc 140V, vLLM 0.30.0+xpu, eager)

Provenance