PakornF

MANGO1.5-Qwen3.5-9B-W8A8-INT8-mtp

README

License: apache-2.0

Why W8A8 on A100

The A100 has INT8 tensor cores (about twice its BF16 throughput) but no FP8. Weight-only 4-bit models (W4A16, e.g. AWQ) store small weights but still compute in BF16, so they do not speed up prompt processing. W8A8 runs the matrix multiplies in INT8; vLLM serves it with CutlassInt8ScaledMMLinearKernel.

About this quantization

Base: CMKL/MANGO1.5-Qwen3.5-9B (official BF16 weights)
Tool: llm-compressor, QuantizationModifier(targets="Linear", scheme="W8A8")
Scheme: INT8 symmetric per-channel weights; INT8 dynamic per-token activations
Calibration data: none (data-free: no dataset was used)
Kept in BF16 (not quantized): lm_head, the vision tower, the MTP head, and the small Gated-DeltaNet gate projections (in_proj_a, in_proj_b, in_proj_ba)
MTP head: the 15 mtp.* tensors are copied unchanged (BF16) from the base checkpoint
Format: compressed-tensors (the quantization settings are in config.json)

Results (GreenMind Final test set, 1x A100-SXM4 40GB, vLLM 0.32.0)

Measured by the hackathon grader against the organizers' baseline (BF16 base model, default vllm serve):

Best score 99.7 / 100; median 99.2 over 7 repeated Final runs with the configuration below
Accuracy relative to the BF16 baseline (R): 0.998 to 1.053 across all our INT8 Final runs
On-time requests per joule: up to 3.64x the baseline (164 J per request)
MTP with 2 draft tokens: about 2.7 tokens accepted per decoding step

How to use (vLLM)

bash

vllm serve PakornF/MANGO1.5-Qwen3.5-9B-W8A8-INT8-mtp \  --max-model-len 49152 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

The configuration we used for the scores above adds:

bash

  --gpu-memory-utilization 0.92 --enable-prefix-caching \  --max-num-seqs 64 --max-num-batched-tokens 2048 --long-prefill-token-threshold 1024 \  --scheduling-policy priority

with a small wrapper that gives shorter prompts a higher priority (see the GitHub repository).

License

Apache 2.0, inherited from the base model (Qwen3.5-9B license).

Credits

Base model: CMKL/MANGO1.5-Qwen3.5-9B by CMKL University, built on Qwen3.5-9B
Quantization: llm-compressor
Serving: vLLM

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

PakornF

Model Tree

Base

CMKL/MANGO1.5-Qwen3.5-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Why W8A8 on A100

About this quantization

Base: CMKL/MANGO1.5-Qwen3.5-9B (official BF16 weights)
Tool: llm-compressor, QuantizationModifier(targets="Linear", scheme="W8A8")
Scheme: INT8 symmetric per-channel weights; INT8 dynamic per-token activations
Calibration data: none (data-free: no dataset was used)
Kept in BF16 (not quantized): lm_head, the vision tower, the MTP head, and the small Gated-DeltaNet gate projections (in_proj_a, in_proj_b, in_proj_ba)
MTP head: the 15 mtp.* tensors are copied unchanged (BF16) from the base checkpoint
Format: compressed-tensors (the quantization settings are in config.json)

Results (GreenMind Final test set, 1x A100-SXM4 40GB, vLLM 0.32.0)

Measured by the hackathon grader against the organizers' baseline (BF16 base model, default vllm serve):

Best score 99.7 / 100; median 99.2 over 7 repeated Final runs with the configuration below
Accuracy relative to the BF16 baseline (R): 0.998 to 1.053 across all our INT8 Final runs
On-time requests per joule: up to 3.64x the baseline (164 J per request)
MTP with 2 draft tokens: about 2.7 tokens accepted per decoding step

How to use (vLLM)

bash

vllm serve PakornF/MANGO1.5-Qwen3.5-9B-W8A8-INT8-mtp \  --max-model-len 49152 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

vllm serve PakornF/MANGO1.5-Qwen3.5-9B-W8A8-INT8-mtp \  --max-model-len 49152 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

The configuration we used for the scores above adds:

bash

  --gpu-memory-utilization 0.92 --enable-prefix-caching \  --max-num-seqs 64 --max-num-batched-tokens 2048 --long-prefill-token-threshold 1024 \  --scheduling-policy priority

  --gpu-memory-utilization 0.92 --enable-prefix-caching \  --max-num-seqs 64 --max-num-batched-tokens 2048 --long-prefill-token-threshold 1024 \  --scheduling-policy priority

with a small wrapper that gives shorter prompts a higher priority (see the GitHub repository).

License

Apache 2.0, inherited from the base model (Qwen3.5-9B license).

Credits

Base model: CMKL/MANGO1.5-Qwen3.5-9B by CMKL University, built on Qwen3.5-9B
Quantization: llm-compressor
Serving: vLLM