What is quantized
Table with columns: Tensors, Weights, Input activations| Tensors | Weights | Input activations |
|---|
Linear attention (Gated DeltaNet): in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj | INT4, per output channel | INT8, per token (dynamic) |
Full attention: q_proj (incl. output gate), k_proj, v_proj, o_proj | INT4, per output channel | INT8, per token (dynamic) |
Routed experts (256 per layer): gate_proj, up_proj, down_proj | INT4, per output channel | INT8, per token (dynamic) |
Shared expert: gate_proj, up_proj, down_proj | INT4, per output channel | INT8, per token (dynamic) |
MoE router mlp.gate, shared-expert gate mlp.shared_expert_gate | INT8, per output channel | INT8, per token (dynamic) |
lm_head | INT8, per output channel | INT8, per token (dynamic) |
| MTP layer (same modules as a decoder layer) | as above | as above |
These tensors are left in BF16, as in the base model:
embed_tokens: a table lookup, not a matmul (quantize per row if needed);
linear_attn.conv1d: the 4-tap depthwise causal convolution;
- all RMSNorm weights,
A_log, dt_bias;
mtp.fc;
- the vision encoder (
model.visual.*).
The checkpoint doesn't prescribe a precision for non-matmul math (norms, SiLU, sigmoid gates, softmax, RoPE, the Gated
DeltaNet recurrent state) or for the KV cache. Choose fixed-point formats for those to suit the target hardware.
Activation quantization (online)
Every quantized module in the table above runs as an integer matmul. Its input is quantized on the fly, separately
for each token (each row of the input) and for each module input, as follows:
x : one token's input vector to the module, length K (hidden size or intermediate size)s_x = max(max_k |x_k|, 1e-12) / 127 # FP32, computed at runtime, never storedq_x[k] = clamp(round(x_k / s_x), -127, 127) # INT8; round half to even; symmetric (no -128)acc[o] = sum_k q_x[k] * q_w[o, k] # exact integer accumulation, fits in INT32y[o] = acc[o] * s_x * s_w[o] # rescale (FP32 or a fixed-point multiplier)
- Granularity: one scale per token per module input. There are no per-channel or per-group activation scales, no
static or calibrated scales, and no outlier handling (rotation or smoothing). The weight scale
s_w[o] is per output
channel, so it factors out of the sum and the whole reduction stays in integers.
- Shared inputs: modules that read the same vector see the same quantized input, so it can be quantized once and
reused:
q/k/v_proj;
in_proj_qkv/z/a/b;
- the router, the shared-expert gate and the
gate_proj/up_proj of the shared and routed experts (all read the
post-attention RMSNorm output).
- Second-stage inputs are quantized again with their own per-token scale:
down_proj input: silu(gate) * up, per token and per selected expert (and separately for the shared expert);
out_proj input (Gated DeltaNet): the gated RMSNorm output;
o_proj input: the attention output after the sigmoid output gate;
lm_head input: the final RMSNorm output.
- Accumulator range: INT4 x INT8 with K ≤ 4096 gives |acc| ≤ 4096·8·127 ≈ 4.2 M. INT8 x INT8 (router,
lm_head;
K = 2048) gives |acc| ≤ 2048·128·127 ≈ 33.3 M. Both fit comfortably in INT32.
- Outputs: matmul outputs (
y) are higher precision. Everything between matmuls (residual stream, norms,
activations, attention, the recurrent state) is outside this spec.
Standard compressed-tensors pack-quantized format, with three
tensors per quantized module:
Table with columns: Tensor, dtype / shape, Meaning| Tensor | dtype / shape | Meaning |
|---|
<module>.weight_packed | int32 [out, in·bits/32] | integer codes packed along the input dimension |
<module>.weight_scale | bfloat16 [out, 1] | per-output-channel scale s_w |
<module>.weight_shape | int [2] | [out, in] |
- Codes: INT4 codes are in [-8, 7] and INT8 codes in [-128, 127]. The dequantized weight is
w = q_w * s_w; there
is no zero point.
- Packing: within each row, element
k occupies bits [k·b, k·b + b) of the row's little-endian int32 words (LSB
first), so INT4 has 8 codes per word and INT8 has 4. Each code is stored as the unsigned value q + 2^(b-1). All
input dimensions are multiples of 32, so no code crosses a word boundary.
The quantization config in config.json lists every module's scheme: weights (bits, strategy: channel,
symmetric: true) and input_activations (num_bits: 8, type: int, strategy: token, dynamic: true).
Reference implementation (PyTorch)
import torch def unpack(packed: torch.Tensor, bits: int, shape) -> torch.Tensor: """weight_packed (int32) -> integer codes (int8), shape [out, in].""" out_f, in_f = (int(v) for v in shape) per = 32 // bits p = packed.to(torch.int64) & 0xFFFFFFFF # int32 words as unsigned shifts = torch.arange(per, dtype=torch.int64) * bits u = (p.unsqueeze(-1) >> shifts) & ((1 << bits) - 1) # element k of a row: bits [k*b, k*b+b), LSB first return (u.reshape(out_f, -1)[:, :in_f] - (1 << (bits - 1))).to(torch.int8) def quantize_activation(x: torch.Tensor): """Online per-token INT8: one scale per row of x ([tokens, K]).""" s = x.float().abs().amax(-1, keepdim=True).clamp_min(1e-12) / 127.0 q = torch.round(x.float() / s).clamp(-127, 127).to(torch.int8) return q, s def int_linear(x, q_w, s_w): """y = x @ W^T with INT8 activations, integer weights and an exact integer accumulation.""" q_x, s_x = quantize_activation(x) acc = q_x.cpu().long() @ q_w.cpu().long().T # exact integer accumulation (fits in INT32) return acc.float() * s_x.cpu() * s_w.cpu().float().view(1, -1) # e.g. for a module m with b = 4 (INT4) or 8 (INT8):# q_w = unpack(sd[f"{m}.weight_packed"], b, sd[f"{m}.weight_shape"].tolist())# y = int_linear(x, q_w, sd[f"{m}.weight_scale"])
unpack matches compressed-tensors' own unpacker bit-for-bit. int_linear matches a floating-point
quantize-dequantize forward to within float rounding (~1e-7 relative).
Quality
Measured against the official FP8 model, teacher-forced on 448 responses that the FP8 model generated (about 1.01 M
response tokens). The prompts are 64 each from MATH-500, HumanEval, OpenCodeReasoning, SWE-bench Lite, IFEval, MMLU-Pro
and UltraChat. Each model was scored in 4 passes. The metric is
EAR (expected acceptance rate): the mean over positions of Σ_k min(P_ref(k), Q(k)), i.e. 1 − total variation
distance over the reference's top-64 tokens. It is the probability that the two models emit the same next token under
an optimal coupling. Activations were quantized exactly as specified above, emulated by quantize-dequantize.
Table with columns: Model, Weights, Activations, EAR vs FP8| Model | Weights | Activations | EAR vs FP8 |
|---|
| FP8 (official) vs itself, run-to-run noise | FP8 | FP8 | 99.85% |
Same weight method, INT8 attention / shared expert / lm_head (not published) | INT4 experts, INT8 rest | 16-bit | 96.8% |
| This model | INT4 (INT8 lm_head, router) | INT8 per token | 90.7% (±0.2) |
Expect a noticeable quality drop compared with the FP8 or BF16 model: on average, the next-token distribution
differs from the FP8 model's by about 9% of its probability mass (total variation). The gap to the 96.8% row comes from
two changes made together: INT4 instead of INT8 for the non-expert linears, and INT8 per-token activations with no
outlier handling. If accuracy matters more than size, those are the two things to revisit.
Running on GPUs
This checkpoint targets integer hardware. GPU engines have no kernels for INT4/INT8 weights with dynamic INT8
activations, and the router and shared-expert gate are stored as integers. To evaluate on a GPU, dequantize the
weights (q_w * s_w) and apply the quantize-dequantize of quantize_activation to every quantized module's input. The
quality numbers above were measured exactly that way.
License
Apache 2.0, the same as the base model.