keenanpepper

Kimi-K2.6-FP8-fused

README

License: modified-mit

Why this exists

As of September 2026 there is no published Kimi K2.6 checkpoint that runs under stock transformers:

moonshotai/Kimi-K2.6 is int4 compressed-tensors (~571 GB of packed experts). vLLM and SGLang serve it on 8×H200 because they have W4A16 kernels that dequantize inside the matmul. transformers has no such kernel: its only gated kernel path is FP8, so int4 falls through to the generic compressed-tensors route, whose forward pre-hook decompresses the whole model to bf16 in place on first use. That is ~1.015 T expert parameters × 2 bytes ≈ 2 TB resident, against 1128 GB of node. It OOMs, always.
RedHatAI/Kimi-K2.6-FP8-BLOCK is the right numbers in the wrong wrapper. Its 128×128 block scales are the same grid and orientation DeepSeek-V3 and GLM-5 FP8 checkpoints use, but transformers' compressed-tensors FP8 kernel (CompressedTensorsFP8Linear) is row-wise only: it allocates weight_scale as [1, out_features], a block-scaled checkpoint does not fit it, and the loader falls back to the same decompress-everything route as above.

The difference between FP8-BLOCK and a checkpoint transformers runs natively is a tensor name, a dtype, and a config block. This repo is that difference, applied.

Who this is for: anyone who needs K2.6 inside transformers rather than behind a serving engine. Interpretability work, activation and residual-stream logging, custom decoding and cache experiments, anything that hooks modules directly. If you only want tokens out, use vLLM or SGLang with the original int4 or with FP8-BLOCK; they are faster and better tested for that.

What changed, exactly

Step 1: relabel (`tools/fp8_relabel.py`)

Table with columns: source (compressed-tensors, block), here (finegrained_fp8)
	source (`compressed-tensors`, block)	here (`finegrained_fp8`)
scale tensor name	`<proj>.weight_scale`	`<proj>.weight_scale_inv`
scale dtype	bf16	fp32 (exact upcast)
scale shape	`[ceil(out/128), ceil(in/128)]`	same
`quantization_config`

69 486 scale tensors renamed and recast. No FP8 weight byte was read or written. The skip list was checked on the meta device: the set of modules the list would quantize equals the set that carries a scale on disk.

Step 2: fuse (`tools/fp8_fuse_experts.py`)

transformers declares this conversion for the model and applies it at load time:

markdown
source: ['mlp.experts.*.gate_proj.weight', 'mlp.experts.*.up_proj.weight']
target: ['mlp.experts.gate_up_proj']
ops   : [MergeModulelist(dim=0), Concatenate(dim=1)]

On K2.6 that stack is 384 × 2048 × 7168 FP8 = 5.25 GiB per half, built on whichever GPU the layer landed on. With 61 layers over 8 cards, at least five cards hold 8 layers (~127 GiB resident) and have ~12 GiB left to absorb it. Measured allocator fragmentation on an 8×H200 node came in at 6.70–7.27 GiB against a 6.68 GiB bar, so the un-fused load succeeded roughly one attempt in six, with byte-identical settings.

MergeModulelist passes an already-stacked tensor straight through, so a checkpoint that ships the fused tensor never pays the transient. This step does that stack and concatenation once, offline, on CPU, in the exact layout transformers' FP8Experts allocates:

Table with columns: fused tensor, shape, dtype
fused tensor	shape	dtype
`mlp.experts.gate_up_proj`	`[384, 4096, 7168]`	float8_e4m3fn
`mlp.experts.gate_up_proj_scale_inv`	`[384, 32, 56]`	float32
`mlp.experts.down_proj`	`[384, 7168, 2048]`	float8_e4m3fn

92 fused tensors written from 52 992 per-expert tensors, across the 60 expert-bearing layers (layer 0 is dense). Every input byte appears in the output exactly once. Shapes were asserted against the config, not assumed.

Note the fused scale name: gate_up_proj_scale_inv, a sibling parameter, not gate_up_proj.weight_scale_inv. FP8Experts holds the projection as a bare Parameter, not a Linear, and a wrongly named scale key does not raise: the load completes with the checkpoint key reported UNEXPECTED, the parameter MISSING, and the scales left as uninitialised memory. This repo's keys were verified by reading every fused scale back after the fix (see FUSE.json, repaired). If you ever re-derive this checkpoint, read the load report.

Loading

Tested with transformers==5.15.0, torch==2.13.0+cu130, 8×H200 (141 GB), bf16 activations. Loads in about 7 minutes and has come up first try every time.

python
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"  # set before torch imports

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

path = "keenanpepper/Kimi-K2.6-FP8-fused"

model = AutoModelForImageTextToText.from_pretrained(
    path,
    trust_remote_code=False,           # native kimi_k25 class; no repo code needed
    dtype=torch.bfloat16,
    device_map="auto",
    max_memory={i: "128GiB" for i in range(8)},
    attn_implementation="eager",
)
model.eval()

# Tokenizer: use the repo's TikTokenTokenizer, NOT the converted fast tokenizer
# (see "Tokenizer" below for why).
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Things in that snippet that are pinned rather than chosen:

max_memory of 128 GiB per card, from both sides. 61 layers over 8 cards means at least five cards must hold 8 layers; the largest layer is 15.91 GiB, so the cap must admit ≥127.3 GiB. A cap in [126, 127] GiB admits only 7 layers per card (56 < 61) and device_map="auto" then silently spills the rest to CPU or disk. Do not derive the cap from the average layer size. Print model.hf_device_map and check that nothing landed on cpu or disk.
expandable_segments:True. Without it, measured allocator fragmentation during load of the un-fused checkpoint tripled (~21 GiB). It has not been re-measured without the setting on the fused checkpoint, so keep it. Set it in the environment before torch is imported.
Multi-GPU in one process routes FP8 through Triton / grouped_mm, not DeepGEMM. transformers logs this at load. DeepGEMM's cached kernels are bound to one CUDA context; the Triton path is correct across devices, just slower.
Use or the class directly.

Steady state after load is comfortable: roughly 20 GiB free per card with the weights resident, which is plenty of room for KV cache and activations at ordinary context lengths.

Tokenizer

The checkpoint ships a tiktoken BPE vocabulary (tiktoken.model) and Moonshot's tokenization_kimi.py. With trust_remote_code=False, transformers converts the vocabulary into a fast tokenizer, and that conversion renumbers 7 of the 12 added control tokens, packing them contiguously and skipping the id gaps the checkpoint leaves:

Table with columns: literal, converted (wrong), checkpoint (right)
literal	converted (wrong)	checkpoint (right)
`<\|start_header_id\|>`	163589	163590
`<\|end_header_id\|>`	163590	163591
`[EOT]`	163591	163593
`<\|im_system\|>`	163592	163594

The converted tokenizer is self-consistent (it round-trips its own ids), so nothing raises and the model still produces coherent text, because <|im_end|> (163586) happens to survive. But the chat template is then feeding the model <|media_content|> where it should see <think>. The model itself settles which numbering is correct: asked to think, it emits 163607 to close its reasoning. Load the tokenizer with trust_remote_code=True so the repo's TikTokenTokenizer is used, and the ids match the embedding table.

transformers also warns that the tiktoken-converted pre-tokenizer regex may differ from the original (fix_mistral_regex). That warning was not investigated for this release; it concerns the fast-tokenizer conversion, which the recommendation above avoids.

Verification, and its limits

Verified with an engine-vs-reference harness that runs a separate implementation of the same architecture against transformers' own forward on these weights, on 8×H200 in bf16:

Table with columns: check, result
check	result
full-prefill logits vs `transformers`	max abs diff 0.875 on a logit scale of 25 (3.5 % relative)
incremental decode logits vs `transformers`	1.19 (4.75 % relative)
free-running greedy continuation	token-for-token match
`transformers` vs itself (prefill path vs decode path, same weights)	0.70

The independent engine sits inside transformers' own self-disagreement between its two code paths, and the residual differences are what bf16 activations over 61 layers of E4M3 weights predict (a 16B unquantized bf16 control on the same kernels shows ~2 % relative; fp32 on the same code shows ~1e-6). A 1 % relative tolerance is below the floor for a model of this depth in this precision; 5 % passes.

What this does not establish:

Nothing here was compared against the int4 moonshotai/Kimi-K2.6 original. No reference of it fits on one node under transformers. The quantization quality claim rests entirely on RedHatAI's evaluations of FP8-BLOCK, which transfer because the bytes are identical.
The vision tower is untested. It is carried through unchanged (it is in modules_to_not_convert, so it is bf16), and the processor files are included, but only the text path has been exercised.
Only greedy decoding and logit agreement were checked, not benchmarks.

Caveats

This layout is coupled to transformers 5.x internals. The fused tensor names and shapes are whatever FP8Experts in integrations/finegrained_fp8.py allocates in 5.15. If a later release changes that layout or the sibling _scale_inv naming, this checkpoint will load with MISSING and UNEXPECTED keys rather than fail loudly. Check the load report on any new transformers version. The un-fused relabelled form (step 1 alone) is more portable and is ~30 minutes to regenerate (see below).
RedHatAI's card describes FP8-BLOCK as preliminary. This repo is pinned to revision 317300b0ef4ec429ad7296b46e4278ab1922dd08 and will not track upstream changes.
Do not attempt this on fewer than 8×H200-class GPUs without expecting offload. Resident weights are ~1 TB.

Reproducing it

Both scripts are in tools/ and depend only on torch, safetensors, transformers (for --check in the relabel step) and huggingface_hub. Measured wall-clock on a 16-core CPU job with fast storage:

Table with columns: step, command, time
step	command	time
download FP8-BLOCK (1031 GB)	`hf download RedHatAI/Kimi-K2.6-FP8-BLOCK --revision 317300b0ef4ec429ad7296b46e4278ab1922dd08`	~8.5 min
relabel	`python tools/fp8_relabel.py RedHatAI/Kimi-K2.6-FP8-BLOCK --revision 317300b… --out-dir <relabelled>` (the skip-list check always runs first; `--check-only` runs just that)	~30 min
fuse	`python tools/fp8_fuse_experts.py <relabelled> --out-dir <fused>`	~25 min

Each step needs ~1 TB of scratch for its output; neither holds more than one shard in memory. The relabel step refuses to write unless its derived skip list matches the modules that actually carry scales on disk; the fuse step refuses if any layer's experts are split across shards or the geometry disagrees with the config.

Files

Table with columns: file, what
file	what
`model-000NN-of-000064.safetensors`	64 shards, 1031 GB total; each MoE layer's experts live in a single shard
`model.safetensors.index.json`	weight map
`config.json`	`KimiK25ForConditionalGeneration`, `quant_method: fp8`, no `auto_map`
`RELABEL.json`, `FUSE.json`	provenance for each step: source, revision, version, what was renamed or fused

Licence and credits

Weights: Moonshot AI, under the Modified MIT License (MIT plus an attribution requirement for very large commercial deployments). Redistributed with the licence and notices intact.
FP8 quantization: RedHatAI, via LLM Compressor. Their evaluations are in UPSTREAM_README.md.
Relabel and fuse: Keenan Pepper (AE Studio), as part of the Serger project. The conversion scripts in tools/ are released under the MIT licence.

If you find a problem with this repack specifically (as opposed to the model or the quantization), open a discussion on this repo.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

keenanpepper

Model Tree

Base

RedHatAI/Kimi-K2.6-FP8-BLOCK

Quantized

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: modified-mit

Why this exists

As of September 2026 there is no published Kimi K2.6 checkpoint that runs under stock transformers:

moonshotai/Kimi-K2.6 is int4 compressed-tensors (~571 GB of packed experts). vLLM and SGLang serve it on 8×H200 because they have W4A16 kernels that dequantize inside the matmul. transformers has no such kernel: its only gated kernel path is FP8, so int4 falls through to the generic compressed-tensors route, whose forward pre-hook decompresses the whole model to bf16 in place on first use. That is ~1.015 T expert parameters × 2 bytes ≈ 2 TB resident, against 1128 GB of node. It OOMs, always.
RedHatAI/Kimi-K2.6-FP8-BLOCK is the right numbers in the wrong wrapper. Its 128×128 block scales are the same grid and orientation DeepSeek-V3 and GLM-5 FP8 checkpoints use, but transformers' compressed-tensors FP8 kernel (CompressedTensorsFP8Linear) is row-wise only: it allocates weight_scale as [1, out_features], a block-scaled checkpoint does not fit it, and the loader falls back to the same decompress-everything route as above.

The difference between FP8-BLOCK and a checkpoint transformers runs natively is a tensor name, a dtype, and a config block. This repo is that difference, applied.

What changed, exactly

Step 1: relabel (`tools/fp8_relabel.py`)

Table with columns: source (compressed-tensors, block), here (finegrained_fp8)
	source (`compressed-tensors`, block)	here (`finegrained_fp8`)
scale tensor name	`<proj>.weight_scale`	`<proj>.weight_scale_inv`
scale dtype	bf16	fp32 (exact upcast)
scale shape	`[ceil(out/128), ceil(in/128)]`	same
`quantization_config`

Step 2: fuse (`tools/fp8_fuse_experts.py`)

transformers declares this conversion for the model and applies it at load time:

markdown
source: ['mlp.experts.*.gate_proj.weight', 'mlp.experts.*.up_proj.weight']
target: ['mlp.experts.gate_up_proj']
ops   : [MergeModulelist(dim=0), Concatenate(dim=1)]

Table with columns: fused tensor, shape, dtype
fused tensor	shape	dtype
`mlp.experts.gate_up_proj`	`[384, 4096, 7168]`	float8_e4m3fn
`mlp.experts.gate_up_proj_scale_inv`	`[384, 32, 56]`	float32
`mlp.experts.down_proj`	`[384, 7168, 2048]`	float8_e4m3fn

Loading

Tested with transformers==5.15.0, torch==2.13.0+cu130, 8×H200 (141 GB), bf16 activations. Loads in about 7 minutes and has come up first try every time.

python
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"  # set before torch imports

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

path = "keenanpepper/Kimi-K2.6-FP8-fused"

model = AutoModelForImageTextToText.from_pretrained(
    path,
    trust_remote_code=False,           # native kimi_k25 class; no repo code needed
    dtype=torch.bfloat16,
    device_map="auto",
    max_memory={i: "128GiB" for i in range(8)},
    attn_implementation="eager",
)
model.eval()

# Tokenizer: use the repo's TikTokenTokenizer, NOT the converted fast tokenizer
# (see "Tokenizer" below for why).
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Things in that snippet that are pinned rather than chosen:

max_memory of 128 GiB per card, from both sides. 61 layers over 8 cards means at least five cards must hold 8 layers; the largest layer is 15.91 GiB, so the cap must admit ≥127.3 GiB. A cap in [126, 127] GiB admits only 7 layers per card (56 < 61) and device_map="auto" then silently spills the rest to CPU or disk. Do not derive the cap from the average layer size. Print model.hf_device_map and check that nothing landed on cpu or disk.
expandable_segments:True. Without it, measured allocator fragmentation during load of the un-fused checkpoint tripled (~21 GiB). It has not been re-measured without the setting on the fused checkpoint, so keep it. Set it in the environment before torch is imported.
Multi-GPU in one process routes FP8 through Triton / grouped_mm, not DeepGEMM. transformers logs this at load. DeepGEMM's cached kernels are bound to one CUDA context; the Triton path is correct across devices, just slower.
Use or the class directly.

Steady state after load is comfortable: roughly 20 GiB free per card with the weights resident, which is plenty of room for KV cache and activations at ordinary context lengths.

Tokenizer

Table with columns: literal, converted (wrong), checkpoint (right)
literal	converted (wrong)	checkpoint (right)
`<\|start_header_id\|>`	163589	163590
`<\|end_header_id\|>`	163590	163591
`[EOT]`	163591	163593
`<\|im_system\|>`	163592	163594

Verification, and its limits

Verified with an engine-vs-reference harness that runs a separate implementation of the same architecture against transformers' own forward on these weights, on 8×H200 in bf16:

Table with columns: check, result
check	result
full-prefill logits vs `transformers`	max abs diff 0.875 on a logit scale of 25 (3.5 % relative)
incremental decode logits vs `transformers`	1.19 (4.75 % relative)
free-running greedy continuation	token-for-token match
`transformers` vs itself (prefill path vs decode path, same weights)	0.70

What this does not establish:

Nothing here was compared against the int4 moonshotai/Kimi-K2.6 original. No reference of it fits on one node under transformers. The quantization quality claim rests entirely on RedHatAI's evaluations of FP8-BLOCK, which transfer because the bytes are identical.
The vision tower is untested. It is carried through unchanged (it is in modules_to_not_convert, so it is bf16), and the processor files are included, but only the text path has been exercised.
Only greedy decoding and logit agreement were checked, not benchmarks.

Caveats

This layout is coupled to transformers 5.x internals. The fused tensor names and shapes are whatever FP8Experts in integrations/finegrained_fp8.py allocates in 5.15. If a later release changes that layout or the sibling _scale_inv naming, this checkpoint will load with MISSING and UNEXPECTED keys rather than fail loudly. Check the load report on any new transformers version. The un-fused relabelled form (step 1 alone) is more portable and is ~30 minutes to regenerate (see below).
RedHatAI's card describes FP8-BLOCK as preliminary. This repo is pinned to revision 317300b0ef4ec429ad7296b46e4278ab1922dd08 and will not track upstream changes.
Do not attempt this on fewer than 8×H200-class GPUs without expecting offload. Resident weights are ~1 TB.

Reproducing it

Table with columns: step, command, time
step	command	time
download FP8-BLOCK (1031 GB)	`hf download RedHatAI/Kimi-K2.6-FP8-BLOCK --revision 317300b0ef4ec429ad7296b46e4278ab1922dd08`	~8.5 min
relabel	`python tools/fp8_relabel.py RedHatAI/Kimi-K2.6-FP8-BLOCK --revision 317300b… --out-dir <relabelled>` (the skip-list check always runs first; `--check-only` runs just that)	~30 min
fuse	`python tools/fp8_fuse_experts.py <relabelled> --out-dir <fused>`	~25 min

Files

Table with columns: file, what
file	what
`model-000NN-of-000064.safetensors`	64 shards, 1031 GB total; each MoE layer's experts live in a single shard
`model.safetensors.index.json`	weight map
`config.json`	`KimiK25ForConditionalGeneration`, `quant_method: fp8`, no `auto_map`
`RELABEL.json`, `FUSE.json`	provenance for each step: source, revision, version, what was renamed or fused

Licence and credits

Weights: Moonshot AI, under the Modified MIT License (MIT plus an attribution requirement for very large commercial deployments). Redistributed with the licence and notices intact.
FP8 quantization: RedHatAI, via LLM Compressor. Their evaluations are in UPSTREAM_README.md.
Relabel and fuse: Keenan Pepper (AE Studio), as part of the Serger project. The conversion scripts in tools/ are released under the MIT licence.

If you find a problem with this repack specifically (as opposed to the model or the quantization), open a discussion on this repo.

Kimi-K2.6-FP8-fused

README

Why this exists

What changed, exactly

Step 1: relabel (tools/fp8_relabel.py)

Step 2: fuse (tools/fp8_fuse_experts.py)

Loading

Tokenizer

Verification, and its limits

Caveats

Reproducing it

Files

Licence and credits

Explore FriendliAI today

README

Why this exists

What changed, exactly

Step 1: relabel (tools/fp8_relabel.py)

Step 2: fuse (tools/fp8_fuse_experts.py)

Loading

Tokenizer

Verification, and its limits

Caveats

Reproducing it

Files

Licence and credits

Step 1: relabel (`tools/fp8_relabel.py`)

Step 2: fuse (`tools/fp8_fuse_experts.py`)

Step 1: relabel (`tools/fp8_relabel.py`)

Step 2: fuse (`tools/fp8_fuse_experts.py`)