GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
RedHatAI
Model Tree
moonshotai/Kimi-K2.6
Input Modalities
Output Modalities
Supported Functionality
KimiK25ForConditionalGeneration)This model is a quantized variant of moonshotai/Kimi-K2.6, exported in compressed-tensors format for vLLM deployment and evaluated on instruction-following, reasoning, function-calling, and agentic coding workloads.
This checkpoint applies NVFP4 quantization to transformer linear layers with group-wise FP4 weights and activations, using FP8 scale tensors. The format is optimized for efficient low-precision serving while preserving strong benchmark quality on Kimi-K2.6 evaluations.
The model is exported in compressed-tensors format and is intended for OpenAI-compatible inference with vLLM.
This model was quantized with LLM Compressor and exported as compressed-tensors. The script below is a representative reference script aligned with recipe.yaml and the published quantization configuration.
python
from compressed_tensors.entrypoints.convert import CompressedTensorsDequantizerfrom llmcompressor import model_free_ptq MODEL_ID = "moonshotai/Kimi-K2.6"SAVE_DIR = "Kimi-K2.6-NVFP4" ignore = [ "re:.*mlp.gate$", "re:.*lm_head", "re:.*self_attn.*", "re:.*kv_a_proj_with_mqa$", "re:.*q_a_proj$", "re:.*vision_tower.*", "re:.*embed_tokens$", "re:.*norm$", "re:.*mm_projector.*", "re:.*vision.*",] model_free_ptq( model_stub=MODEL_ID, save_directory=SAVE_DIR, scheme="NVFP4", ignore=ignore, converter=CompressedTensorsDequantizer( MODEL_ID, quant_config_key="text_config.quantization_config", ignore=ignore, ), max_workers=2, device="cuda:0",)bash
vllm serve RedHatAI/Kimi-K2.6-NVFP4 \ --trust-remote-code \ --mm-encoder-tp-mode data \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2 \ --enable-auto-tool-choicepython
from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="RedHatAI/Kimi-K2.6-NVFP4", messages=[{"role": "user", "content": "Explain how transformers use attention."}],) print(resp.choices[0].message.content)We evaluated this model with lm-evaluation-harness, lighteval, BFCL v4, and SWE-Bench Lite served through a vLLM (0.22.1) OpenAI-compatible endpoint.
| Category | Benchmark | Score |
|---|---|---|
| Reasoning and instruction following | AIME25 (pass@1, avg@8) | 96.25% |
| Reasoning and instruction following | GPQA Diamond (pass@1, avg@3) | 91.08% |
| Reasoning and instruction following | MATH-500 (pass@1, avg@3) | 93.13% |
| Reasoning and instruction following | MMLU-Pro Chat (custom-extract, avg@3) | 86.75% |
| Reasoning and instruction following | GSM8K Platinum CoT (strict-match, avg@3) | 92.50% |
| Reasoning and instruction following |
BFCL rows report category accuracy. SWE-Bench follows the official harness score style. For run transparency: 7 of 23 tasks were resolved, and 21 instances produced non-empty graded patches.
| Benchmark | Base model (moonshotai/Kimi-K2.6) | This model |
|---|---|---|
| GSM8K Platinum accuracy | 94.29% | 93.96% |
| Recovery | - | 99.6% |
Representative commands used to produce and aggregate these runs:
bash
lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Kimi-K2.6-NVFP4,max_length=40960,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_gsm8k_platinum.json \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=1.5,repetition_penalty=1.0,seed=1234"yaml
model_parameters: provider: "hosted_vllm" model_name: "hosted_vllm/RedHatAI/Kimi-K2.6-NVFP4" base_url: "http://127.0.0.1:8000/v1" api_key: "EMPTY" timeout: 3600 max_model_length: 40960 concurrent_requests: 8 generation_parameters: temperature: 1.0 max_new_tokens: 65536 top_p: 0.95 seed: 1234 top_k: 20 presence_penalty: 1.5bash
lighteval endpoint litellm litellm_config.yaml \ "aime25@1@8|0,math_500@1@3|0,gpqa:diamond@1@3|0" \ --output-dir results_lighteval \ --save-detailsbash
# BFCL categories: non_live, live, multi_turn, memory, web_search./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 non_live./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 live./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 multi_turn./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 memory./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 web_searchbash
# SWE-Bench Lite dev (full split)SWEBENCH_SUBSET=lite SWEBENCH_SPLIT=dev SWEBENCH_SLICE= \ ./scripts/swebench/run_swebench_lite_local.sh kimi_nvfp4 # Official SWE-bench resolved-rate evaluation/home/shubhra/environments/mini-swe-agent/bin/python -m swebench.harness.run_evaluation \ --dataset_name princeton-nlp/SWE-Bench_Lite \ --split dev \ --predictions_path /home/shubhra/kimik2.6_evals/results/swebench_resolved_eval/kimi_nvfp4_lite_dev_preds_merged.json \ --max_workers 4 \ --run_id kimi_nvfp4_lite_dev_20260701_resolvedMost lm-eval/lighteval tasks were run with 3 seeds and then averaged; AIME25 was run with 8 seeds. BFCL v4 and SWE-Bench Lite numbers come from the aggregated run artifacts listed below.
every_eval_ever/aime25.jsonevery_eval_ever/gpqa_diamond.jsonevery_eval_ever/gsm8k_platinum_cot_llama.jsonevery_eval_ever/ifeval.jsonevery_eval_ever/math_500.jsonevery_eval_ever/mmlu_pro_chat.jsonevery_eval_ever/bfcl_v4.jsonevery_eval_ever/swebench_lite_dev.json| GSM8K Platinum CoT (flexible-extract, avg@3) |
| 96.94% |
| Reasoning and instruction following | IFEval (prompt-level strict, avg@3) | 94.02% |
| Reasoning and instruction following | IFEval (instruction-level strict, avg@3) | 95.96% |
| Agentic function calling (accuracy) | BFCL v4 non_live | 86.48% |
| Agentic function calling (accuracy) | BFCL v4 live | 78.98% |
| Agentic function calling (accuracy) | BFCL v4 multi_turn | 63.75% |
| Agentic function calling (accuracy) | BFCL v4 memory | 64.95% |
| Agentic function calling (accuracy) | BFCL v4 web_search | 42.00% |
| Agentic coding | SWE-Bench Lite (dev) | 30.43% |
from compressed_tensors.entrypoints.convert import CompressedTensorsDequantizerfrom llmcompressor import model_free_ptq MODEL_ID = "moonshotai/Kimi-K2.6"SAVE_DIR = "Kimi-K2.6-NVFP4" ignore = [ "re:.*mlp.gate$", "re:.*lm_head", "re:.*self_attn.*", "re:.*kv_a_proj_with_mqa$", "re:.*q_a_proj$", "re:.*vision_tower.*", "re:.*embed_tokens$", "re:.*norm$", "re:.*mm_projector.*", "re:.*vision.*",] model_free_ptq( model_stub=MODEL_ID, save_directory=SAVE_DIR, scheme="NVFP4", ignore=ignore, converter=CompressedTensorsDequantizer( MODEL_ID, quant_config_key="text_config.quantization_config", ignore=ignore, ), max_workers=2, device="cuda:0",)from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="RedHatAI/Kimi-K2.6-NVFP4", messages=[{"role": "user", "content": "Explain how transformers use attention."}],) print(resp.choices[0].message.content)vllm serve RedHatAI/Kimi-K2.6-NVFP4 \ --trust-remote-code \ --mm-encoder-tp-mode data \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2 \ --enable-auto-tool-choicelm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Kimi-K2.6-NVFP4,max_length=40960,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_gsm8k_platinum.json \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=1.5,repetition_penalty=1.0,seed=1234"lighteval endpoint litellm litellm_config.yaml \ "aime25@1@8|0,math_500@1@3|0,gpqa:diamond@1@3|0" \ --output-dir results_lighteval \ --save-details# BFCL categories: non_live, live, multi_turn, memory, web_search./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 non_live./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 live./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 multi_turn./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 memory./scripts/bfcl/run_bfcl_local.sh kimi_nvfp4 web_search# SWE-Bench Lite dev (full split)SWEBENCH_SUBSET=lite SWEBENCH_SPLIT=dev SWEBENCH_SLICE= \ ./scripts/swebench/run_swebench_lite_local.sh kimi_nvfp4 # Official SWE-bench resolved-rate evaluation/home/shubhra/environments/mini-swe-agent/bin/python -m swebench.harness.run_evaluation \ --dataset_name princeton-nlp/SWE-Bench_Lite \ --split dev \ --predictions_path /home/shubhra/kimik2.6_evals/results/swebench_resolved_eval/kimi_nvfp4_lite_dev_preds_merged.json \ --max_workers 4 \ --run_id kimi_nvfp4_lite_dev_20260701_resolvedmodel_parameters: provider: "hosted_vllm" model_name: "hosted_vllm/RedHatAI/Kimi-K2.6-NVFP4" base_url: "http://127.0.0.1:8000/v1" api_key: "EMPTY" timeout: 3600 max_model_length: 40960 concurrent_requests: 8 generation_parameters: temperature: 1.0 max_new_tokens: 65536 top_p: 0.95 seed: 1234 top_k: 20 presence_penalty: 1.5