Model Files [GGUF]
Table with columns: File Name, Quant Type, File Size, File Link, Description| File Name | Quant Type | File Size | File Link | Description |
|---|
| Phai-A1-35B-1M-MTP.MXFP4_MOE.gguf | MXFP4_MOE | 20.3 GB | Link | MXFP4_MOE: Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance. |
| Phai-A1-35B-1M-MTP.Q4_K_M.gguf | Q4_K_M | 21.7 GB | Link | Good quality, default size for most use cases, recommended. |
| Phai-A1-35B-1M-MTP.Q5_K_M.gguf | Q5_K_M | 25.3 GB | Link | High quality, recommended. |
| Phai-A1-35B-1M-MTP.Q5_K_S.gguf | Q5_K_S | 24.6 GB | Link | High quality with slight space savings, recommended. |
| Phai-A1-35B-1M-MTP.Q8_0.gguf | Q8_0 | 37.8 GB | Link | Extremely high quality, generally unneeded but max available standard quant. |
| Phai-A1-35B-1M-MTP.mmproj-bf16.gguf | mmproj-bf16 | 903 MB | Link | Multimodal projection file in BF16 format. Used for vision/image capabilities. |
Table with columns: File Name, File Type, File Size, File Link, Description| File Name | File Type | File Size | File Link | Description |
|---|
| chat_template.jinja | Jinja Template | 7.87 kB | Link | Chat template for formatting conversations and prompts. |
| config.json | JSON | 3.29 kB | Link | Core model architecture and configuration parameters. |
| generation_config.json | JSON | 202 Bytes | Link | Default text generation parameters (e.g., temperature, top_p). |
| model-00001-of-00012.safetensors | Safetensors | 5.89 GB | Link | Model weight shard 1/12. Required to load the model. |
| model-00002-of-00012.safetensors | Safetensors | 6.19 GB | Link | Model weight shard 2/12. Required to load the model. |
| model-00003-of-00012.safetensors | Safetensors | 6.19 GB | Link | Model weight shard 3/12. Required to load the model. |
| model-00004-of-00012.safetensors | Safetensors | 6.12 GB | Link | Model weight shard 4/12. Required to load the model. |
| model-00005-of-00012.safetensors | Safetensors | 6.19 GB | Link | Model weight shard 5/12. Required to load the model. |
| model-00006-of-00012.safetensors | Safetensors | 5.59 GB | Link | Model weight shard 6/12. Required to load the model. |
| model-00007-of-00012.safetensors | Safetensors | 6.18 GB | Link | Model weight shard 7/12. Required to load the model. |
| model-00008-of-00012.safetensors | Safetensors | 6.6 GB | Link | Model weight shard 8/12. Required to load the model. |
| model-00009-of-00012.safetensors | Safetensors | 5.65 GB | Link | Model weight shard 9/12. Required to load the model. |
| model-00010-of-00012.safetensors | Safetensors | 6.19 GB | Link | Model weight shard 10/12. Required to load the model. |
| model-00011-of-00012.safetensors | Safetensors | 5.65 GB | Link | Model weight shard 11/12. Required to load the model. |
| model-00012-of-00012.safetensors | Safetensors | 5.46 GB | Link | Model weight shard 12/12. Required to load the model. |
| model.safetensors.index.json | JSON | 98.4 kB | Link | Index file mapping model weights to their respective shards. |
| preprocessor_config.json | JSON | 390 Bytes | Link | Configuration for preprocessing vision/image inputs. |
| processor_config.json | JSON | 1.19 kB | Link | Configuration for the multimodal processor. |
| tokenizer.json | JSON | 20 MB | Link | Complete tokenizer vocabulary and encoding rules. |
| tokenizer_config.json | JSON | 1.17 kB | Link | Tokenizer-specific configuration parameters. |
Run Inference (llama.cpp)
/path/to/llama.cpp/build/bin/llama-cli \ -m /path/to/models/Phai-A1-35B-1M-MTP.MXFP4_MOE.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -ngl 99 \ -c 8192 \ --cache-type-k q8_0 \ --seed 42 \ -n 4096 \ -st \ -p "Hello, who are you?"
Output (llama.cpp)
Hello, who are you?
[Start thinking]
The user is asking a simple, direct question: "Hello, who are you?"
This is a basic conversational opener. I should respond with a friendly greeting and introduce myself as Kimi, an AI assistant made by Moonshot AI. The response should be brief and natural since this is just a greeting. I shouldn't over-explain or write a long response for such a simple question.
Format considerations: This is casual conversation, so plain text with a warm, concise tone is appropriate. No headers, lists, or special formatting needed. I should keep it short—maybe 2-4 sentences.
[End thinking]
Hello! I'm Kimi, an AI assistant made by Moonshot AI. I can help with a wide range of things—answering questions, writing, coding, brainstorming, analysis, and more.
Is there something I can help you with today?
[ Prompt: 37.9 t/s | Generation: 244.1 t/s ]
Exiting...
[Prompt: 37.9 t/s | Generation: 244.1 t/s] - Generated with full GPU offloading on an NVIDIA RTX PRO 6000 Blackwell Server Edition using MTP (draft-n-max: 2). GGUF: Phai-A1-35B-1M-MTP.MXFP4_MOE.gguf
llama.cpp
LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp
MTP-repackaged and quantized using llama.cpp by — prithivMLmods.