Ministral-3-8B-Reasoning-2512-AWQ-8bit API & Inference Endpoint

Model Details

Quantization Details

Quantization Method: AWQ
Bits: 8
Group Size: 32
Calibration Dataset: 5CD-AI/LLaVA-CoT-o1-Instruct
Quantization Tool: llm-compressor

Memory Usage

Table with columns: Type, Ministral-3-8B-Reasoning-2512, Ministral-3-8B-Reasoning-2512-AWQ-8bit
Type	Ministral-3-8B-Reasoning-2512	Ministral-3-8B-Reasoning-2512-AWQ-8bit
Memory Size	33.2 GB	20.3 GB

Evaluations

Table with columns: Benchmarks, Ministral-3-8B-Reasoning-2512, Ministral-3-8B-Reasoning-2512-AWQ-8bit
Benchmarks	Ministral-3-8B-Reasoning-2512	Ministral-3-8B-Reasoning-2512-AWQ-8bit
Perplexity	1.55808	1.55813

Evaluation Context Length: 16384

Inference

Prerequisite

bash

pip install -U vllm

Basic Usage

bash

vllm serve cyankiwi/Ministral-3-8B-Reasoning-2512-AWQ-8bit --tokenizer_mode mistral --config_format mistral --load_format mistral --enable-auto-tool-choice --tool-call-parser mistral

Additional Information

Changelog

v1.0.0 - Initial quantized release

Authors

Name: Ton Cao
Contacts: ton@cyan.kiwi

Ministral 3 8B Reasoning 2512

A balanced model in the Ministral 3 family, Ministral 3 8B is a powerful, efficient tiny language model with vision capabilities.

This model is the reasoning post-trained version, trained for reasoning tasks, making it ideal for math, coding and stem related use cases.

The Ministral 3 family is designed for edge deployment, capable of running on a wide range of hardware. Ministral 3 8B can even be deployed locally, capable of fitting in 24GB of VRAM in BF16, and less than 12GB of RAM/VRAM when quantized.

Key Features

Ministral 3 8B consists of two main architectural components:

8.4B Language Model
0.4B Vision Encoder

The Ministral 3 8B Reasoning model offers the following capabilities:

Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
System Prompt: Maintains strong adherence and support for system prompts.
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
Reasoning: Excels at complex, multi-step reasoning and dynamic problem-solving.
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
Large Context Window: Supports a 256k context window.

Use Cases

Perfect for balanced performance in local or embedded systems, combining versatility with efficiency.

Chat interfaces in constrained environments
Local daily-driver AI assistant
Image/document description and understanding
Translation and content generation
Specialized agentic use cases
Fine-tuning and specialization
And more...

Bringing advanced AI capabilities to resource-constrained environments.

Ministral 3 Family

Table with columns: Model Name, Type, Precision, Link
Model Name	Type	Precision	Link
Ministral 3 3B Base 2512	Base pre-trained	BF16	Hugging Face
Ministral 3 3B Instruct 2512	Instruct post-trained	FP8	Hugging Face
Ministral 3 3B Reasoning 2512	Reasoning capable	BF16

Other formats available here.

Benchmark Results

We compare Ministral 3 to similar sized models.

Reasoning

Table with columns: Model, AIME25, AIME24, GPQA Diamond, LiveCodeBench
Model	AIME25	AIME24	GPQA Diamond	LiveCodeBench
Ministral 3 14B	0.850	0.898	0.712	0.646
Qwen3-14B (Thinking)	0.737	0.837	0.663	0.593

Instruct

Table with columns: Model, Arena Hard, WildBench, MATH Maj@1, MM MTBench
Model	Arena Hard	WildBench	MATH Maj@1	MM MTBench
Ministral 3 14B	0.551	68.5	0.904	8.49
Qwen3 14B (Non-Thinking)	0.427	65.1	0.870	NOT MULTIMODAL
Gemma3-12B-Instruct	0.436	63.2	0.854	6.70

Base

Table with columns: Model, Multilingual MMLU, MATH CoT 2-Shot, AGIEval 5-shot, MMLU Redux 5-shot, MMLU 5-shot, TriviaQA 5-shot
Model	Multilingual MMLU	MATH CoT 2-Shot	AGIEval 5-shot	MMLU Redux 5-shot	MMLU 5-shot	TriviaQA 5-shot
Ministral 3 14B	0.742	0.676	0.648	0.820	0.794	0.749
Qwen3 14B Base	0.754	0.620	0.661	0.837	0.804

Usage

The model can be used with the following frameworks;

vllm: See here
transformers: See here

vLLM

We recommend using this model with vLLM.

Installation

Make sure to install most recent vllm:

text

uv pip install -U vllm \    --torch-backend=auto \    --extra-index-url https://wheels.vllm.ai/nightly

Doing so should automatically install mistral_common >= 1.8.6.

To check:

text

python -c "import mistral_common; print(mistral_common.__version__)"

You can also make use of a ready-to-go docker image or on the docker hub.

Serve

Due to their size, Ministral-3-3B-Reasoning-2512 and Ministral-3-8B-Reasoning-2512 can run on a single 1xH200 GPU.

A simple launch command is:

bash

vllm serve mistralai/Ministral-3-8B-Reasoning-2512 \  --tokenizer_mode mistral --config_format mistral --load_format mistral \  --enable-auto-tool-choice --tool-call-parser mistral \  --reasoning-parser mistral

Key parameter notes:

enable-auto-tool-choice: Required when enabling tool usage.
tool-call-parser mistral: Required when enabling tool usage.
reasoning-parser mistral: Required when enabling reasoning.

Additional flags:

You can set --max-model-len to preserve memory. By default it is set to 262144 which is quite large but not necessary for most scenarios.
You can set --max-num-batched-tokens to balance throughput and latency, higher means higher throughput but higher latency.

Usage of the model

Here we asumme that the model mistralai/Ministral-3-8B-Reasoning-2512 is served and you can ping it to the domain localhost with the port 8000 which is the default for vLLM.

Let's see if the Ministral 3 model knows when to pick a fight !

python

from typing import Any from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.7TOP_P = 0.95MAX_TOK = 262144client = OpenAI(    api_key=openai_api_key,    base_url=openai_api_base,) models = client.models.list()model = models.data[0].id  def load_system_prompt(repo_id: str, filename: str) -> dict[str, Any]:    file_path = hf_hub_download(repo_id=repo_id, filename=filename)    with open(file_path, "r") as file:        system_prompt = file.read()     index_begin_think = system_prompt.find("[THINK]")    index_end_think = system_prompt.find("[/THINK]")     return {        "role": "system",        "content": [            {"type": "text", "text": system_prompt[:index_begin_think]},            {                "type": "thinking",                "thinking": system_prompt[                    index_begin_think + len("[THINK]") : index_end_think                ],                "closed": True,            },            {                "type": "text",                "text": system_prompt[index_end_think + len("[/THINK]") :],            },        ],    }  SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" messages = [    SYSTEM_PROMPT,    {        "role": "user",        "content": [            {                "type": "text",                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",            },            {"type": "image_url", "image_url": {"url": image_url}},        ],    },]  stream = client.chat.completions.create(    model=model,    messages=messages,    stream=True,    temperature=TEMP,    top_p=TOP_P,    max_tokens=MAX_TOK,) print("client: Start streaming chat completions...:\n")printed_reasoning_content = Falseanswer = [] for chunk in stream:    reasoning_content = None    content = None    # Check the content is reasoning_content or content    if hasattr(chunk.choices[0].delta, "reasoning_content"):        reasoning_content = chunk.choices[0].delta.reasoning_content    if hasattr(chunk.choices[0].delta, "content"):        content = chunk.choices[0].delta.content     if reasoning_content is not None:        if not printed_reasoning_content:            printed_reasoning_content = True            print("Start reasoning:\n", end="", flush=True)        print(reasoning_content, end="", flush=True)    elif content is not None:        # Extract and print the content        if not reasoning_content and printed_reasoning_content:            answer.extend(content)        print(content, end="", flush=True) if answer:    print("\n\n=============\nAnswer\n=============\n")    print("".join(answer))else:    print("\n\n=============\nNo Answer\n=============\n")    print(        "No answer was generated by the model, probably because the maximum number of tokens was reached."    )

Now we'll make it compute some maths !

python

from typing import Any from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.7TOP_P = 0.95MAX_TOK = 262144client = OpenAI(    api_key=openai_api_key,    base_url=openai_api_base,) models = client.models.list()model = models.data[0].id  def load_system_prompt(repo_id: str, filename: str) -> dict[str, Any]:    file_path = hf_hub_download(repo_id=repo_id, filename=filename)    with open(file_path, "r") as file:        system_prompt = file.read()     index_begin_think = system_prompt.find("[THINK]")    index_end_think = system_prompt.find("[/THINK]")     return {        "role": "system",        "content": [            {"type": "text", "text": system_prompt[:index_begin_think]},            {                "type": "thinking",                "thinking": system_prompt[                    index_begin_think + len("[THINK]") : index_end_think                ],                "closed": True,            },            {                "type": "text",                "text": system_prompt[index_end_think + len("[/THINK]") :],            },        ],    }  SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") image_url = "https://i.ytimg.com/vi/5Y3xLHeyKZU/hqdefault.jpg" messages = [    SYSTEM_PROMPT,    {        "role": "user",        "content": [            {                "type": "text",                "text": "Solve the equations. If they contain only numbers, use your calculator, else only think. Answer in the language of the image.",            },            {"type": "image_url", "image_url": {"url": image_url}},        ],    },] stream = client.chat.completions.create(    model=model,    messages=messages,    stream=True,    temperature=TEMP,    top_p=TOP_P,    max_tokens=MAX_TOK,) print("client: Start streaming chat completions...:\n")printed_reasoning_content = Falseanswer = [] for chunk in stream:    reasoning_content = None    content = None    # Check the content is reasoning_content or content    if hasattr(chunk.choices[0].delta, "reasoning_content"):        reasoning_content = chunk.choices[0].delta.reasoning_content    if hasattr(chunk.choices[0].delta, "content"):        content = chunk.choices[0].delta.content     if reasoning_content is not None:        if not printed_reasoning_content:            printed_reasoning_content = True            print("Start reasoning:\n", end="", flush=True)        print(reasoning_content, end="", flush=True)    if content is not None:        # Extract and print the content        if not reasoning_content and printed_reasoning_content:            answer.extend(content)        print(content, end="", flush=True) if answer:    print("\n\n=============\nAnswer\n=============\n")    print("".join(answer))else:    print("\n\n=============\nNo Answer\n=============\n")    print(        "No answer was generated by the model, probably because the maximum number of tokens was reached."    )

Let's do more maths and leave it up to the model to figure out how to achieve a result.

python

from typing import Anyfrom openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.7TOP_P = 0.95MAX_TOK = 262144client = OpenAI(    api_key=openai_api_key,    base_url=openai_api_base,) models = client.models.list()model = models.data[0].id  def load_system_prompt(repo_id: str, filename: str) -> dict[str, Any]:    file_path = hf_hub_download(repo_id=repo_id, filename=filename)    with open(file_path, "r") as file:        system_prompt = file.read()     index_begin_think = system_prompt.find("[THINK]")    index_end_think = system_prompt.find("[/THINK]")     return {        "role": "system",        "content": [            {"type": "text", "text": system_prompt[:index_begin_think]},            {                "type": "thinking",                "thinking": system_prompt[                    index_begin_think + len("[THINK]") : index_end_think                ],                "closed": True,            },            {                "type": "text",                "text": system_prompt[index_end_think + len("[/THINK]") :],            },        ],    }  SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") query = "Use each number in 2,5,6,3 exactly once, along with any combination of +, -, ×, ÷ (and parentheses for grouping), to make the number 24." messages = [    SYSTEM_PROMPT,    {"role": "user", "content": query}]stream = client.chat.completions.create(  model=model,  messages=messages,  stream=True,  temperature=TEMP,  top_p=TOP_P,  max_tokens=MAX_TOK,) print("client: Start streaming chat completions...:\n")printed_reasoning_content = Falseanswer = [] for chunk in stream:    reasoning_content = None    content = None    # Check the content is reasoning_content or content    if hasattr(chunk.choices[0].delta, "reasoning_content"):        reasoning_content = chunk.choices[0].delta.reasoning_content    if hasattr(chunk.choices[0].delta, "content"):        content = chunk.choices[0].delta.content     if reasoning_content is not None:        if not printed_reasoning_content:            printed_reasoning_content = True            print("Start reasoning:\n", end="", flush=True)        print(reasoning_content, end="", flush=True)    if content is not None:        # Extract and print the content        if not reasoning_content and printed_reasoning_content:            answer.extend(content)        print(content, end="", flush=True) if answer:    print("\n\n=============\nAnswer\n=============\n")    print("".join(answer))else:    print("\n\n=============\nNo Answer\n=============\n")    print("No answer was generated by the model, probably because the maximum number of tokens was reached.")

Transformers

You can also use Ministral 3 3B Reasoning 2512 with Transformers ! Make sure to install Transformers from its first v5 release candidate or from "main":

text

pip install transformers==5.0.0rc0

To make the best use of our model with Transformers make sure to have installed mistral-common >= 1.8.6 to use our tokenizer.

bash

pip install mistral-common --upgrade

Then load our tokenizer along with the model and generate:

python

import torchfrom transformers import Mistral3ForConditionalGeneration, MistralCommonBackend model_id = "mistralai/Ministral-3-8B-Reasoning-2512" tokenizer = MistralCommonBackend.from_pretrained(model_id)model = Mistral3ForConditionalGeneration.from_pretrained(    model_id, torch_dtype=torch.bfloat16, device_map="auto") image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" messages = [    {        "role": "user",        "content": [            {                "type": "text",                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",            },            {"type": "image_url", "image_url": {"url": image_url}},        ],    },] tokenized = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True) tokenized["input_ids"] = tokenized["input_ids"].to(device="cuda")tokenized["pixel_values"] = tokenized["pixel_values"].to(dtype=torch.bfloat16, device="cuda")image_sizes = [tokenized["pixel_values"].shape[-2:]] output = model.generate(    **tokenized,    image_sizes=image_sizes,    max_new_tokens=8092,)[0] decoded_output = tokenizer.decode(output[len(tokenized["input_ids"][0]):])print(decoded_output)

License

This model is licensed under the Apache 2.0 License.

You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.

Model Details

Quantization Details

Quantization Method: AWQ
Bits: 8
Group Size: 32
Calibration Dataset: 5CD-AI/LLaVA-CoT-o1-Instruct
Quantization Tool: llm-compressor

Memory Usage

Table with columns: Type, Ministral-3-8B-Reasoning-2512, Ministral-3-8B-Reasoning-2512-AWQ-8bit
Type	Ministral-3-8B-Reasoning-2512	Ministral-3-8B-Reasoning-2512-AWQ-8bit
Memory Size	33.2 GB	20.3 GB

Evaluations

Table with columns: Benchmarks, Ministral-3-8B-Reasoning-2512, Ministral-3-8B-Reasoning-2512-AWQ-8bit
Benchmarks	Ministral-3-8B-Reasoning-2512	Ministral-3-8B-Reasoning-2512-AWQ-8bit
Perplexity	1.55808	1.55813

Evaluation Context Length: 16384

Inference

Prerequisite

bash

pip install -U vllm

Basic Usage

bash

vllm serve cyankiwi/Ministral-3-8B-Reasoning-2512-AWQ-8bit --tokenizer_mode mistral --config_format mistral --load_format mistral --enable-auto-tool-choice --tool-call-parser mistral

Additional Information

Changelog

v1.0.0 - Initial quantized release

Authors

Name: Ton Cao
Contacts: ton@cyan.kiwi

Ministral 3 8B Reasoning 2512

A balanced model in the Ministral 3 family, Ministral 3 8B is a powerful, efficient tiny language model with vision capabilities.

This model is the reasoning post-trained version, trained for reasoning tasks, making it ideal for math, coding and stem related use cases.

Key Features

Ministral 3 8B consists of two main architectural components:

8.4B Language Model
0.4B Vision Encoder

The Ministral 3 8B Reasoning model offers the following capabilities:

Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
System Prompt: Maintains strong adherence and support for system prompts.
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
Reasoning: Excels at complex, multi-step reasoning and dynamic problem-solving.
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
Large Context Window: Supports a 256k context window.

Use Cases

Perfect for balanced performance in local or embedded systems, combining versatility with efficiency.

Chat interfaces in constrained environments
Local daily-driver AI assistant
Image/document description and understanding
Translation and content generation
Specialized agentic use cases
Fine-tuning and specialization
And more...

Bringing advanced AI capabilities to resource-constrained environments.

Ministral 3 Family

Table with columns: Model Name, Type, Precision, Link
Model Name	Type	Precision	Link
Ministral 3 3B Base 2512	Base pre-trained	BF16	Hugging Face
Ministral 3 3B Instruct 2512	Instruct post-trained	FP8	Hugging Face
Ministral 3 3B Reasoning 2512	Reasoning capable	BF16

Other formats available here.

Benchmark Results

We compare Ministral 3 to similar sized models.

Reasoning

Table with columns: Model, AIME25, AIME24, GPQA Diamond, LiveCodeBench
Model	AIME25	AIME24	GPQA Diamond	LiveCodeBench
Ministral 3 14B	0.850	0.898	0.712	0.646
Qwen3-14B (Thinking)	0.737	0.837	0.663	0.593

Instruct

Table with columns: Model, Arena Hard, WildBench, MATH Maj@1, MM MTBench
Model	Arena Hard	WildBench	MATH Maj@1	MM MTBench
Ministral 3 14B	0.551	68.5	0.904	8.49
Qwen3 14B (Non-Thinking)	0.427	65.1	0.870	NOT MULTIMODAL
Gemma3-12B-Instruct	0.436	63.2	0.854	6.70

Base

Table with columns: Model, Multilingual MMLU, MATH CoT 2-Shot, AGIEval 5-shot, MMLU Redux 5-shot, MMLU 5-shot, TriviaQA 5-shot
Model	Multilingual MMLU	MATH CoT 2-Shot	AGIEval 5-shot	MMLU Redux 5-shot	MMLU 5-shot	TriviaQA 5-shot
Ministral 3 14B	0.742	0.676	0.648	0.820	0.794	0.749
Qwen3 14B Base	0.754	0.620	0.661	0.837	0.804

Usage

The model can be used with the following frameworks;

vllm: See here
transformers: See here

vLLM

We recommend using this model with vLLM.

Installation

Make sure to install most recent vllm:

text

uv pip install -U vllm \    --torch-backend=auto \    --extra-index-url https://wheels.vllm.ai/nightly

Doing so should automatically install mistral_common >= 1.8.6.

To check:

text

python -c "import mistral_common; print(mistral_common.__version__)"

You can also make use of a ready-to-go docker image or on the docker hub.

Serve

Due to their size, Ministral-3-3B-Reasoning-2512 and Ministral-3-8B-Reasoning-2512 can run on a single 1xH200 GPU.

A simple launch command is:

bash

vllm serve mistralai/Ministral-3-8B-Reasoning-2512 \  --tokenizer_mode mistral --config_format mistral --load_format mistral \  --enable-auto-tool-choice --tool-call-parser mistral \  --reasoning-parser mistral

Key parameter notes:

enable-auto-tool-choice: Required when enabling tool usage.
tool-call-parser mistral: Required when enabling tool usage.
reasoning-parser mistral: Required when enabling reasoning.

Additional flags:

You can set --max-model-len to preserve memory. By default it is set to 262144 which is quite large but not necessary for most scenarios.
You can set --max-num-batched-tokens to balance throughput and latency, higher means higher throughput but higher latency.

Usage of the model

Here we asumme that the model mistralai/Ministral-3-8B-Reasoning-2512 is served and you can ping it to the domain localhost with the port 8000 which is the default for vLLM.

Let's see if the Ministral 3 model knows when to pick a fight !

python

from typing import Any from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.7TOP_P = 0.95MAX_TOK = 262144client = OpenAI(    api_key=openai_api_key,    base_url=openai_api_base,) models = client.models.list()model = models.data[0].id  def load_system_prompt(repo_id: str, filename: str) -> dict[str, Any]:    file_path = hf_hub_download(repo_id=repo_id, filename=filename)    with open(file_path, "r") as file:        system_prompt = file.read()     index_begin_think = system_prompt.find("[THINK]")    index_end_think = system_prompt.find("[/THINK]")     return {        "role": "system",        "content": [            {"type": "text", "text": system_prompt[:index_begin_think]},            {                "type": "thinking",                "thinking": system_prompt[                    index_begin_think + len("[THINK]") : index_end_think                ],                "closed": True,            },            {                "type": "text",                "text": system_prompt[index_end_think + len("[/THINK]") :],            },        ],    }  SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" messages = [    SYSTEM_PROMPT,    {        "role": "user",        "content": [            {                "type": "text",                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",            },            {"type": "image_url", "image_url": {"url": image_url}},        ],    },]  stream = client.chat.completions.create(    model=model,    messages=messages,    stream=True,    temperature=TEMP,    top_p=TOP_P,    max_tokens=MAX_TOK,) print("client: Start streaming chat completions...:\n")printed_reasoning_content = Falseanswer = [] for chunk in stream:    reasoning_content = None    content = None    # Check the content is reasoning_content or content    if hasattr(chunk.choices[0].delta, "reasoning_content"):        reasoning_content = chunk.choices[0].delta.reasoning_content    if hasattr(chunk.choices[0].delta, "content"):        content = chunk.choices[0].delta.content     if reasoning_content is not None:        if not printed_reasoning_content:            printed_reasoning_content = True            print("Start reasoning:\n", end="", flush=True)        print(reasoning_content, end="", flush=True)    elif content is not None:        # Extract and print the content        if not reasoning_content and printed_reasoning_content:            answer.extend(content)        print(content, end="", flush=True) if answer:    print("\n\n=============\nAnswer\n=============\n")    print("".join(answer))else:    print("\n\n=============\nNo Answer\n=============\n")    print(        "No answer was generated by the model, probably because the maximum number of tokens was reached."    )

Now we'll make it compute some maths !

python

from typing import Any from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.7TOP_P = 0.95MAX_TOK = 262144client = OpenAI(    api_key=openai_api_key,    base_url=openai_api_base,) models = client.models.list()model = models.data[0].id  def load_system_prompt(repo_id: str, filename: str) -> dict[str, Any]:    file_path = hf_hub_download(repo_id=repo_id, filename=filename)    with open(file_path, "r") as file:        system_prompt = file.read()     index_begin_think = system_prompt.find("[THINK]")    index_end_think = system_prompt.find("[/THINK]")     return {        "role": "system",        "content": [            {"type": "text", "text": system_prompt[:index_begin_think]},            {                "type": "thinking",                "thinking": system_prompt[                    index_begin_think + len("[THINK]") : index_end_think                ],                "closed": True,            },            {                "type": "text",                "text": system_prompt[index_end_think + len("[/THINK]") :],            },        ],    }  SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") image_url = "https://i.ytimg.com/vi/5Y3xLHeyKZU/hqdefault.jpg" messages = [    SYSTEM_PROMPT,    {        "role": "user",        "content": [            {                "type": "text",                "text": "Solve the equations. If they contain only numbers, use your calculator, else only think. Answer in the language of the image.",            },            {"type": "image_url", "image_url": {"url": image_url}},        ],    },] stream = client.chat.completions.create(    model=model,    messages=messages,    stream=True,    temperature=TEMP,    top_p=TOP_P,    max_tokens=MAX_TOK,) print("client: Start streaming chat completions...:\n")printed_reasoning_content = Falseanswer = [] for chunk in stream:    reasoning_content = None    content = None    # Check the content is reasoning_content or content    if hasattr(chunk.choices[0].delta, "reasoning_content"):        reasoning_content = chunk.choices[0].delta.reasoning_content    if hasattr(chunk.choices[0].delta, "content"):        content = chunk.choices[0].delta.content     if reasoning_content is not None:        if not printed_reasoning_content:            printed_reasoning_content = True            print("Start reasoning:\n", end="", flush=True)        print(reasoning_content, end="", flush=True)    if content is not None:        # Extract and print the content        if not reasoning_content and printed_reasoning_content:            answer.extend(content)        print(content, end="", flush=True) if answer:    print("\n\n=============\nAnswer\n=============\n")    print("".join(answer))else:    print("\n\n=============\nNo Answer\n=============\n")    print(        "No answer was generated by the model, probably because the maximum number of tokens was reached."    )

Let's do more maths and leave it up to the model to figure out how to achieve a result.

python

from typing import Anyfrom openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.7TOP_P = 0.95MAX_TOK = 262144client = OpenAI(    api_key=openai_api_key,    base_url=openai_api_base,) models = client.models.list()model = models.data[0].id  def load_system_prompt(repo_id: str, filename: str) -> dict[str, Any]:    file_path = hf_hub_download(repo_id=repo_id, filename=filename)    with open(file_path, "r") as file:        system_prompt = file.read()     index_begin_think = system_prompt.find("[THINK]")    index_end_think = system_prompt.find("[/THINK]")     return {        "role": "system",        "content": [            {"type": "text", "text": system_prompt[:index_begin_think]},            {                "type": "thinking",                "thinking": system_prompt[                    index_begin_think + len("[THINK]") : index_end_think                ],                "closed": True,            },            {                "type": "text",                "text": system_prompt[index_end_think + len("[/THINK]") :],            },        ],    }  SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") query = "Use each number in 2,5,6,3 exactly once, along with any combination of +, -, ×, ÷ (and parentheses for grouping), to make the number 24." messages = [    SYSTEM_PROMPT,    {"role": "user", "content": query}]stream = client.chat.completions.create(  model=model,  messages=messages,  stream=True,  temperature=TEMP,  top_p=TOP_P,  max_tokens=MAX_TOK,) print("client: Start streaming chat completions...:\n")printed_reasoning_content = Falseanswer = [] for chunk in stream:    reasoning_content = None    content = None    # Check the content is reasoning_content or content    if hasattr(chunk.choices[0].delta, "reasoning_content"):        reasoning_content = chunk.choices[0].delta.reasoning_content    if hasattr(chunk.choices[0].delta, "content"):        content = chunk.choices[0].delta.content     if reasoning_content is not None:        if not printed_reasoning_content:            printed_reasoning_content = True            print("Start reasoning:\n", end="", flush=True)        print(reasoning_content, end="", flush=True)    if content is not None:        # Extract and print the content        if not reasoning_content and printed_reasoning_content:            answer.extend(content)        print(content, end="", flush=True) if answer:    print("\n\n=============\nAnswer\n=============\n")    print("".join(answer))else:    print("\n\n=============\nNo Answer\n=============\n")    print("No answer was generated by the model, probably because the maximum number of tokens was reached.")

Transformers

You can also use Ministral 3 3B Reasoning 2512 with Transformers ! Make sure to install Transformers from its first v5 release candidate or from "main":

text

pip install transformers==5.0.0rc0

To make the best use of our model with Transformers make sure to have installed mistral-common >= 1.8.6 to use our tokenizer.

bash

pip install mistral-common --upgrade

Then load our tokenizer along with the model and generate:

python

import torchfrom transformers import Mistral3ForConditionalGeneration, MistralCommonBackend model_id = "mistralai/Ministral-3-8B-Reasoning-2512" tokenizer = MistralCommonBackend.from_pretrained(model_id)model = Mistral3ForConditionalGeneration.from_pretrained(    model_id, torch_dtype=torch.bfloat16, device_map="auto") image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" messages = [    {        "role": "user",        "content": [            {                "type": "text",                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",            },            {"type": "image_url", "image_url": {"url": image_url}},        ],    },] tokenized = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True) tokenized["input_ids"] = tokenized["input_ids"].to(device="cuda")tokenized["pixel_values"] = tokenized["pixel_values"].to(dtype=torch.bfloat16, device="cuda")image_sizes = [tokenized["pixel_values"].shape[-2:]] output = model.generate(    **tokenized,    image_sizes=image_sizes,    max_new_tokens=8092,)[0] decoded_output = tokenizer.decode(output[len(tokenized["input_ids"][0]):])print(decoded_output)

License

This model is licensed under the Apache 2.0 License.

You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.

Ministral-3-8B-Reasoning-2512-AWQ-8bit

README

Model Details

Quantization Details

Memory Usage

Evaluations

Inference

Prerequisite

Basic Usage

Additional Information

Changelog

Authors

Ministral 3 8B Reasoning 2512

Key Features

Use Cases

Ministral 3 Family

Benchmark Results

Reasoning

Instruct

Base

Usage

vLLM

Installation

Serve

Usage of the model

Transformers

License

Explore FriendliAI today

README

Model Details

Quantization Details

Memory Usage

Evaluations

Inference

Prerequisite

Basic Usage

Additional Information

Changelog

Authors

Ministral 3 8B Reasoning 2512

Key Features

Use Cases

Ministral 3 Family

Benchmark Results

Reasoning

Instruct

Base

Usage

vLLM

Installation

Serve

Usage of the model

Transformers

License