GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
symdroid
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
symdroid
Model Tree
mistralai/Mistral-Large-3-675B-Base-2512
Input Modalities
Output Modalities
Supported Functionality
Mistral Large 3 consists of two main architectural components:
The Mistral Large 3 Instruct model offers the following capabilities:
With powerful long-context performance, stable and consistent cross-domain behavior, Mistral Large 3 is perfect for:
And enterprise-grade use cases requiring frontier capabilities.
We recommend deploying Large 3 in a client-server configuration with the following best practices:
We compare Mistral Large 3 to similar sized models.
The model can be used with the following frameworks;
[!Note] We sadly didn't have enough time to add Mistral Large 3 to transformers, but we would be very happy for a community contribution by opening a PR to huggingface/transformers.
We recommend using this model with vLLM.
Make sure to install vllm >= 1.12.0:
text
pip install vllm --upgradeDoing so should automatically install mistral_common >= 1.8.6.
To check:
text
python -c "import mistral_common; print(mistral_common.__version__)"You can also make use of a ready-to-go docker image or on the docker hub.
The Mistral Large 3 Instruct FP8 format can be used on one 8xH200 node. We recommend to use this format if you plan to fine-tuning as it can be more precise than NVFP4 in some situations.
Simple
A simple launch command is:
bash
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \ --max-model-len 262144 --tensor-parallel-size 8 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistralKey parameter notes:
Additional flags:
--max-model-len to preserve memory. By default it is set to 262144 which is quite large but not necessary for most scenarios.--max-num-batched-tokens to balance throughput and latency, higher means higher throughput but higher latency.Accelerated with speculative decoding
For maximum performance we recommend serving the checkpoint with its customized draft model Mistral-Large-3-675B-Instruct-2512-Eagle:
bash
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \ --tensor-parallel-size 8 \ --load-format mistral \ --tokenizer-mode mistral \ --config-format mistral \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --limit-mm-per-prompt '{"image": 10}' \ --speculative_config '{ "model": "mistralai/Mistral-Large-3-675B-Instruct-2512-Eagle", "num_speculative_tokens": 3, "method": "eagle", "max_model_len": "16384" }'For more information on the draft model, please have a look at Mistral-Large-3-675B-Instruct-2512-Eagle.
Here we asumme that the model mistralai/Mistral-Large-3-675B-Instruct-2512 is served and you can ping it to the domain localhost with the port 8000 which is the default for vLLM.
Let's see if Mistral Large 3 knows when to pick a fight !
python
from datetime import datetime, timedelta from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.15MAX_TOK = 262144 client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) models = client.models.list()model = models.data[0].id def load_system_prompt(repo_id: str, filename: str) -> str: file_path = hf_hub_download(repo_id=repo_id, filename=filename) with open(file_path, "r") as file: system_prompt = file.read() today = datetime.today().strftime("%Y-%m-%d") yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d") model_name = repo_id.split("/")[-1] return system_prompt.format(name=model_name, today=today, yesterday=yesterday) SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" messages = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": [ { "type": "text", "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.", }, {"type": "image_url", "image_url": {"url": image_url}}, ], },] response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK,) print(response.choices[0].message.content)Let's solve some equations thanks to our simple Python calculator tool.
python
import jsonfrom openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.15MAX_TOK = 262144 client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) models = client.models.list()model = models.data[0].id def load_system_prompt(repo_id: str, filename: str) -> str: file_path = hf_hub_download(repo_id=repo_id, filename=filename) with open(file_path, "r") as file: system_prompt = file.read() return system_prompt SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") image_url = "https://math-coaching.com/img/fiche/46/expressions-mathematiques.jpg" def my_calculator(expression: str) -> str: return str(eval(expression)) tools = [ { "type": "function", "function": { "name": "my_calculator", "description": "A calculator that can evaluate a mathematical equation and compute its results.", "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "The mathematical expression to evaluate.", }, }, "required": ["expression"], }, }, }, { "type": "function", "function": { "name": "rewrite", "description": "Rewrite a given text for improved clarity", "parameters": { "type": "object", "properties": { "text": { "type": "string", "description": "The input text to rewrite", } }, }, }, },] messages = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": [ { "type": "text", "text": "Thanks to your calculator, compute the results for the equations that involve numbers displayed in the image.", }, { "type": "image_url", "image_url": { "url": image_url, }, }, ], },] response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK, tools=tools, tool_choice="auto",) tool_calls = response.choices[0].message.tool_calls results = []for tool_call in tool_calls: function_name = tool_call.function.name function_args = tool_call.function.arguments if function_name == "my_calculator": result = my_calculator(**json.loads(function_args)) results.append(result) messages.append({"role": "assistant", "tool_calls": tool_calls})for tool_call, result in zip(tool_calls, results): messages.append( { "role": "tool", "tool_call_id": tool_call.id, "name": tool_call.function.name, "content": result, } ) response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK,) print(response.choices[0].message.content)Mistral Large 3 can follow your instructions down to the letter.
python
from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.15MAX_TOK = 262144 client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) models = client.models.list()model = models.data[0].id def load_system_prompt(repo_id: str, filename: str) -> str: file_path = hf_hub_download(repo_id=repo_id, filename=filename) with open(file_path, "r") as file: system_prompt = file.read() return system_prompt SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") messages = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet - start with 'a' and end with 'z'.", },] response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK,) assistant_message = response.choices[0].message.contentprint(assistant_message)This model is licensed under the Apache 2.0 License.
You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \ --max-model-len 262144 --tensor-parallel-size 8 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistralvllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \ --tensor-parallel-size 8 \ --load-format mistral \ --tokenizer-mode mistral \ --config-format mistral \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --limit-mm-per-prompt '{"image": 10}' \ --speculative_config '{ "model": "mistralai/Mistral-Large-3-675B-Instruct-2512-Eagle", "num_speculative_tokens": 3, "method": "eagle", "max_model_len": "16384" }'from datetime import datetime, timedelta from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.15MAX_TOK = 262144 client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) models = client.models.list()model = models.data[0].id def load_system_prompt(repo_id: str, filename: str) -> str: file_path = hf_hub_download(repo_id=repo_id, filename=filename) with open(file_path, "r") as file: system_prompt = file.read() today = datetime.today().strftime("%Y-%m-%d") yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d") model_name = repo_id.split("/")[-1] return system_prompt.format(name=model_name, today=today, yesterday=yesterday) SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" messages = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": [ { "type": "text", "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.", }, {"type": "image_url", "image_url": {"url": image_url}}, ], },] response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK,) print(response.choices[0].message.content)import jsonfrom openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.15MAX_TOK = 262144 client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) models = client.models.list()model = models.data[0].id def load_system_prompt(repo_id: str, filename: str) -> str: file_path = hf_hub_download(repo_id=repo_id, filename=filename) with open(file_path, "r") as file: system_prompt = file.read() return system_prompt SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") image_url = "https://math-coaching.com/img/fiche/46/expressions-mathematiques.jpg" def my_calculator(expression: str) -> str: return str(eval(expression)) tools = [ { "type": "function", "function": { "name": "my_calculator", "description": "A calculator that can evaluate a mathematical equation and compute its results.", "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "The mathematical expression to evaluate.", }, }, "required": ["expression"], }, }, }, { "type": "function", "function": { "name": "rewrite", "description": "Rewrite a given text for improved clarity", "parameters": { "type": "object", "properties": { "text": { "type": "string", "description": "The input text to rewrite", } }, }, }, },] messages = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": [ { "type": "text", "text": "Thanks to your calculator, compute the results for the equations that involve numbers displayed in the image.", }, { "type": "image_url", "image_url": { "url": image_url, }, }, ], },] response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK, tools=tools, tool_choice="auto",) tool_calls = response.choices[0].message.tool_calls results = []for tool_call in tool_calls: function_name = tool_call.function.name function_args = tool_call.function.arguments if function_name == "my_calculator": result = my_calculator(**json.loads(function_args)) results.append(result) messages.append({"role": "assistant", "tool_calls": tool_calls})for tool_call, result in zip(tool_calls, results): messages.append( { "role": "tool", "tool_call_id": tool_call.id, "name": tool_call.function.name, "content": result, } ) response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK,) print(response.choices[0].message.content)from openai import OpenAIfrom huggingface_hub import hf_hub_download # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1" TEMP = 0.15MAX_TOK = 262144 client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) models = client.models.list()model = models.data[0].id def load_system_prompt(repo_id: str, filename: str) -> str: file_path = hf_hub_download(repo_id=repo_id, filename=filename) with open(file_path, "r") as file: system_prompt = file.read() return system_prompt SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") messages = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet - start with 'a' and end with 'z'.", },] response = client.chat.completions.create( model=model, messages=messages, temperature=TEMP, max_tokens=MAX_TOK,) assistant_message = response.choices[0].message.contentprint(assistant_message)