GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
swiss-ai/Apertus-70B-Instruct-2509
Input Modalities
Output Modalities
Supported Functionality
Quantized version of swiss-ai/Apertus-70B-2509.
This model was obtained by quantizing the weights and activations of swiss-ai/Apertus-70B-2509 to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks are quantized.
text
vllm serve RedHatAI/Apertus-70B-Instruct-2509-quantized.w4a16python
from openai import OpenAI # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) model = "RedHatAI/Apertus-70B-Instruct-2509-quantized.w4a16" messages = [ {"role": "user", "content": "Give me a short introduction to large language model."},] outputs = client.chat.completions.create( model=model, messages=messages,) generated_text = outputs.choices[0].message.contentprint(generated_text)This model was created with llm-compressor by running the code snippet below.
python
from llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.transformers import oneshotfrom transformers import AutoModelForCausalLM, AutoTokenizer # Load modelmodel_stub = "swiss-ai/Apertus-70B-Instruct-2509"model_name = model_stub.split("/")[-1] model = AutoModelForCausalLM.from_pretrained(model_stub, dtype="auto") tokenizer = AutoTokenizer.from_pretrained(model_stub) # Configure the quantization algorithm and schemerecipe = QuantizationModifier( ignore=["lm_head"], targets="Linear", scheme="FP8_dynamic",) # Apply quantizationoneshot( model=model, recipe=recipe,) # Save to disk in compressed-tensors formatsave_path = model_name + "-quantized.w4a16"model.save_pretrained(save_path)tokenizer.save_pretrained(save_path)print(f"Model and tokenizer saved to: {save_path}")The model was evaluated on OpenLLM Leaderboard V1, using the following command:
OpenLLM Leaderboard V1:
text
lm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Apertus-70B-Instruct-2509-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=2,gpu_memory_utilization=0.2,enable_chunked_prefill=True \ --tasks openllm \ --write_out \ --batch_size auto \ --output_path output_dir \ --show_configfrom openai import OpenAI # Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,) model = "RedHatAI/Apertus-70B-Instruct-2509-quantized.w4a16" messages = [ {"role": "user", "content": "Give me a short introduction to large language model."},] outputs = client.chat.completions.create( model=model, messages=messages,) generated_text = outputs.choices[0].message.contentprint(generated_text)from llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.transformers import oneshotfrom transformers import AutoModelForCausalLM, AutoTokenizer # Load modelmodel_stub = "swiss-ai/Apertus-70B-Instruct-2509"model_name = model_stub.split("/")[-1] model = AutoModelForCausalLM.from_pretrained(model_stub, dtype="auto") tokenizer = AutoTokenizer.from_pretrained(model_stub) # Configure the quantization algorithm and schemerecipe = QuantizationModifier( ignore=["lm_head"], targets="Linear", scheme="FP8_dynamic",) # Apply quantizationoneshot( model=model, recipe=recipe,) # Save to disk in compressed-tensors formatsave_path = model_name + "-quantized.w4a16"model.save_pretrained(save_path)tokenizer.save_pretrained(save_path)print(f"Model and tokenizer saved to: {save_path}")lm_eval \ --model vllm \ --model_args pretrained="RedHatAI/Apertus-70B-Instruct-2509-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=2,gpu_memory_utilization=0.2,enable_chunked_prefill=True \ --tasks openllm \ --write_out \ --batch_size auto \ --output_path output_dir \ --show_config