GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
thinkingmachines/Inkling
Input Modalities
Output Modalities
Supported Functionality
This model is a quantized version of thinkingmachines/Inkling, a 975B total / 41B active parameter multimodal Mixture-of-Experts model that accepts text, image, and audio inputs and generates text outputs.
This model was obtained by quantizing the weights and activations of thinkingmachines/Inkling to FP8 data type using block-wise quantization, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Weights are quantized statically using 128x128 block-wise FP8 scaling, and activations are quantized dynamically at inference time using per-group scaling (group size 128). Only the weights and activations of the linear (attention and MoE expert) layers within the language backbone are quantized using LLM Compressor. The vision encoder, audio encoder, token embedding and unembedding layers, normalization layers, biases, MoE routing/gating logic, shared experts, and the model's early dense MLP layers are kept in their original precision.
This model can be deployed using vLLM.
Note: FP8 quantization support for Inkling requires vllm-project/vllm#48876.
markdown
vllm serve RedHatAI/Inkling-FP8-block \--tensor-parallel-size 8 \--max-model-len 131072 \--gpu-memory-utilization 0.90 \--limit-mm-per-prompt '{"image": 4, "audio": 1}'
Tip: For text-only workloads, pass
--limit-mm-per-prompt '{"image": 0, "audio": 0}'to skip the vision/audio encoder memory allocation and free up GPU memory for a longer context window.
python
from openai import OpenAIopenai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1"client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)model = "RedHatAI/Inkling-FP8-block"messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]outputs = client.chat.completions.create(model=model,messages=messages,)generated_text = outputs.choices[0].message.contentprint(generated_text)