GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
prithivMLmods
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
prithivMLmods
Model Tree
BAAI/AREX-2
Input Modalities
Output Modalities
Supported Functionality
| Attribute | Value |
|---|---|
| Base model | BAAI/AREX-2 (derived from Qwen/Qwen3.8-27B) |
| Quantized model | prithivMLmods/AREX-2-FP8 |
| Parameters | 27B |
| Context length | 262,144 tokens |
| Modality | Image-Text-to-Text |
| Quantization scheme | FP8_DYNAMIC (W8A8) |
| Weights | FP8, per-channel |
| Activations | FP8, per-token, dynamic |
| Calibration data | Not required |
| Format | compressed-tensors (safetensors) |
| Tooling | LLM Compressor |
| License | Apache 2.0 |
Only Linear layers are quantized. The following modules are excluded and remain in their original precision to preserve output quality: the language-model head, the token embeddings, the entire vision tower, and all linear-attention layers.
recipe.yaml
yaml
default_stage: default_modifiers: QuantizationModifier: targets: [Linear] ignore: ['re:.*lm_head', 're:.*embed_tokens$', 're:.*visual.*', 're:.*model.visual.*', 're:.*linear_attn.*'] scheme: FP8_DYNAMIC bypass_divisibility_checks: false requires_calibration_data: false| Setting | Value |
|---|---|
| targets | Linear |
| ignore | lm_head, embed_tokens, visual, model.visual, linear_attn |
| scheme | FP8_DYNAMIC |
| bypass_divisibility_checks | false |
Install a recent vLLM release with Qwen3.8 support.
bash
pip install -U vllmServe
bash
vllm serve prithivMLmods/AREX-2-FP8 \ --max-model-len 32768 \ --trust-remote-codeRaise --max-model-len toward the native 262,144 tokens as GPU memory allows. Add --tensor-parallel-size N for multi-GPU setups.
Query the OpenAI-compatible endpoint
python
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="prithivMLmods/AREX-2-FP8", messages=[ { "role": "user", "content": "Propose a solution and explain how you would improve it over several rounds.", } ], max_tokens=1024, seed=42,)print(response.choices[0].message.content)Offline inference
python
from vllm import LLM, SamplingParams llm = LLM( model="prithivMLmods/AREX-2-FP8", max_model_len=32768, trust_remote_code=True,) params = SamplingParams(max_tokens=1024, seed=42)messages = [{ "role": "user", "content": "Propose a solution and explain how you would improve it over several rounds.",}] outputs = llm.chat(messages, params)print(outputs[0].outputs[0].text)bash
pip install -U llmcompressor transformers accelerate torchpython
import torchfrom transformers import AutoModelForMultimodalLM, AutoProcessorfrom llmcompressor import oneshot model_id = "BAAI/AREX-2"save_dir = "AREX-2-FP8" model = AutoModelForMultimodalLM.from_pretrained( model_id, dtype=torch.bfloat16, device_map="auto")processor = AutoProcessor.from_pretrained(model_id) oneshot(model=model, recipe="recipe.yaml") model.save_pretrained(save_dir, save_compressed=True)processor.save_pretrained(save_dir)default_stage: default_modifiers: QuantizationModifier: targets: [Linear] ignore: ['re:.*lm_head', 're:.*embed_tokens$', 're:.*visual.*', 're:.*model.visual.*', 're:.*linear_attn.*'] scheme: FP8_DYNAMIC bypass_divisibility_checks: false requires_calibration_data: false| requires_calibration_data | false |