GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Xananthium
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Xananthium
Model Tree
allenai/OLMoE-1B-7B-0924
Input Modalities
Output Modalities
Supported Functionality
I converted AllenAI's OLMoE-1B-7B-0924 to INT8 W8A8 using llmcompressor. The attention and expert projections use 8-bit weights and dynamic 8-bit activations. I kept the routing gates, embeddings, normalization and output head in BF16.
I tested the checkpoint in vLLM 0.30.0 on a single RTX 3090, including prompt processing and token-by-token generation. I used eager execution and disabled the optional FlashInfer sampler. The four weight shards total about 6.7 GiB.
I used data-free round-to-nearest quantization, without a calibration dataset. I haven't run formal accuracy or throughput benchmarks.