GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
biosai-hf
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
biosai-hf
Model Tree
thinkingmachines/Inkling
Input Modalities
Output Modalities
Supported Functionality
This model was built by applying AMD Quark MXFP4 quantization to the BF16 Thinking Machines Lab Inkling checkpoint. The quantization targets the MoE routed experts, while attention layers and shared experts are kept in BF16.
The quantization workflow was prepared on an AMD gfx950 system. The inspected container environment was:
Create and activate the Quark environment:
bash
python3 -m venv ~/.venv-quarksource ~/.venv-quark/bin/activate
Install the required packages:
bash
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simplepython -m pip install safetensors transformers accelerate tqdm
The model was quantized with the Quark file-to-file flow. This avoids loading the full BF16 checkpoint into GPU memory at once, which is important for very large MoE checkpoints. Run the quantization script:
bash
python quantize_quark.py \--model_dir /path/to/model \--output_dir /path/to/output \--quant_scheme mxfp4 \--file2file_quantization
The script applies the model-specific exclusion policy automatically in file-to-file mode. The resulting checkpoint stores MXFP4 routed-expert weights and scales while preserving non-routed-expert components in BF16.
This model can be served with TokenSpeed:
bash
tokenspeed serve \--model lightseekorg/Inkling-MXFP4 \--attn-tp-size 4 \--moe-tp-size 4 \--max-model-len 81920 \--max-num-seqs 16 \--max-prefill-tokens 8192 \--chunked-prefill-size 8192 \--gpu-memory-utilization 0.95 \--disable-cuda-graph-padding \--trust-remote-code \--dtype bfloat16 \--disable-kvstore \--kvstore-ratio 0 \--block-size 128 \--host 127.0.0.1 \--port 22015
The following validation results are placeholders and will be updated before public release.
| Benchmark | BF16 Reference | MXFP4 |
|---|---|---|
| BFCL exact calls | 78.3% | 79.1% |
| BFCL all-live macro | 75.4% | 75.3% |
| MMAU | 77.2% | 76.0% |
| GPQA Diamond | 88.1% | 85.4% |
| AIME26 | 96.4% | 96.7% |