GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
lightseekorg
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
lightseekorg
Model Tree
thinkingmachines/Inkling-Small
Input Modalities
Output Modalities
Supported Functionality
0.12.post1+rocm72.torch2.11)
This checkpoint was produced by applying AMD Quark MXFP4 file-to-file quantization to the BF16 Inkling-Small checkpoint. Attention layers, shared experts, router weights, embeddings, normalization layers, multimodal components, MTP weights, and transformer layers 0-2 remain in BF16.
The quantization was performed on an AMD gfx950 system with the following software:
Create and activate a dedicated Quark environment:
bash
python3 -m venv ~/.venv-quarksource ~/.venv-quark/bin/activate
Install the required packages:
bash
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simplepython -m pip install safetensors transformers accelerate tqdm
The included quantize_quark.py uses Quark's file-to-file flow to process safetensor shards without loading the full BF16 checkpoint into GPU memory. It automatically applies the Inkling-Small exclusion policy and quantizes only routed expert weights in layers 3-41.
bash
python quantize_quark.py \--model_dir /path/to/Inkling-Small \--output_dir /path/to/Inkling-Small-MXFP4 \--quant_scheme mxfp4 \--file2file_quantization
This model can be served with TokenSpeed:
bash
tokenspeed serve \--model lightseekorg/Inkling-Small-MXFP4 \--attn-tp-size 1 \--moe-tp-size 1 \--max-model-len 81920 \--max-num-seqs 16 \--max-prefill-tokens 8192 \--chunked-prefill-size 8192 \--gpu-memory-utilization 0.95 \--disable-cuda-graph-padding \--trust-remote-code \--dtype bfloat16 \--disable-kvstore \--kvstore-ratio 0 \--block-size 128 \--speculative-algorithm MTP \--speculative-num-steps 3 \--speculative-eagle-topk 1 \--speculative-num-draft-tokens 4 \--host 127.0.0.1 \--port 22015