GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
lightseekorg
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
lightseekorg
Model Tree
thinkingmachines/Inkling-Small
Input Modalities
Output Modalities
Supported Functionality
0.12.post1+rocm72.torch2.11)
This checkpoint was produced by applying AMD Quark MXFP4 file-to-file quantization to the BF16 Inkling-Small checkpoint. Attention layers, shared experts, router weights, embeddings, normalization layers, multimodal components, MTP weights, and transformer layers 0-2 remain in BF16.
The quantization was performed on an AMD gfx950 system with the following software:
Create and activate a dedicated Quark environment:
bash
python3 -m venv ~/.venv-quarksource ~/.venv-quark/bin/activateInstall the required packages:
bash
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simplepython -m pip install safetensors transformers accelerate tqdmThe included quantize_quark.py uses Quark's file-to-file flow to process safetensor shards without loading the full BF16 checkpoint into GPU memory. It automatically applies the Inkling-Small exclusion policy and quantizes only routed expert weights in layers 3-41.
bash
python quantize_quark.py \ --model_dir /path/to/Inkling-Small \ --output_dir /path/to/Inkling-Small-MXFP4 \ --quant_scheme mxfp4 \ --file2file_quantizationThis model can be served with TokenSpeed:
bash
tokenspeed serve \ --model lightseekorg/Inkling-Small-MXFP4 \ --attn-tp-size 1 \ --moe-tp-size 1 \ --max-model-len 81920 \ --max-num-seqs 16 \ --max-prefill-tokens 8192 \ --chunked-prefill-size 8192 \ --gpu-memory-utilization 0.95 \ --disable-cuda-graph-padding \ --trust-remote-code \ --dtype bfloat16 \ --disable-kvstore \ --kvstore-ratio 0 \ --block-size 128 \ --speculative-algorithm MTP \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --host 127.0.0.1 \ --port 22015python3 -m venv ~/.venv-quarksource ~/.venv-quark/bin/activatepython -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simplepython -m pip install safetensors transformers accelerate tqdmpython quantize_quark.py \ --model_dir /path/to/Inkling-Small \ --output_dir /path/to/Inkling-Small-MXFP4 \ --quant_scheme mxfp4 \ --file2file_quantizationtokenspeed serve \ --model lightseekorg/Inkling-Small-MXFP4 \ --attn-tp-size 1 \ --moe-tp-size 1 \ --max-model-len 81920 \ --max-num-seqs 16 \ --max-prefill-tokens 8192 \ --chunked-prefill-size 8192 \ --gpu-memory-utilization 0.95 \ --disable-cuda-graph-padding \ --trust-remote-code \ --dtype bfloat16 \ --disable-kvstore \ --kvstore-ratio 0 \ --block-size 128 \ --speculative-algorithm MTP \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --host 127.0.0.1 \ --port 22015