GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Shrinkinator
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Shrinkinator
Model Tree
Input Modalities
Output Modalities
Supported Functionality
vLLM's deepseek_v4 backend uses hand-written CUDA kernels (FlashMLA, fused qnorm-rope-kv-insert, DeepGEMM
lightning indexer) that hardcode the production MLA geometry. To be a genuine drop-in, this clone must
match them exactly, shrinking only elsewhere:
| field | value | why |
|---|---|---|
head_dim | 512 | FlashMLA head_dim_v=512; get_supported_head_sizes()==[512] (nope 448 + rope 64) |
qk_rope_head_dim | 64 | rope dim hardcoded in the fused insert kernel |
index_n_heads | 32 | DeepGEMM fp8_fp4_mqa_logits asserts heads ∈ {32, 64} |
num_attention_heads | 6 | kernel zero-pads q to 64 heads at runtime, so real heads stay cheap |
hidden_size | 384 | all FP8 contraction dims must be % 128 |
| layers / experts | 8 / 8 (top-2) + 1 shared | shrunk freely |
| attention schedule | [swa, swa, CSA, CSA, HCA, CSA, swa, swa] | compress_ratios = [0,0,4,4,128,4,0,0] |
FP8: block-quantized e4m3 (128×128 blocks, ue8m0 scales), quantization_config.quant_method = "fp8".
python
from vllm import LLM, SamplingParamsllm = LLM(model="Shrinkinator/nanowhale-100m-csa-fp8", kv_cache_dtype="fp8", enforce_eager=True, max_model_len=1024)print(llm.generate(["The capital of France is"], SamplingParams(max_tokens=16))[0].outputs[0].text)Needs a CUDA-13.0-matched toolchain for the DeepGEMM/TileLang JIT, and (for this miniaturized hidden size)
the mHC ops routed to their torch fallback. See the shrink project notes.