GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
olka-fi
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
olka-fi
Model Tree
thinkingmachines/Inkling-Small
Input Modalities
Output Modalities
Supported Functionality
| Component | Precision | Why |
|---|---|---|
Routed experts (model.llm.layers.{3..41}.mlp.experts.w13_weight / w2_weight) | MXFP4 (4-bit) | 251B of 276B params — the only place worth the size win |
| First MoE layer (layer 2) experts | BF16 | left unquantized, matching both the official Inkling-Small-NVFP4 and RedHatAI/Inkling-FP8-dynamic |
| Shared experts (2/layer, always active), attention + short-conv, router gates | BF16 | run on every token / sensitive — bit-identical to the source |
| Dense MLP layers 0–1, embeddings, unembed, norms | BF16 | unchanged |
| Vision + audio towers | BF16 | multimodal capability equals the base model |
| MTP (8 next-token-prediction layers) | BF16 | speculative-decoding draft path — kept intact and usable |
78 tensors quantized (39 layers × w13 + w2). Everything else is verified
bit-identical: all 970 remaining tensors (29.4 GB) compared byte-for-byte against
the source and unchanged — 504 attention/short-conv, 160 MTP, 120 router gates,
84 norms/sconv, 80 shared-expert, 10 vision/audio, 6 dense-MLP, 4 embedding/norm,
and the 2 layer-2 expert tensors.
| Metric | Result |
|---|---|
| GSM8K (full 1319-problem test set, chain-of-thought) | 95.3% (1257/1319) |
| Routed-expert reconstruction SQNR vs BF16 source | 18.99 dB (sd 0.04) |
| Cosine similarity, quantized vs original expert weights | 0.9938 |
| Non-expert tensors bit-identical to source | 970 / 970 |
SQNR ≈ 19 dB is the unavoidable 4-bit rounding floor and nothing more — a damaged
checkpoint would sit far below this. Measured over 7 randomly sampled experts
(seed 0) spanning layers 8–40 and both projections: min 18.96, max 19.07,
sd 0.040 dB — the error is uniform, with no outlier expert. w13 runs ~0.1 dB
cleaner than w2 because it has 128 scale groups per row against w2's 64.
Cosine is computed in float64; a float32 reduction over 16.7M elements drifts
enough to inflate it. Reconstruction norm is 0.9939× the source — the slight
shrinkage the MSE clipping search trades for lower total error.
The quantization error in this checkpoint is confined entirely to those 78 expert tensors, since every other tensor is byte-for-byte unchanged.
Scale selection: weight scales use an MSE-optimal clipping search
(memoryless_mse) rather than plain group amax, which measured 0.29–0.42% lower
reconstruction error on every expert tested, for ~3.5 min of extra quantization time.
Built with llm-compressor model_free_ptq
(MXFP4A16, group size 32) streaming shard-to-shard — the 532 GB source is never
materialized. Requires two patches on top of llm-compressor main:
w13_weight / w2_weight) are not matched by the stock
model-free path.state["weight"], but MXFP4 emits
weight_packed, so the suffix must carry into the re-stacked fused tensor.Verified on 2× RTX PRO 6000 Blackwell (SM120, 96 GB) at -tp 2: 152 GiB of weights,
~450–530 tok/s aggregate generation at 8 concurrent requests, with native MTP
speculative decoding enabled.
olkafi/inkling-mxfp4:sm120 keeps the upstream ENTRYPOINT ["vllm","serve"], so it is
a drop-in replacement for vllm/vllm-openai and takes the usual arguments:
bash
docker run --rm --gpus all --ipc=host -p 8000:8000 \-v /path/to/Inkling-Small-MXFP4:/model:ro \olkafi/inkling-mxfp4:sm120 \/model -tp 2 --gpu-memory-utilization 0.93 \--max-model-len 4096 --max-num-seqs 8 \--tokenizer-mode inkling \--speculative-config '{"method":"mtp","num_speculative_tokens":4}' \--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}'
A wrapper with these defaults is installed as inkling-serve if you want it
(--entrypoint inkling-serve, tunable via TP / MTP_K / MAX_MODEL_LEN / …), but
it is opt-in — the image does not second-guess your flags. The Dockerfile and full
patch set are in vllm/ to audit or rebuild.
No patches needed:
bash
vllm serve olka-fi/Inkling-Small-MXFP4 \-tp 2 --gpu-memory-utilization 0.93 \--max-model-len 4096 --max-num-seqs 8 \--tokenizer-mode inkling \--speculative-config '{"method":"mtp","num_speculative_tokens":4}' \--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}'
num_speculative_tokens > 1 requires
vllm#48768 (Inkling multi-depth
MTP — open at the time of writing). Without it, stock vLLM raises
Inkling MTP currently supports exactly one speculative token and only
num_speculative_tokens: 1 works. The PR needs two hand-fixes if your vLLM is newer
than its base d4b456291: multi_stream_utils.py hunk 2, and
_build_draft_attn_metadata, which loses its seq_lens_cpu_upper_bound / step
parameters in the rename (surfaces only at runtime as a NameError).
On SM120 specifically, stock vLLM cannot serve Inkling at all: its CuTe FA4
attention has no paged-KV support (Paged KV not supported on SM 12.0), and vLLM v1
has no unpaged mode. This is a base-model kernel gap unrelated to quantization — the
BF16 model fails identically, and NVFP4 would too.
A ready-to-run image is published as olkafi/inkling-mxfp4:sm120 — see the serving
recipe above. Its Dockerfile and the complete patch set ship in vllm/ in this
repo, so you can audit or rebuild it:
bash
cd vllm && docker build -t inkling-mxfp4:sm120 .
It builds on the official vllm/vllm-openai nightly (pinned by commit) and applies
the contiguous-KV gather and memory-safety clamps from
blockmos/inkling-sparks-gb10,
vllm#48768 for multi-depth MTP, and EAGLE3 aux hidden states — verifying each layer
at build time. See vllm/README.md. Because the workaround
re-gathers KV rather than paging it, long-context decode degrades; prefill is
unaffected. SM90 / SM100 need none of this.
Do not drop
"mode": 0. Without it vLLM resolves to fulltorch.compile, which silently corrupts this model's output rather than erroring.
Inkling ships an 8-layer MTP head, kept BF16 here — so speculative decoding works
out of the box. Measured at num_speculative_tokens=5 over the full GSM8K run
(135,507 draft steps, 677,535 drafted tokens), conditional acceptance is flat
at ~79–83% through depth 5 — it does not decay with depth:
| position | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| unconditional | 78.9% | 62.2% | 49.7% | 40.1% | 33.1% |
| conditional | 78.9% | 78.9% | 79.8% | 80.7% | 82.7% |
52.8% of drafted tokens accepted overall → 3.64 tokens per verify step. Because
acceptance does not decay, deeper drafting keeps paying: the checkpoint carries 8 MTP
layers, so num_speculative_tokens up to 8 is worth testing for single-stream latency
(needs vllm#48768 — see above).
At high concurrency the extra verify width starts to cost more than it returns — k=4
is a reasonable default there, and k=1 works on stock vLLM.
Inherits Apache-2.0 from the base model. This is a derivative (quantized) work of
thinkingmachines/Inkling-Small.
Inkling-Small is a general-purpose multimodal model that accepts text, image and audio inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and multimodal tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers.
Languages: English, with general multilingual capabilities across other languages.
Try Inkling-Small on the Tinker Playground or access via API using the Tinker Cookbook.
Inkling-Small supports local deployment using the following open-source libraries:
API access is also available through third party inference providers.
Multimodal autoregressive transformer
A 42-layer decoder-only transformer with a sparse Mixture-of-Experts (MoE) feed-forward backbone: each token is routed to 6 of 256 experts, plus 2 shared experts active on every token. Attention is a hybrid of local and global layers. The model is natively multimodal — images are encoded via a hierarchical patch encoder, and audio via discrete token encoding — with all modalities projected into a shared hidden space and processed jointly by the decoder.
276B total, 12B active
BF16 and NVFP4
Inkling-Small accepts text, image, and audio inputs:
Inkling-Small generates output as UTF-8 encoded text.
Training data includes a broad variety of content types, including text, images, audio, video.
Training data for the model was drawn from publicly available sources, acquired from third-parties, or synthetically generated or augmented. Publicly available data includes content from the public internet and publicly accessible repositories.
The training data curation process includes cleaning, processing, and modifying datasets. These processing steps, which vary by data type, may include deduplication and filtering to remove junk or other low-quality data, or to advance safety or other objectives.
We conducted safety evaluations ahead of release, spanning both everyday human-AI interaction and dangerous-capability testing. Because Inkling-Small is multimodal, we paid attention to whether safety behavior held consistently across text, audio, and image inputs. We applied mitigations to reduce risks before release.
For everyday interaction, we evaluated sycophancy, harmful manipulation, and psychological-harm patterns like parasocial dependency and validation of delusional reasoning, including through multi-turn, open-ended external red-teaming designed to surface issues that only emerge over longer conversations. We also assessed whether the model refuses genuinely harmful requests without over-refusing benign ones. For CBRN and cyber, we assessed knowledge and procedural uplift through internal evaluations, external testing, and refusal-suppressed variants intended to estimate latent capability with safeguards removed. For loss of control, we evaluated agentic capability, strategic deception, and sabotage potential, benchmarked against public frontier models, and found the model materially below frontier capabilities.
Across all areas, we concluded that Inkling-Small did not present risk of material uplift beyond what's already available in the open-weight ecosystem.
The residual risks identified in our evaluations — specifically, Inkling-Small’s occasional tendency to comply with role-play and indirectly framed prompts concerning harmful topics — are consistent with what you would see from any open-weight model, and are best addressed with defense-in-depth rather than relying on the model's refusals alone. Common downstream moderation tools, such as Llama Guard, are compatible with Inkling-Small and can be layered around the model to catch jailbreak attempts, filter unsafe outputs, and enforce use-case-specific policies. We would encourage treating this kind of input/output classification as a part of your deployment stack, especially for consumer-facing or high-traffic applications where adversarial prompting is more likely.
Inkling-Small may exhibit general limitations common to foundation models, including hallucination (generating plausible but factually incorrect or unsupported content), occasional failures to follow instructions precisely, and degraded performance in long multi-turn conversations. As with other large-scale models trained on web-derived and synthetic data, Inkling-Small may reflect biases present in its training data, including demographic, cultural, or linguistic biases, and may perform unevenly across languages, dialects, or subject domains that were less represented during training.
Inkling-Small's knowledge is limited to information available as of its training cutoff, and it may not reflect events, developments, or changes that occurred afterward.
We recommend that downstream developers and deployers apply appropriate human oversight and review for outputs used in high-stakes or safety-critical contexts, rather than relying on Inkling-Small's outputs without verification.