tbmod

Inkling-mini-nvfp4

README

License: apache-2.0

Attribution

All weights here are the work of Thinking Machines Lab, sliced without modification from thinkingmachines/Inkling-NVFP4 (revision main) and redistributed under that model's Apache-2.0 licence.

To be precise about what "minimized" means: the retained tensors are the real trained values, bit-for-bit — nothing is randomly initialized and no weight is altered. The reduction is purely deletion, keeping a prefix of the layers and the first 16 experts of each MoE layer. That is what makes the fixture useful for shape and dataflow testing, and it is also why upstream attribution and licensing apply in full.

No claim is made about this artifact's quality, and no endorsement by Thinking Machines Lab is implied.

Routed experts are NVFP4 (packed uint8 weights with F8_E4M3 block scales at block size 16, per-expert F32 scale2, and a BF16 input_amax), except layer 2, whose experts are BF16 upstream and stay BF16 here. Attention, gates, shared experts, embeddings and norms are all BF16, matching the 479-entry exclude_modules list in hf_quant_config.json. Requires Blackwell FP4 tensor cores.

What was reduced

Table with columns: upstream, here
	upstream	here
text layers	66	12
global attention layers	11	2 (at 5, 11)
dense MLP layers	0-1	0-1
routed experts	256	16
experts per token	6	6
shared expert sinks	2	2

19.88 GiB, 370 tensors, 6 shards.

The retained layers are a prefix of the upstream schedule, so layer indices, local_layer_ids and dense_mlp_idx carry over unchanged and the real 5:1 sliding-window / full-attention alternation is preserved rather than hand-written. Hidden size, head counts and head_dim are untouched so kernel shapes match production. The vision and audio towers are complete.

Why the output is gibberish

This is expected and by design, not a bug to report. At 12 of 66 layers and 16 of 256 experts the output distribution is broad and near-uniform. Greedy decoding collapses onto a single special token; temperature 1.0 returns unrelated vocabulary tokens. Logprobs are finite and greedy decoding is deterministic, which is all the fixture needs to be.

This fixture validates plumbing — tensor shapes, the four short-convolution sites per layer, per-layer-type KV head counts and relative-bias widths, gate-plus-sink routing, embedding norm, muP logit scaling, the sampler — and it is a suitable input for a logit-level comparison against a reference implementation. It does not validate correctness on its own, and it should not be evaluated for quality.

Serving

Verified with vLLM 0.26.0 on a single B200:

bash
export VLLM_USE_V2_MODEL_RUNNER=1
export MAX_JOBS=16                # FlashInfer's ninja JIT otherwise fans out
                                  # to the host CPU count and can OOM the box
vllm serve Inkling-mini-nvfp4 \
  --tokenizer-mode inkling \
  --tensor-parallel-size 1 \
  --kernel-config.enable_flashinfer_autotune=False \
  --max-model-len 4096 \
  --max-num-seqs 16

--max-num-seqs matters: vLLM allocates max_num_seqs * max_model_len * 4 B of pinned host memory, and the default can stall cudaHostAlloc for minutes. scipy is an undeclared runtime dependency of the vision tower. At TP=1 the fused Lamport collective logs a ValueError: TP world size must be 2, 4, or 8 traceback at ERROR level; it is a benign fallback to NCCL, not a failure.

Licence

Apache-2.0, inherited from thinkingmachines/Inkling-NVFP4. The weights are Thinking Machines Lab's; this repository only removes layers and experts from them.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

tbmod

Model Tree

Base

thinkingmachines/Inkling-NVFP4

Fine-tuned

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality