DeepSeek-V4-Flash-0731-Latent-Reasoning API & Inference Endpoint

Benchmark

BBH (BIG-Bench Hard), cot_zeroshot, 27 subtasks — aggregate 0.880 ± 0.008

Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint, thinking enabled, 50 items per subtask (1350 items total), metric exact_match / flexible-extract.

Table with columns: Subtask, Score, Subtask, Score
Subtask	Score	Subtask	Score
tracking_shuffled_objects_three_objects	1.00	date_understanding	0.92
tracking_shuffled_objects_five_objects	1.00	sports_understanding	0.88
tracking_shuffled_objects_seven_objects	1.00	logical_deduction_five_objects	0.88
penguins_in_a_table	1.00	web_of_lies	0.86
formal_fallacies	1.00	snarks	0.84
boolean_expressions	1.00	ruin_names	0.84
word_sorting	0.98	movie_recommendation	0.84
temporal_sequences	0.98	salient_translation_error_detection	0.76
object_counting	0.98	geometric_shapes	0.74
navigate	0.98	causal_judgement	0.66
logical_deduction_three_objects	0.98	disambiguation_qa	0.58
reasoning_about_colored_objects	0.96	dyck_languages	0.26
hyperbaton	0.96
multistep_arithmetic_two	0.94
logical_deduction_seven_objects	0.94

Strongest on multi-step state tracking and logical deduction; weakest on dyck_languages (bracket matching), which is the clear outlier.

Read flexible-extract, not strict-match. BBH's strict-match filter regexes for the literal phrase The answer is X, which this model does not emit. Its near-zero strict-match score reflects answer formatting, not reasoning ability. Raw numbers: bench/bbh_cot_zeroshot.json.

Scores are at 50 items/subtask, so per-subtask values carry roughly ±0.05–0.07; the aggregate is the reliable figure.

Quantization

Table

source	`deepseek-ai/DeepSeek-V4-Flash-0731`
scheme	`MIXED_PRECISION` — NVFP4 (group size 16) on routed MoE experts
kept at higher precision	attention, shared experts, LM head, draft block
draft block	3-layer DSpark, preserved from source
weights	48 shards, `bfloat16` non-quantized tensors

Routed expert projections in all 43 layers are converted to NVFP4; everything matched by *.attn.*, *.ffn.shared_experts.*, head, and mtp.* is excluded. NVFP4 needs a Blackwell-class GPU (compute capability 12.0 / sm120) for native kernel support.

Architecture

markdown
layer 35 hidden (4096-d)
                             |
                             v  LayerNorm
        +--------- ReasoningCompressionHead ----------+
        |  Linear 4096 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048  ->  [mu, log_sigma]   |
        |                                             |
        |  stop_head:                                 |
        |     Linear 4096 -> 1024 . SiLU              |
        |     Linear 1024 -> 1                        |  -> end-of-reasoning
        +---------------------------------------------+
                             | mu (1024-d latent)
                             v  LayerNorm
        +-------------- LatentDecoder ----------------+
        |  Linear 1024 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 4096                        |
        +---------------------------------------------+
                             |
                             v  written back into the residual stream
              DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4)

Table with columns: Config, Value
Config	Value
hidden_size	4096
latent_dim	1024
mlp_dim	2048
source_layer / target_layer	35 / 42
activation	SiLU
learned stop head	yes
head + decoder params	35.7M (`float32`)
backbone layers	43

The head is latent_reasoning_head.safetensors (~152 MB), a single flat tensor dict whose submodules are distinguished by key prefix:

markdown
reasoning_head.net.0.weight        [2048, 4096]   reasoning_head.net.0.bias        [2048]
reasoning_head.net.2.weight        [2048, 2048]   reasoning_head.net.2.bias        [2048]
reasoning_head.net.4.weight        [2048, 2048]   reasoning_head.net.4.bias        [2048]
reasoning_head.stop_head.0.weight  [1024, 4096]   reasoning_head.stop_head.0.bias  [1024]
reasoning_head.stop_head.2.weight  [1, 1024]      reasoning_head.stop_head.2.bias  [1]
decoder.net.0.weight               [2048, 1024]   decoder.net.0.bias               [2048]
decoder.net.2.weight               [2048, 2048]   decoder.net.2.bias               [2048]
decoder.net.4.weight               [4096, 2048]   decoder.net.4.bias               [4096]
target_proj.weight                 [1024, 4096]

Geometry is mirrored in the file's safetensors metadata and in latent_reasoning_config.json.

target_proj is a frozen Linear(4096, 1024, bias=False) that defined the head's regression target. It is included for completeness and is not used at inference.

Sample code

Load the head

examples/load_latent_head.py rebuilds the head from the checkpoint's own metadata and runs one latent step — no first-party imports, no dependency on any serving stack.

python
import json
import torch
import torch.nn.functional as F
from torch import nn
from safetensors import safe_open
from safetensors.torch import load_file

CKPT = "latent_reasoning_head.safetensors"


class ReasoningCompressionHead(nn.Module):
    def __init__(self, hidden_size, latent_dim, mlp_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(hidden_size, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, 2 * latent_dim),
        )
        self.stop_head = nn.Sequential(
            nn.Linear(hidden_size, mlp_dim // 2), nn.SiLU(),
            nn.Linear(mlp_dim // 2, 1),
        )

    def forward(self, h):
        mu, log_sigma = self.net(h).chunk(2, dim=-1)
        return mu, log_sigma.clamp(-10.0, 2.0)

    def stop_logit(self, h):
        return self.stop_head(h)


class LatentDecoder(nn.Module):
    def __init__(self, hidden_size, latent_dim, mlp_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, hidden_size),
        )

    def forward(self, z):
        return self.net(z)


with safe_open(CKPT, framework="pt") as f:
    cfg = json.loads(f.metadata()["config"])
hs, ld = cfg["hidden_size"], cfg["latent_dim"]

flat = load_file(CKPT)
mlp_dim = flat["reasoning_head.net.0.weight"].shape[0]
sub = lambda p: {k[len(p):]: v for k, v in flat.items() if k.startswith(p)}

head = ReasoningCompressionHead(hs, ld, mlp_dim)
head.load_state_dict(sub("reasoning_head.")); head.eval()
decoder = LatentDecoder(hs, ld, mlp_dim)
decoder.load_state_dict(sub("decoder.")); decoder.eval()

# One latent step. h_src = layer-35 hidden at the current position, shape (B, 4096).
h_src = torch.randn(2, hs)
h_n = F.layer_norm(h_src, (hs,))
mu, _ = head(h_n)
inject = decoder(F.layer_norm(mu, (ld,)))       # (B, 4096) -> back into the stream
p_stop = head.stop_logit(h_n).sigmoid()         # end reasoning when > threshold

Query a served endpoint

examples/chat_openai_client.py. The one non-obvious requirement is the thinking flag:

python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")

resp = client.chat.completions.create(
    model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
    messages=[{"role": "user", "content": "..."}],
    extra_body={"chat_template_kwargs": {"thinking": True}},   # REQUIRED
    max_tokens=4096,
    temperature=0.6,
)
print(resp.choices[0].message.content)

chat_template_kwargs={"thinking": True} is required. Without it the reasoning phase is not enabled and answer quality drops sharply. All benchmark numbers above were produced with it set.

Serving

The backbone, tokenizer, and DSpark draft block in this repo load with a standard NVFP4-capable inference stack on sm120 hardware. Settings that matter:

Table with columns: Setting, Value, Why
Setting	Value	Why
speculative decoding	DSpark, 4 draft tokens	matches the 3-layer draft block shipped here
KV cache dtype	`fp8`	the model is large; fp8 KV is what makes long context fit
tensor parallel	2	measured on 2x 96 GiB
stop threshold	0.5	`sigmoid(stop_logit) > 0.5` ends the latent phase
min / max latent steps	4 / 256

Two behaviors worth knowing before you judge output quality:

Give the answer real token headroom. The latent reasoning phase and the answer draw from the same output-token budget, so a tight max_tokens can be consumed entirely by reasoning and return an empty or truncated answer.
Warm up before trusting output. The first request or two after a cold start can come back as degenerate repetition, then settle and stay correct. Send a throwaway request after startup; treat a single bad early answer as unwarmed rather than as a broken model.

compression_factor: 6 in the config records how the head was fit. It is not a budget enforced at inference — the learned stop, bounded by the min/max latent steps, is what terminates the reasoning phase.

The serving runtime

The runtime that drives the closed latent loop is released separately — the weights here are complete, but something has to read layer-35 hidden states and write decoded latents back into the residual stream mid-generation. Two pieces:

Table with columns: Repository, What it is
Repository	What it is
`nickmitchko/ds4-reasoning-addon`	The latent-reasoning addon: the loop itself, plus a serve script, pinned requirements and GPU sizing. Start here.
`nickmitchko/vllm-ds4-sm120`	The vLLM fork it runs on (branch `ds4-sm120-preview-dev`). Required — upstream vLLM cannot serve this model.

Upstream vLLM will not work: this model needs the sm120 sparse-MLA path, DeepSeek-V4's hash-MoE routing and the per-layer MoE quant dispatch, none of which are upstream yet.

bash
# 1. the engine (a full build takes a while)
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation

# 2. the addon + its pinned deps
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
    --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11

# 3. serve -- no arguments needed; the head is resolved from this repo
release/serve_ds4_reasoning.sh

The serve script defaults to this model and fetches latent_reasoning_head.safetensors (~152 MiB) from here on its own, so the settings in the table above are already applied.

Limitations

Requires Blackwell-class hardware (sm120) for native NVFP4 kernels.
Driving the latent loop requires runtime support. The weights here are complete, but reading layer-35 hidden states and writing decoded latents back into the residual stream mid-generation is not something a stock transformers forward pass does. Without that, you get the backbone; you do not get latent reasoning. The runtime is released — see The serving runtime — but it is a vLLM fork, not stock transformers.
Evaluation is BBH-only at 50 items/subtask. No multi-task or long-context benchmark suite is reported here.
dyck_languages at 0.26 is a genuine weak spot, not a formatting artifact.
Reasoning happens in latent space, so the surfaced trace is not a faithful token-level record of the computation that produced the answer.

License

MIT, inherited from deepseek-ai/DeepSeek-V4-Flash-0731.

Subtask

Score

Subtask

Score

tracking_shuffled_objects_three_objects

1.00

date_understanding

0.92

tracking_shuffled_objects_five_objects

1.00

sports_understanding

0.88

tracking_shuffled_objects_seven_objects

1.00

logical_deduction_five_objects

0.88

penguins_in_a_table

1.00

web_of_lies

0.86

formal_fallacies

1.00

snarks

0.84

boolean_expressions

1.00

ruin_names

0.84

word_sorting

0.98

movie_recommendation

0.84

temporal_sequences

0.98

salient_translation_error_detection

0.76

object_counting

0.98

geometric_shapes

0.74

navigate

0.98

causal_judgement

0.66

logical_deduction_three_objects

0.98

disambiguation_qa

0.58

reasoning_about_colored_objects

0.96

dyck_languages

0.26

hyperbaton

0.96

multistep_arithmetic_two

0.94

logical_deduction_seven_objects

0.94

source

deepseek-ai/DeepSeek-V4-Flash-0731

scheme

MIXED_PRECISION — NVFP4 (group size 16) on routed MoE experts

kept at higher precision

attention, shared experts, LM head, draft block

draft block

3-layer DSpark, preserved from source

weights

48 shards, bfloat16 non-quantized tensors

markdown

layer 35 hidden (4096-d)
                             |
                             v  LayerNorm
        +--------- ReasoningCompressionHead ----------+
        |  Linear 4096 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048  ->  [mu, log_sigma]   |
        |                                             |
        |  stop_head:                                 |
        |     Linear 4096 -> 1024 . SiLU              |
        |     Linear 1024 -> 1                        |  -> end-of-reasoning
        +---------------------------------------------+
                             | mu (1024-d latent)
                             v  LayerNorm
        +-------------- LatentDecoder ----------------+
        |  Linear 1024 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 4096                        |
        +---------------------------------------------+
                             |
                             v  written back into the residual stream
              DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4)

Config

Value

hidden_size

4096

latent_dim

1024

mlp_dim

2048

source_layer / target_layer

35 / 42

activation

SiLU

learned stop head

yes

head + decoder params

35.7M (float32)

backbone layers

markdown

reasoning_head.net.0.weight        [2048, 4096]   reasoning_head.net.0.bias        [2048]
reasoning_head.net.2.weight        [2048, 2048]   reasoning_head.net.2.bias        [2048]
reasoning_head.net.4.weight        [2048, 2048]   reasoning_head.net.4.bias        [2048]
reasoning_head.stop_head.0.weight  [1024, 4096]   reasoning_head.stop_head.0.bias  [1024]
reasoning_head.stop_head.2.weight  [1, 1024]      reasoning_head.stop_head.2.bias  [1]
decoder.net.0.weight               [2048, 1024]   decoder.net.0.bias               [2048]
decoder.net.2.weight               [2048, 2048]   decoder.net.2.bias               [2048]
decoder.net.4.weight               [4096, 2048]   decoder.net.4.bias               [4096]
target_proj.weight                 [1024, 4096]

python

import json
import torch
import torch.nn.functional as F
from torch import nn
from safetensors import safe_open
from safetensors.torch import load_file

CKPT = "latent_reasoning_head.safetensors"


class ReasoningCompressionHead(nn.Module):
    def __init__(self, hidden_size, latent_dim, mlp_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(hidden_size, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, 2 * latent_dim),
        )
        self.stop_head = nn.Sequential(
            nn.Linear(hidden_size, mlp_dim // 2), nn.SiLU(),
            nn.Linear(mlp_dim // 2, 1),
        )

    def forward(self, h):
        mu, log_sigma = self.net(h).chunk(2, dim=-1)
        return mu, log_sigma.clamp(-10.0, 2.0)

    def stop_logit(self, h):
        return self.stop_head(h)


class LatentDecoder(nn.Module):
    def __init__(self, hidden_size, latent_dim, mlp_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, hidden_size),
        )

    def forward(self, z):
        return self.net(z)


with safe_open(CKPT, framework="pt") as f:
    cfg = json.loads(f.metadata()["config"])
hs, ld = cfg["hidden_size"], cfg["latent_dim"]

flat = load_file(CKPT)
mlp_dim = flat["reasoning_head.net.0.weight"].shape[0]
sub = lambda p: {k[len(p):]: v for k, v in flat.items() if k.startswith(p)}

head = ReasoningCompressionHead(hs, ld, mlp_dim)
head.load_state_dict(sub("reasoning_head.")); head.eval()
decoder = LatentDecoder(hs, ld, mlp_dim)
decoder.load_state_dict(sub("decoder.")); decoder.eval()

# One latent step. h_src = layer-35 hidden at the current position, shape (B, 4096).
h_src = torch.randn(2, hs)
h_n = F.layer_norm(h_src, (hs,))
mu, _ = head(h_n)
inject = decoder(F.layer_norm(mu, (ld,)))       # (B, 4096) -> back into the stream
p_stop = head.stop_logit(h_n).sigmoid()         # end reasoning when > threshold

python

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")

resp = client.chat.completions.create(
    model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
    messages=[{"role": "user", "content": "..."}],
    extra_body={"chat_template_kwargs": {"thinking": True}},   # REQUIRED
    max_tokens=4096,
    temperature=0.6,
)
print(resp.choices[0].message.content)

Setting

Value

Why

speculative decoding

DSpark, 4 draft tokens

matches the 3-layer draft block shipped here

KV cache dtype

fp8

the model is large; fp8 KV is what makes long context fit

tensor parallel

measured on 2x 96 GiB

stop threshold

0.5

sigmoid(stop_logit) > 0.5 ends the latent phase

min / max latent steps

4 / 256

Repository

What it is

nickmitchko/ds4-reasoning-addon

The latent-reasoning addon: the loop itself, plus a serve script, pinned requirements and GPU sizing. Start here.

nickmitchko/vllm-ds4-sm120

The vLLM fork it runs on (branch ds4-sm120-preview-dev). Required — upstream vLLM cannot serve this model.

bash

# 1. the engine (a full build takes a while)
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation

# 2. the addon + its pinned deps
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
    --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11

# 3. serve -- no arguments needed; the head is resolved from this repo
release/serve_ds4_reasoning.sh

DeepSeek-V4-Flash-0731-Latent-Reasoning

README

Benchmark

Quantization

Architecture

Sample code

Load the head

Query a served endpoint

Serving

The serving runtime

Limitations

License

Explore FriendliAI today

README

Benchmark

Quantization

Architecture

Sample code

Load the head

Query a served endpoint

Serving

The serving runtime

Limitations

License