GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
cebeuq
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
cebeuq
Model Tree
deepseek-ai/DeepSeek-V4-Flash-0731
Input Modalities
Output Modalities
Supported Functionality
| Base | deepseek-ai/DeepSeek-V4-Flash-0731 (DeepSeek-V4 MoE, 284B total / 13B active) |
| Architecture | 43 layers, hidden 4096, 256 experts (top-6) + 1 shared, hybrid CSA + HCA attention with Lightning Indexer, mHC hyper-connections (hc_mult=4), inline 3-stage DSpark draft head |
| Context | 1,048,576 native (served here at 262,144) |
| Precision | unchanged — FP4 (E2M1 + ue8m0) routed experts, FP8 e4m3 (128×128 block scales) elsewhere |
| Edit | rank-1 orthogonal projection, λ = 2.5, on 43 layers.*.attn.wo_b + 3 mtp.*.attn.wo_b |
| Untouched | routed + shared experts, wo_a, embed, head, routers, norms, tid2eid, all mHC params |
| Size | ~167 GB, 48 safetensors shards + a 1.54 GB overlay |
| Runtime | vLLM 0.25.2 (ghcr.io/anemll/dspark-vllm-gx10:0.1.1), TP=2 across 2× DGX Spark |
Refusal measured on AdvBench harmful_behaviors with long generations (V4 exhibits a documented delayed refusal — educational framing for a stretch before pivoting), scored on the parsed answer content with the thinking block excluded. Tool-calling measured against a 3-tool schema using the native DSML format and --tool-call-parser deepseek_v4; compliance means the completion parses, contains a call, names a valid tool, and supplies every required parameter.
| Metric | Base 0731 | This model (abliterated) |
|---|---|---|
| Refusal — chat (n=48) | 95.8 % | 0.0 % |
| Refusal — think-high (n=24) | — | 0.0 % |
| Refusal — think-max (n=24) | 95.8 % | 0.0 % |
| Tool-call compliance (all 3 modes) | 1.000 | 1.000 |
| Correct tool selected | — |
Zero refusals observed. At these sample sizes the 95 % Wilson upper bounds are ≈7.4 % (chat) and ≈13.8 % (thinking) — the observed rate is 0, not a proven ceiling.
Serving performance (2× DGX Spark, TP=2 over ConnectX-7 RoCE, single-stream decode, TTFT excluded):
| workload | decode tok/s | draft acceptance | mean accept length |
|---|---|---|---|
| structured (JSON) | 75.5 | 82.5 % | 5.12 / 5 |
| code | 71.7 | ~69 % | — |
| technical prose | 50.6 | 47.9 % | 3.39 |
Aggregate 183 tok/s at 8 concurrent streams. Throughput is strongly content-dependent because speculative decoding lives or dies on how predictable the output is — quote the workload with the number.
Prefill with prefix caching on: 41.9 s cold → 0.37 s warm at 71k context (0.65 s at 156k), which is what makes agentic tool loops usable.
attn.wo_b modules of a live vLLM TP=2 deployment captured last-token sublayer outputs over 256 AdvBench harmful vs 256 Alpaca harmless prompts, in all three reasoning modes (chat, think-high, think-max), giving per-layer difference-of-means directions r_ℓ ∈ ℝ⁴⁰⁹⁶. Cross-rank digests were asserted identical on every prompt, since wo_b is RowParallelLinear and a pre-all-reduce hook would silently capture a half-direction. Stability: median split-half cosine 0.9863, median held-out AUC 0.9976, all 43 layers above 0.90.W ← W − λ·r̂(r̂ᵀW) at λ = 2.5, in float64, directly on the FP8 e4m3 weights: dequantize with the ue8m0 128×128 block scales, project, then re-quantize holding the original block exponents fixed so every element the projection did not move re-encodes to its identical original byte. Overflow is clamped rather than rescaled (6,877 of 94,208 blocks overflowed; 7,821 of 1.54 B elements clamped; max overshoot 1.186×). The 3 DSpark stages inherit the deepest backbone direction — an unedited drafter would propose refusal tokens the edited verifier rejects, collapsing acceptance.Mean relative edit size 0.0587. Written as a tensor overlay with a repointed model.safetensors.index.json, so the 48 original shards are never rewritten. The bake is deterministic — three independent runs produced overlay sha256 20d2559987a19cb5….
Never staged a BF16 upcast; the whole pipeline is numpy + safetensors, torch-free, and runs on CPU in ~70 s.
The model (~167 GB) exceeds a single 128 GB Spark, so serve it across both nodes with tensor-parallel = 2. Config that works on GB10:
bash
vllm serve /path/to/this-model \--served-model-name ds4-abliterated --trust-remote-code \--tensor-parallel-size 2 --distributed-executor-backend mp \--nnodes 2 --node-rank $RANK --master-addr $HEAD --master-port 25000 \--moe-backend flashinfer_b12x --async-scheduling \--max-model-len 262144 --kv-cache-dtype fp8 --block-size 256 \--gpu-memory-utilization 0.85 --max-num-seqs 8 \--max-num-batched-tokens 16384 --max-cudagraph-capture-size 48 \--enable-prefix-caching --enable-chunked-prefill \--speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \--tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \--tool-call-parser deepseek_v4 --enable-auto-tool-choice
GB10 notes, each of which cost real debugging time:
--moe-backend flashinfer_b12x is not optional. auto resolves to DEEPGEMM_MXFP4, the slow path on sm_121. VLLM_USE_B12X_MOE=1 alone does not override it — the CLI flag does.--linear-backend must stay auto; flashinfer_b12x has no kernel for V4's FP8 block-scaled linears and hard-fails at load.NCCL_IB_HCA must name a device that exists. GB10 renames its ConnectX-7 RDMA devices to rocep*/roceP2p* — there is no mlx5_N. NCCL_IB_HCA is a prefix filter: a non-matching value makes NCCL log "no device found" and silently fall back to TCP sockets, costing (12 → 27 tok/s here). Verify with .chat, think-high, think-max. There is no HuggingFace chat template — prompts go through the repo's encoding/encoding_dsv4.py (encode_messages(messages, thinking_mode, reasoning_effort)). Calling tokenizer.apply_chat_template silently falls back to a generic format instead of raising.chat_template_kwargs: {"thinking": true, "reasoning_effort": "max"}. A top-level reasoning_effort field is ignored — vLLM's deepseek_v4 parser gates on chat_template_kwargs["thinking"] AND reasoning_effort != "none". Reasoning is returned in the reasoning field, separate from content.Three things measured here that we could not find published elsewhere:
[B,S,4,4096] and output_hidden_states returns that carrier rather than a hidden state. But the mHC write is rank-1 across streams — hc_post broadcasts one 4096-vector into all four with per-token scalars — so removing a direction at the sublayer output removes it from every stream at once. Measured: split-half cosine 0.9863, held-out AUC 0.9976, in every reasoning mode.1/√4096 of the row's component — measured at 0.253× the e4m3 quantization step — so most elements round back to their original byte. Best achievable direction removal on a real tensor is ~68 %, minimised near λ≈1.5. This is why λ must be calibrated against the baked model rather than inherited from a search run, and plausibly why published work converges on over-projection at λ≈2.5 rather than a "clean" λ=1.Stated plainly, because a card that is silent on these reads as if they were checked.
config.json, model-*-of-00048.safetensors (+ repointed index), model-overlay-00001-of-00001.safetensors, tokenizer.json/tokenizer_config.json, generation_config.json, encoding/ (the official encoding_dsv4.py prompt encoder), inference/ (DeepSeek's reference implementation). See docs/ for the full validation report and the per-tensor edit audit (abliteration_report.json), and scripts/ for the capture / search / bake / validate pipeline used to build this.
MIT (inherited from the base model). Base model: deepseek-ai/DeepSeek-V4-Flash-0731. Abliteration follows the residual-direction method (Arditi et al., Refusal in LLMs is mediated by a single direction, NeurIPS 2024); tooling inspired by p-e-w/heretic and elder-plinius/OBLITERATUS, neither of which supports this architecture. attn.wo_b targeting and the λ≈2.5 operating point independently replicate lovesenko/DeepSeek-V4-Flash-DSpark-Abliterated (their reported mean Frobenius δ 0.059 vs 0.0587 measured here). Serving stack: anemll/dspark-vllm-gx10. Not affiliated with or endorsed by the base-model authors.
| Empty-answer rate | 0.000 | 0.000 |
| DSpark draft acceptance | ~48 % | 48.7 % — drafter at parity |
NCCL_DEBUG=INFO | grep 'NET/IB'num_speculative_tokens must be ≤ 5 (dspark_block_size). The model card's 7 boots unvalidated on vLLM 0.25.x and then misbehaves — the drafter emits exactly block_size tokens per pass and multi-block drafting is unimplemented.--max-num-batched-tokens ≥ 8192 + max_num_seqs × (k−1), else vLLM silently degrades the scheduler budget.VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1 is set (vLLM #47930 otherwise collapses draft acceptance to ~0.5 %). Measured with it on: acceptance 48.7 %, unchanged.nvidia-smi reports memory as N/A on GB10 and torch.cuda.mem_get_info is misleading on unified memory — use free -h.parse_message_from_completion_text raises on malformed output, which makes it a clean compliance signal.usage.completion_tokens, never count SSE deltas. Under speculative decoding vLLM emits one chunk per decode step carrying every accepted token, so counting chunks measures steps/s and under-reports by the acceptance length (~3.4× here).