GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Israeli-AI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Israeli-AI
Model Tree
Base
deepseek-ai/DeepSeek-V4-Flash
Quantized
this model
Input Modalities
Text
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
markdown
deepseek-ai/DeepSeek-V4-Flash (284B MoE, 256 routed experts)└─ BF16 upcast: RedHatAI/DeepSeek-V4-Flash-BF16└─ REAP prune K160: keep 160/256 experts per layer (Cerebras REAP keep-maps),router/hash tables remapped → bit-identical to 0xSero/DeepSeek-V4-Flash-180Bon the hash-routed layers (0-2)└─ W4A16 g128: data-free RTN, routed experts int4; attention kept BF16└─ THIS MODEL
tid2eid tables, so routing on those layers is identical to the reference 180B checkpoint.Production-validated: this checkpoint runs as the live primary brain of a tool-calling agent (reasoning + native tool calls + multi-turn), not just offline benchmarks.
Objective ground-truth battery + stability + retrieval:
| Category | Score |
|---|---|
| Factual / Math / Reasoning / Code(exec) / Instruction | 9/10 · 9/10 · 7/8 · 4/5 · 8/8 |
| Overall capability (our battery, absolute) | 37/41 = 90% |
| Needle-in-haystack retrieval (to 3.5K tokens) | 4/4 pass — long-context retrieval survives the prune |
| Tool-call-shaped JSON extraction | 14/15 (93%) |
Native tool-calling (--tool-call-parser deepseek_v4) | multi-tool select 3/3, reliability 8/8, multi-turn round-trip ✓ |
| Stability (180 terse-prompt runs) | ~2–3% "derailment" tail (rare essay/CJK on terse prompts) |
The 90% is the model's absolute score on our objective battery, not a retention figure vs the unpruned base (which we did not run). Treat it as "scores 90% on these tasks," not "90% of the original."
Tool-calling works out of the box with --enable-auto-tool-choice --tool-call-parser deepseek_v4 (DSML format): correct tool selection, valid JSON args, multi-turn tool-result round-trips, and no spurious calls on plain questions. Tool prompts show lower derailment than terse ones.
Recommendations: run with thinking enabled (suppresses the derailment tail to ~0; costs more output tokens, not lower tok/s). For agent/tool use, wrap tool-call steps in a retry/validate loop — a small per-step tail compounds over long chains. This is a pruned derivative: evaluate on your own tasks before production.
Two paths depending on your GPU generation.
The weights are standard deepseek_v4 + compressed-tensors W4A16. Serve with an upstream/1Cat vLLM build that supports DeepSeek-V4, no patches:
bash
vllm serve DeepSeek-V4-Flash-W4A16 \--tensor-parallel-size <N> \--reasoning-parser deepseek_v4 \--enable-auto-tool-choice --tool-call-parser deepseek_v4
Stock vLLM's deepseek_v4 path is FP8/FP4 + FlashMLA-sparse + CUTLASS-DSL — none of which exist on pre-Hopper GPUs. The port engages automatically for device_capability.major < 9 (Volta and Ampere) and adds a pure-Triton path:
Tested on 4× V100 (sm_70). Ampere (A100 sm_80, RTX 3090 sm_86) uses the identical arch-independent Triton path and should work, but was not verified here — validate before relying on it.
fp8e4nv)Install:
bash
# 1) a 1Cat-vLLM venv that already registers DeepseekV4ForCausalLM (with SM70 quant support)# 2) apply the port:VLLM=/path/to/venv/lib/python3.12/site-packages/vllm bash sm70-port/apply-port.sh
Serve (4× V100 32 GB — see serve-dsv4-180b.sh). Non-obvious but required:
bash
export CUDA_DEVICE_ORDER=PCI_BUS_IDexport CUDA_VISIBLE_DEVICES=<your 4 V100 indices>export NCCL_P2P_DISABLE=1 # NVLink P2P OOMs at these weight sizesexport VLLM_WORKER_MULTIPROC_METHOD=spawnexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueexport VLLM_SM70_QUANT_BACKEND=turbomind # route W4A16 to turbomind, not marlinexport VLLM_SM70_COMPRESSED_TENSORS_TURBOMIND=1vllm serve DeepSeek-V4-Flash-W4A16 \--tensor-parallel-size 4 --disable-custom-all-reduce \--gpu-memory-utilization 0.93 \--kv-cache-memory-bytes 1395864371 \ # MUST pin: weights ~26.5GiB/GPU leave little room--max-model-len 4096 --max-num-seqs 2 --max-num-batched-tokens 2048 \--kv-cache-dtype fp8_e4m3 \--reasoning-parser deepseek_v4 \--enable-auto-tool-choice --tool-call-parser deepseek_v4 \--compilation-config '{"cudagraph_mode":"full_and_piecewise","cudagraph_capture_sizes":[1,2]}'
Gotchas (learned the hard way):
--kv-cache-memory-bytes. Weights are ~26.5 GiB on a 32 GiB card; without pinning, either the KV check fails or Triton kernel launches OOM. Adjust the value to leave ~2 GiB free per GPU.--enforce-eager is ~2.5× slower — keep CUDA graphs on.pipeline/ contains the end-to-end recipe:
prune_dsv4_reap.py — REAP prune of the BF16 base using reap_plan_k160.json keep-mapsquant-dsv4-180b.py — streaming RTN W4A16 g128 pack-quantize (routed experts only)quality_eval.py, quality_eval2.py — the objective eval harness aboveMIT for the recipe/tooling; base weights follow the DeepSeek-V4-Flash license (review before use).
deepseek-ai/DeepSeek-V4-Flash; BF16 upcast: RedHatAI/DeepSeek-V4-Flash-BF160xSero/DeepSeek-V4-Flash-180BThis is an experimental derivative, not the full DeepSeek-V4-Flash. Validate before trusting it in production.