GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
yobo2u
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
yobo2u
Model Tree
deepseek-ai/DeepSeek-V4-Flash-0731
Input Modalities
Output Modalities
Supported Functionality
Base model: deepseek-ai/DeepSeek-V4-Flash-0731
The SM80 architecture used by A100/A800 does not support the native FP8 execution path that the original model targets on newer GPUs. The weights in this repository were converted offline so they can run on SM80 GPUs together with a dedicated SGLang 0.5.16 monkeypatch:
This repository provides converted model weights only. The full runtime environment, pinned versions, monkeypatch source, launch arguments and benchmark scripts are maintained in the GitHub configuration repository.
Do not reuse launch commands written for stock vLLM or newer-architecture GPUs. SM80 deployment requires a pinned SGLang version plus the A100 monkeypatch:
fdebc938f7f4d16fe6b9f55dcd9a767cf0899ea1bash
hf download yobo2u/DeepSeek-V4-Flash-0731-A100 \--local-dir /path/to/models/DeepSeek-V4-Flash-0731-A100
For a model of this size, use low download concurrency and resumable transfers, then verify that all 48 shards referenced by model.safetensors.index.json are present.
The same weights are also published on ModelScope: yobo2u/DeepSeek-V4-Flash-0731-A100
| Item | Value |
|---|---|
| GPU | 8× NVIDIA A800-SXM4-80GB (SM80) |
| Tensor parallel | 8 |
| SGLang | 0.5.16 (commit fdebc938) |
| Dtype | BF16 + MXFP4 experts |
| Serving port | 8082 |
| Context length | 1,048,576 |
--mem-fraction-static | 0.85 |
Measured on the verified configuration above (config C), covering 1K / 4K / 16K / 64K / 128K context × concurrency 1 / 8 / 16 × 2 full repeats = 30 groups, 900 requests, 0 errors and 0 timeouts:
| Metric | Value |
|---|---|
| Single-stream decode (concurrency 1, 1000/TPOT) | ~217 tok/s |
| Single-stream aggregate throughput (concurrency 1) | ~205 tok/s (166–222) |
| Aggregate throughput (concurrency 16) | ~1,232 tok/s (peak 1,334) |
| DSpark accept rate | ~60% (0.40–0.86, increases with context) |
| DSpark accept length | ~4.01 (3.01–5.30) |
| TTFT (concurrency 1) | ~296 ms |
| Peak VRAM | ~51.9 GB/GPU |
Accept rate and accept length rise substantially with context length: about 0.40 / 3.0 at 1K, and about 0.82 / 5.1 at 128K. The table reports means across all 30 groups; per-group values and raw data are in the JSON files under the GitHub
benchmarksdirectory.
A three-way factor decomposition over mem-fraction-static and chunked-prefill-size (30 groups / 900 requests each):
| Config | mem-fraction | chunked-prefill | Mean throughput | Peak VRAM |
|---|---|---|---|---|
| A | 0.85 | 16,384 | 746.2 tok/s | 48.58 GiB |
| B | 0.90 | 32,768 | 759.8 tok/s | 53.80 GiB |
| C (recommended) | 0.85 | 32,768 | 766.4 tok/s | 51.92 GiB |
mem-fraction-static at 0.85. Raising it to 0.90 squeezes the KV cache headroom, dropping the 128K accept rate from 0.80 to 0.69 and costing roughly 11% throughput at long context.chunked-prefill-size from 16,384 to 32,768 helps at short-to-medium context and higher concurrency, and is not the cause of the 128K regression.Performance varies with input length, concurrency, driver, CUDA version, interconnect topology and the specific A100/A800 SKU. The GitHub repository is the source of truth for methodology and results.
--quantization fp8 is part of how the current SGLang monkeypatch path is wired up; it does not mean every weight is stored in native FP8.Released under the MIT License, following the original repository. Users must also comply with the upstream model repository's license terms and applicable laws and regulations.
--chunked-prefill-size| 32,768 |
--max-running-requests | 16 |
| Speculative decoding | DSpark |