GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
deepseek-ai/DeepSeek-V4-Pro
Input Modalities
Output Modalities
Supported Functionality
experts: NVFP4shared_experts, attn: FP8-E4M3 per-blockThe model was quantized from deepseek-ai/DeepSeek-V4-Pro with
experts quantized to MXFP4, and shared_experts and attn quantized to FP8. Using AMD Quark,
we re-quantized both experts and shared_experts to NVFP4 while keeping attn in FP8.
The end-to-end recipe lives in the Quark examples:
examples/torch/language_modeling/llm_ptq/deepseek_v4/nvfp4, and is driven by
run_pipeline.sh. The quantization scope is controlled by EXCLUDE_LAYERS.
bash
export EXCLUDE_LAYERS="*attn* *ffn.gate mtp* *shared_experts*" \export SRC=deepseek-ai/DeepSeek-V4-Proexport OUT=amd/DeepSeek-V4-Pro-NVFP4bash run_pipeline.sh
This model can be deployed efficiently using the vLLM backend. SGLang is also supported.
The model was evaluated on GSM8K benchmarks.
The GSM8K result was obtained using the lm-evaluation-harness framework, based on the Docker image rocm/vllm-dev:nightly_main_20260714.
Install the lm-eval (Version: 0.4.12) in container first.
markdown
pip install lm-eval[api]
markdown
VLLM_ROCM_USE_AITER=1 \VLLM_ROCM_USE_AITER_MOE=1 \vllm serve amd/DeepSeek-V4-Pro-NVFP4 \--host localhost \--port 8001 \--dtype auto \--kv-cache-dtype fp8 \--tensor-parallel-size 8 \--max-num-seqs 512 \--max-num-batched-tokens 8192 \--distributed-executor-backend mp \--trust-remote-code \--gpu-memory-utilization 0.9 \--tokenizer-mode deepseek_v4 \--reasoning-parser deepseek_v4 \--tool-call-parser deepseek_v4 \--enable-auto-tool-choice \--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
markdown
lm_eval \--model local-completions \--model_args model=amd/DeepSeek-V4-Pro-NVFP4,tokenizer=amd/DeepSeek-V4-Pro-NVFP4,base_url=http://127.0.0.1:8001/v1/completions,num_concurrent=32,max_retries=10,max_gen_toks=2048,timeout=60000 \--batch_size auto \--tasks gsm8k \--num_fewshot 8 \--output_path . \2>&1 | tee -a eval.log
This model is a quantized derivative of deepseek-ai/DeepSeek-V4-Pro and is distributed under the same license as the source model: the MIT License. A copy of the upstream LICENSE is included in this repository.
Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. AMD has modified the model weights of the MoE expert layers by quantizing them to NVFP4 with AMD Quark; the modifications are provided under the same MIT License and are not subject to any separate or different license.