GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
deepseek-ai/DeepSeek-V4.1-Flash
Input Modalities
Output Modalities
Supported Functionality
experts, shared_experts, self_attn, and engram.embed are quantized to PTPC-FP8. The vision part is not quantized.Quantized from deepseek-ai/DeepSeek-V4.1-Flash
with AMD Quark. The source MXFP4
and MXFP8 weights are converted to PTPC-FP8: experts,
shared_experts, self_attn, and engram.embed are quantized with PTPC-FP8.
The vision tower is kept in its original precision.
bash
MODEL_DIR="deepseek-ai/DeepSeek-V4.1-Flash"OUTPUT_DIR="amd/DeepSeek-V4.1-Flash-Quark-PTPC-FP8" HIP_VISIBLE_DEVICES=0 \QUARK_MXFP4_IMPL=triton \python3 Quark/examples/torch/language_modeling/llm_ptq/quantize_quark.py \ --model_dir "${MODEL_DIR}" \ --output_dir "${OUTPUT_DIR}" \ --quant_scheme ptpc_fp8 \ --layer_quant_scheme "*.engram.embed" ptpc_fp8 \ --quant_flow file2file \ --model_export hf_format \ --keep_excluded_layers_as_original_model_state \ --exclude_layers \ "embed" \ "*head*" \ "*ffn.gate*" \ "*attn.compressor*" \ "*attn.indexer.weights_proj*" \ "*attn.indexer.wk*" \ "*hc_*" \ "*vision*" \ "*aligner*"This model can be deployed efficiently using the vLLM backend based on the Docker image vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657.
Required vLLM patches: #57760, #57071.
The model was evaluated on GSM8K (8-shot) using the vLLM framework.
| Benchmark | deepseek-ai/DeepSeek-V4.1-Flash | amd/DeepSeek-V4.1-Flash-Quark-PTPC-FP8 (this model) | Recovery |
|---|---|---|---|
| GSM8K (8-shot, flexible-extract) | 92.95 | 92.95 | 100.0% |
GSM8K is measured with lm-eval. The base and quantized models are run with the same framework, settings, and host.
The evaluation runs against a vLLM server, using the Docker image
vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657.
bash
export HIP_VISIBLE_DEVICES=0,1,2,3export VLLM_ROCM_USE_AITER=1export VLLM_ROCM_USE_AITER_MOE=1export VLLM_ROCM_USE_AITER_MHA=1export SAFETENSORS_FAST_GPU=1export VLLM_MOE_SKIP_PADDING=0vllm serve amd/DeepSeek-V4.1-Flash-Quark-PTPC-FP8 \ --tensor-parallel-size 4 \ --trust-remote-code \ --moe-backend aiter \ --max-model-len 8192 \ --gpu-memory-utilization 0.90 \ --port 8008bash
lm_eval \ --model local-completions \ --tasks gsm8k \ --model_args "model=amd/DeepSeek-V4.1-Flash-Quark-PTPC-FP8,base_url=http://127.0.0.1:8008/v1/completions,num_concurrent=32,max_retries=5,timeout=1800,tokenized_requests=False,tokenizer_backend=None,max_length=8192" \ --gen_kwargs "max_gen_toks=4096" \ --num_fewshot 8 \ --log_samples \ --output_path /tmp/gsm8k-dsv41-ptpc-fp8This model is a quantized derivative of deepseek-ai/DeepSeek-V4.1-Flash and is distributed under the same license as the source model: the MIT License. A copy of the upstream LICENSE is included in this repository.
Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. AMD has modified selected language-model weights by quantizing them to PTPC-FP8 with AMD Quark; the modifications are provided under the same MIT License and are not subject to any separate or different license.
MODEL_DIR="deepseek-ai/DeepSeek-V4.1-Flash"OUTPUT_DIR="amd/DeepSeek-V4.1-Flash-Quark-PTPC-FP8" HIP_VISIBLE_DEVICES=0 \QUARK_MXFP4_IMPL=triton \python3 Quark/examples/torch/language_modeling/llm_ptq/quantize_quark.py \ --model_dir "${MODEL_DIR}" \ --output_dir "${OUTPUT_DIR}" \ --quant_scheme ptpc_fp8 \ --layer_quant_scheme "*.engram.embed" ptpc_fp8 \ --quant_flow file2file \ --model_export hf_format \ --keep_excluded_layers_as_original_model_state \ --exclude_layers \ "embed" \ "*head*" \ "*ffn.gate*" \ "*attn.compressor*" \ "*attn.indexer.weights_proj*" \ "*attn.indexer.wk*" \ "*hc_*" \ "*vision*" \ "*aligner*"export HIP_VISIBLE_DEVICES=0,1,2,3export VLLM_ROCM_USE_AITER=1export VLLM_ROCM_USE_AITER_MOE=1export VLLM_ROCM_USE_AITER_MHA=1export SAFETENSORS_FAST_GPU=1export VLLM_MOE_SKIP_PADDING=0vllm serve amd/DeepSeek-V4.1-Flash-Quark-PTPC-FP8 \ --tensor-parallel-size 4 \ --trust-remote-code \ --moe-backend aiter \ --max-model-len 8192 \ --gpu-memory-utilization 0.90 \ --port 8008lm_eval \ --model local-completions \ --tasks gsm8k \ --model_args "model=amd/DeepSeek-V4.1-Flash-Quark-PTPC-FP8,base_url=http://127.0.0.1:8008/v1/completions,num_concurrent=32,max_retries=5,timeout=1800,tokenized_requests=False,tokenizer_backend=None,max_length=8192" \ --gen_kwargs "max_gen_toks=4096" \ --num_fewshot 8 \ --log_samples \ --output_path /tmp/gsm8k-dsv41-ptpc-fp8