amanwalksdownthestreet

DeepSeek-R1-Distill-Llama-70B-MXFP8

Serving

bash
vllm serve <path-or-repo> --quantization compressed-tensors \
  --max-model-len 49152 --gpu-memory-utilization 0.95

Reasoning model: expect <think>…</think> traces per the base model's chat template (--reasoning-parser deepseek_r1 in vLLM separates them).

Evals (lm-evaluation-harness 0.4.11 on vLLM)

Table with columns: Benchmark, DeepSeek paper (BF16), MXFP8 (this repo)
Benchmark	DeepSeek paper (BF16)	MXFP8 (this repo)
MMLU-Pro (5-shot, non-thinking, /v1/completions)	not published	67.50 ±0.41
GPQA-Diamond	65.2	pending*
AIME24	70.0 (multi-sample avg)	pending*

* The first GPQA/AIME24 run was invalidated by eval-infrastructure contention (195/198 GPQA responses returned empty — two sweeps fought over one endpoint); numbers will land after a clean re-run. The MMLU-Pro row is a deliberate non-thinking self-baseline (raw completions, no chat template, no <think> budget) — do not compare it against thinking-mode numbers.

All credit for the model to DeepSeek; this repo is a Blackwell-native repack.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

amanwalksdownthestreet

Model Tree

Base

deepseek-ai/DeepSeek-R1-Distill-Llama-70B

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Serving

bash
vllm serve <path-or-repo> --quantization compressed-tensors \
  --max-model-len 49152 --gpu-memory-utilization 0.95

Reasoning model: expect <think>…</think> traces per the base model's chat template (--reasoning-parser deepseek_r1 in vLLM separates them).

Evals (lm-evaluation-harness 0.4.11 on vLLM)

Table with columns: Benchmark, DeepSeek paper (BF16), MXFP8 (this repo)
Benchmark	DeepSeek paper (BF16)	MXFP8 (this repo)
MMLU-Pro (5-shot, non-thinking, /v1/completions)	not published	67.50 ±0.41
GPQA-Diamond	65.2	pending*
AIME24	70.0 (multi-sample avg)	pending*

All credit for the model to DeepSeek; this repo is a Blackwell-native repack.

DeepSeek-R1-Distill-Llama-70B-MXFP8

README

Serving

Evals (lm-evaluation-harness 0.4.11 on vLLM)

Explore FriendliAI today

README

Serving

Evals (lm-evaluation-harness 0.4.11 on vLLM)