GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Baekpica
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Baekpica
Model Tree
Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-BF16
Input Modalities
Output Modalities
Supported Functionality
I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.
While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.
Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.
This is the predominantly weight-only NVFP4A16 version of
Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-BF16.
A16) on the NVFP4 paths; this is not
the W4A4 NVFP4 preset. The grouped attention WO_A compatibility path is
the explicit mixed-precision exception described below.llm-compressor model-free PTQ,
scheme="NVFP4A16", using data-free RTN. The shorter repository suffix is
NVFP4; the checkpoint is still weight-only W4A16, not W4A4.quant_config=None. The 43 grouped WO_A attention projections use dynamic
W8A8 E4M3 block FP8 (128×128 weight blocks, 128-element activation groups)
because the native CUDA inverse-RoPE output einsum directly consumes that
grouped FP8 layout. Linear gate_proj weights,
including shared-expert projections, remain part of the NVFP4 fused gate/up
quantization path.nvfp4-pack-quantized checkpoints.The parent model has 119,821,633,111 logical main-model parameters, 43 decoder layers, 104 routed experts per layer, top-6 routing, and approximately 13.802B active parameters/token. DSpark/MTP auxiliary weights are not included.
Build provenance is recorded in nvfp4a16_provenance.json; the checkpoint
header/config audit is in checkpoint_audit.json. See the BF16 parent model card
for the direct-saliency REAM and calibration details.
The single-GPU configuration below was verified on NVIDIA GB10 using the
Baekpica/vllm branch
deepseek-v4-v0.26.0,
based on the upstream v0.26.0 tag. The compatibility changes are pinned in
commit 622008c4e:
diff
# Grouped FP8 WO_A uses the DeepGEMM scale attribute after loading.- weight_scale+ weight_scale_inv# The fused CUDA router has no 104-expert instantiation.- if current_platform.is_xpu():+ if current_platform.is_xpu() or gating_output.shape[-1] == 104:
Install the fork with the precompiled vLLM extension, then install the matching FlashInfer packages. The CUDA-specific index is required for the JIT-cache wheel used by the validated CUDA 13.0 environment.
bash
git clone --branch deepseek-v4-v0.26.0 --single-branch \\https://github.com/Baekpica/vllm.gitcd vllmuv venv --python 3.12source .venv/bin/activateVLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=autouv pip install \\flashinfer-python==0.6.14 \\flashinfer-cubin==0.6.14 \\'flashinfer-jit-cache==0.6.14+cu130' \\--extra-index-url https://flashinfer.ai/whl \\--extra-index-url https://flashinfer.ai/whl/cu130
Use an nvcc/header pair that matches the PyTorch CUDA build. The validated
DGX Spark environment used /usr/local/cuda-13.0:
bash
export CUDA_HOME=/usr/local/cuda-13.0export PATH="$CUDA_HOME/bin:$PATH"vllm serve Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-NVFP4 \\--port 8001 \\--optimization-level 3 \\--max-model-len 196608 \\--max-num-seqs 2 \\--max-num-batched-tokens 16384 \\--gpu-memory-utilization 0.92 \\--enable-prefix-caching \\--enable-chunked-prefill \\--kv-cache-dtype fp8 \\--moe-backend auto \\--attention-backend flashinfer \\--tokenizer-mode deepseek_v4 \\--reasoning-parser deepseek_v4 \\--tool-call-parser deepseek_v4 \\--enable-auto-tool-choice \\--quantization compressed-tensors \\--language-model-only
The checkpoint quantization_config declares the packed format and ignore
list. No runtime adapter or model-repository custom code is required.
The unquantized output tensor intentionally retains the original DeepSeek-V4
checkpoint name head.weight; vLLM's native DeepSeek-V4 mapper exposes it as
lm_head.weight at runtime. Tensor paths otherwise use the original native
DeepSeek-V4 layout expected by that mapper; no adapter or custom model file is
needed.
The compatibility branch and 196,608-token launch configuration above were verified to load and serve this checkpoint on DGX Spark / NVIDIA GB10.
Validated on 4× NVIDIA B200 with vLLM 0.22.0, tensor parallel 4, expert
parallel enabled, 8,192-token context, FP8 KV cache, and CUDA graphs enabled.
The server completed weight loading, the 104-expert routing warmup, CUDA graph
capture, /health (HTTP 200), model discovery, and four OpenAI-compatible chat
requests. Raw responses and timings are in vllm_serving_smoke_2048.json.
| Case | Result | Completion |
|---|---|---|
| Korean Python | Format rubric 4/4; semantic review found incorrect edge-case assertions | 490 tokens, stop |
| Reservoir math | Failed; arithmetic error followed by repetition | 2,048 tokens, length |
| 5,810-token needle | Exact retrieval | 55 tokens, stop |
| Strict JSON | Exact output | 29 tokens, stop |
This smoke test demonstrates that the standalone repository serves end to end; it is not evidence of benchmark-level quality. In particular, the Korean code was fluent and structurally complete but should not be treated as a correctness pass because some generated test expectations were inconsistent with the problem definition.
This is RTN weight-only quantization of an expert-merged model. Quantization and expert merging can both affect quality. The 2,048-token serving smoke exposed a basic arithmetic failure/repetition and incorrect coding edge-case assertions, while long-context retrieval and strict JSON succeeded. Korean fluency and surface code structure therefore do not guarantee logical correctness. Run task-specific evaluations and sandbox generated code before production use. The source model's MIT license and usage considerations apply.