Serving mode on Blackwell (GB10)
On DGX Spark / GB10 with vLLM, this model serves as weight-only FP4: the 4-bit NVFP4 weights are dequantized for each matmul; activations stay BF16. vLLM 0.20.x has no FP4-activation GEMM kernel for Blackwell (sm_120/121), so the NVFP4 path is weight-only regardless of the input_activations field in config.json — on this stack a W4A4-config and a W4A16-config produce bit-identical output. This is the standard, and currently highest-quality, NVFP4 serving mode on Spark. On an FP4-activation-capable stack (TensorRT-LLM, or a future vLLM with a Blackwell FP4 GEMM) the same weights could run as true W4A4.
A practical consequence: because serving is weight-only, the calibration dataset does not affect the served output — the per-tensor weight scales are determined by the weights alone. The calibration pass below is part of the standard modelopt flow but is effectively output-invariant for this serving mode.
Quick facts
Table | |
|---|
| Base model | TheDrummer/Fallen-Command-A-111B-v1.1 (Cohere Command-A finetune) |
| Architecture | Cohere2ForCausalLM — 64 layers, hidden_size 12288, intermediate 36864, 96 attn heads, 8 KV heads, head_dim 128 |
| Notable arch features | Parallel attention/MLP block, hybrid local/global attention (sliding_window 4096, pattern 4), tied input/output embeddings, RoPE θ=50000, 256K max context |
| Original size | ~207 GB (BF16) |
| Quantized size | ~69 GB (14 shards, see Files tab) |
| Quant format | NVFP4 via nvidia-modelopt 0.43.0, group_size=16 |
| lm_head | Kept BF16 (unquantized), listed in quantization_config.ignore |
| Quantized modules | 448 Linear layers (64 × 7: q/k/v/o + gate/up/down) |
| KV cache | Configurable at serve time (FP8 recommended) |
| Calibration | 256-sample pass (~25.7 min); see note above on output-invariance |
| Conversion date | 2026-05-22 |
The hardware: 2× DGX Spark + 1× RTX 3090
The cluster used to produce this artifact:
Table with columns: Node, GPU, Memory, Role| Node | GPU | Memory | Role |
|---|
| DX10-01 (GB10 Spark) | NVIDIA GB10 (sm_121) | 128 GB UMA | shard0: layers 0–29 + embed_tokens |
| DX10-02 (GB10 Spark) | NVIDIA GB10 (sm_121) | 128 GB UMA | shard1: layers 30–59 |
| eGPU host (Proxmox VM) | NVIDIA RTX 3090 (sm_86) | 24 GB VRAM | shard2: layers 60–63 + final norm + lm_head |
A 30/30/4-layer split keeps each Spark well inside its 128 GB UMA budget while the 3090 handles the tail 4 layers plus the norm and lm_head. Ray RPC carries cross-node hidden states transparently; the Ampere 3090 has no native FP4 hardware but only handles BF16 calibration math, so the architecture mismatch is irrelevant until inference time — the exported NVFP4 file is identical to what an all-Blackwell cluster would produce.
The pipeline is open-source at github.com/KaletoAI/distrib-nvfp4 (Apache 2.0): N-way layer splits via --shard-layers a,b,c, memory-sorted node placement so the smallest-VRAM node gets the smallest shard, and disk-checkpointed phases for resumable runs.
Cohere2-specific handling
Command-A's Cohere2 architecture needed three fixes beyond the standard modelopt NVFP4 flow:
- Tied embeddings. Cohere2 sets
use_embedding_sharing=true — the output projection reuses model.embed_tokens.weight and the checkpoint has no separate lm_head.weight. The head-bearing shard reconstructs lm_head from embed_tokens so it can be exported (BF16) into the merged model.
- Norm epsilon name. Cohere2 names the layernorm epsilon
layer_norm_eps (Llama/Mistral use rms_norm_eps); the per-layer export template reads the correct attribute with a fallback.
- Generation config. Cohere2's
generation_config sets cache_implementation=hybrid, which the 1-layer export template (built with use_cache=False) rejects. It is dropped during per-layer export and the real generation_config is restored in the merged model.
Calibration health-check on the run that produced this artifact — clean, no zero or NaN amax statistics:
- shard0 (layers 0–29 + embed): good=210, zero=0, nan=0
- shard1 (layers 30–59): good=210, zero=0, nan=0
- shard2 (layers 60–63 + norm + lm_head): good=28, zero=0, nan=0
(NVFP4_DEFAULT_CFG inserts 7 weight quantizers per layer.)
After merge, config.json is patched to keep lm_head in quantization_config.ignore, set input_activations.dynamic: true, and inject input_scale=1.0 for every weight quantizer (modelopt 0.43 omits these keys, and vLLM's loader otherwise registers an uninitialized parameter and decodes garbage).
Verification
Loaded and smoke-tested on a single DGX Spark (GB10) with vLLM 0.20.2rc1.dev53 — FlashInferCutlassNvFp4LinearKernel for the NVFP4 GEMM, FlashInfer attention backend:
- Model weights occupy ~62.6 GiB; on a 128 GB UMA Spark at
gpu-memory-utilization 0.90 this leaves a ~43 GiB KV-cache pool (≈175K tokens at 4K context).
- All test generations are coherent and accurate — e.g. "The capital of France is" → "Paris. The area of France is 212,935 square miles…"; "17 + 25" → "42."
The weight-scale layout was also verified directly: every down_proj.weight_scale is [12288, 2304] (2304 = intermediate 36864 / group_size 16), and there are no stray _quantizer / _double_scale keys — the checkpoint loads with stock vLLM.
A formal throughput benchmark has not been run yet.
Usage
vLLM (serve)
Verified on GB10 with vLLM 0.20.2rc1:
vllm serve /path/to/Fallen-Command-111B-NVFP4 \
--served-model-name Fallen-Command-111B-NVFP4 \
--attention-backend flashinfer \
--dtype auto \
--kv-cache-dtype fp8 \
--max-model-len 32768 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.90 \
--enable-chunked-prefill \
--enable-prefix-caching \
--port 9007
vLLM auto-detects the modelopt NVFP4 quantization from config.json — no explicit --quantization flag is needed. --gpu-memory-utilization 0.90 leaves enough KV-cache pool for 32K context at max-num-seqs 4 on a 128 GB Spark; drop to 0.85 if you don't need the longer context.
llama-swap entry
"Fallen-Command-111B-NVFP4":
proxy: "http://127.0.0.1:9007"
ttl: 0
checkEndpoint: "/health"
cmd: >-
/home/<user>/vllm-env/bin/python3 -m vllm.entrypoints.openai.api_server
--model /home/<user>/models/Fallen-Command-111B-NVFP4
--served-model-name Fallen-Command-111B-NVFP4
--attention-backend flashinfer
--dtype auto
--kv-cache-dtype fp8
--max-model-len 32768
--max-num-seqs 4
--gpu-memory-utilization 0.90
--enable-chunked-prefill
--enable-prefix-caching
--port 9007
--host 127.0.0.1
Use the Cohere / Command chat template (it ships in tokenizer_config.json, so apply_chat_template and vLLM's OpenAI server handle it automatically). See TheDrummer's original card for finetune-specific usage notes.
Files in this repository
model-NNNNN-of-00014.safetensors — 14 shards, NVFP4-packed weights + scales (~69 GB total)model.safetensors.index.json — weight map (1859 keys: 448 quantized linears × 3 scale/weight keys + injected input_scale keys + 64 layernorms + embed + lm_head)config.json — Cohere2 config with quantization_config.ignore=["lm_head"] and input_activations.dynamic: truehf_quant_config.json, generation_config.json — auxiliary modelopt + generation configstokenizer.json, tokenizer_config.json, special_tokens_map.json — Command-A tokenizer, untouched from upstream
Acknowledgments
- TheDrummer for the Fallen-Command-A-111B finetune
- Cohere / Cohere Labs for the Command-A base model and the Cohere2 architecture
- NVIDIA for the DGX Spark / GB10 platform, the NVFP4 format, and modelopt
- vLLM project for modelopt NVFP4 inference support
License
This NVFP4 quantization inherits the license of the base model TheDrummer/Fallen-Command-A-111B-v1.1, which is derived from Cohere's Command-A — released under CC-BY-NC 4.0 with Cohere's Acceptable Use Policy. For research, evaluation, and personal non-commercial use only.
Status
Single-author release. Feedback welcome — on the model artifact (vLLM behaviour, sampling, RP quality) and on the pipeline that built it.