Laguna-S-2.1-ModelOpt-NVFP4-W4A16-vllm API & Inference Endpoint

Which variant do you want

W4A16 (this repo) — weights 4-bit, activations 16-bit. Higher activation fidelity, and it serves anywhere NVFP4 weight-only is supported. On GB10 a weight-only NVFP4 export carries a QuantKey ending xNone, which the fused NVFP4 CUTLASS MoE kernel rejects — it falls back to MARLIN.
W4A4 — weights and activations 4-bit on the routed experts, which is what lets the fused NVFP4 MoE kernel run on a DGX Spark.

Pick this repo for fidelity/portability, the W4A4 repo for fused-kernel decode throughput.

Serving (vLLM)

bash
vllm serve JasonW2025/Laguna-S-2.1-ModelOpt-NVFP4-W4A16-vllm \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --no-enable-prefix-caching

configuration_laguna.py / modeling_laguna.py ship in the repo, so --trust-remote-code is required.

Caveats — please read

KV scales are declared but not baked. hf_quant_config.json sets kv_cache_quant_algo: FP8, but this export ships no k_scale/v_scale tensors. vLLM will warn Using KV cache scaling factor 1.0 for fp8_e4m3 and, with --calculate-kv-scales, derive them from the first batch — which is non-deterministic across sessions. For a calibrated KV variant see Laguna-S-2.1-ModelOpt-NVFP4-W4A4-KVcal-vllm.
No quality benchmark has been published for this W4A16 arm. The tool-use and agentic numbers quoted on the W4A4 card were measured on that checkpoint and do not transfer here. Treat this repo as unbenchmarked until an arm is run.
Built and validated on a DGX Spark (GB10, sm_121a).

License

Inherits the base model's licence (OpenMDW-1.1, poolside/Laguna-S-2.1).

Which variant do you want

W4A16 (this repo) — weights 4-bit, activations 16-bit. Higher activation fidelity, and it serves anywhere NVFP4 weight-only is supported. On GB10 a weight-only NVFP4 export carries a QuantKey ending xNone, which the fused NVFP4 CUTLASS MoE kernel rejects — it falls back to MARLIN.
W4A4 — weights and activations 4-bit on the routed experts, which is what lets the fused NVFP4 MoE kernel run on a DGX Spark.

Pick this repo for fidelity/portability, the W4A4 repo for fused-kernel decode throughput.

Serving (vLLM)

bash
vllm serve JasonW2025/Laguna-S-2.1-ModelOpt-NVFP4-W4A16-vllm \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --no-enable-prefix-caching

configuration_laguna.py / modeling_laguna.py ship in the repo, so --trust-remote-code is required.

Caveats — please read

KV scales are declared but not baked. hf_quant_config.json sets kv_cache_quant_algo: FP8, but this export ships no k_scale/v_scale tensors. vLLM will warn Using KV cache scaling factor 1.0 for fp8_e4m3 and, with --calculate-kv-scales, derive them from the first batch — which is non-deterministic across sessions. For a calibrated KV variant see Laguna-S-2.1-ModelOpt-NVFP4-W4A4-KVcal-vllm.
No quality benchmark has been published for this W4A16 arm. The tool-use and agentic numbers quoted on the W4A4 card were measured on that checkpoint and do not transfer here. Treat this repo as unbenchmarked until an arm is run.
Built and validated on a DGX Spark (GB10, sm_121a).

License

Inherits the base model's licence (OpenMDW-1.1, poolside/Laguna-S-2.1).

Laguna-S-2.1-ModelOpt-NVFP4-W4A16-vllm

README

Which variant do you want

Serving (vLLM)

Caveats — please read

License

Explore FriendliAI today

README

Which variant do you want

Serving (vLLM)

Caveats — please read

License