Usage (vLLM)
vllm serve CLODEI/Latxa-3-122B-A10B-NVFP4 --quantization modelopt_fp4 --kv-cache-dtype fp8 \ --max-model-len 65536 --reasoning-parser qwen3
Latxa 3 only supports thinking mode, so allow a large max_tokens.
Measured (1x RTX PRO 6000, vLLM 0.32.0, 64K context)
- About 77 tok/s for 1 user, 57 tok/s per user at 4 users, 46 tok/s per user at 8 users (about 260 tok/s total).
- KV cache: about 715K tokens.
- Quick checks in Basque, Spanish, English, maths and code gave correct answers. No full benchmark suite
has been run yet; quality may differ slightly from the BF16 and FP8 versions.
Notes
- Like the official Latxa 3 checkpoints, this model has no MTP (multi-token prediction) head, so MTP
speculative decoding is not available.
- License: Apache-2.0, same as the base model.