Index-Translate-35B-A3B-preview-FP8 API & Inference Endpoint

Scheme: FP8_DYNAMIC (FP8 E4M3 weights, per-token dynamic FP8 activations), produced with llm-compressor (quantization_scheme recorded in recipe.yaml).
All Linear layers of the language model are quantized; the vision tower, multi-modal projector, lm_head, and embeddings are kept in BF16. MTP (multi-token prediction) weights are preserved.
Format: compressed-tensors safetensors — load directly with vLLM (quantization="compressed-tensors") or transformers.

Measured on an NVIDIA A100 against the original BF16 checkpoint (greedy decoding, official translation prompt):

Table with columns: Metric, BF16, FP8, Delta
Metric	BF16	FP8	Delta
Perplexity (fixed corpus)	2.8140	2.8158	+0.06%
zh→en generation identical	—	—	yes
en→zh generation identical	—	—	yes

bash

vllm serve IndexTeam/Index-Translate-35B-A3B-preview-FP8 --host 127.0.0.1 --port 8000 --max-model-len 4096

Translation prompt format (greedy decoding, temperature=0 recommended):

text

请将以下文本翻译为{target-language}，直接输出翻译结果，不要进行任何解释。{source-text}

See the base model card for the full instTrans constrained-translation format and serving presets. GGUF builds for local inference are published in Index-Translate-35B-A3B-preview-GGUF.

Quantized and published by the Index team, 2026-10-03.

Scheme: FP8_DYNAMIC (FP8 E4M3 weights, per-token dynamic FP8 activations), produced with llm-compressor (quantization_scheme recorded in recipe.yaml).
All Linear layers of the language model are quantized; the vision tower, multi-modal projector, lm_head, and embeddings are kept in BF16. MTP (multi-token prediction) weights are preserved.
Format: compressed-tensors safetensors — load directly with vLLM (quantization="compressed-tensors") or transformers.