r0b0tlab

XYZ-Aquila-mini-NVFP4

README

License: apache-2.0

Runtime

bash

docker pull ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814 docker run --gpus all --ipc=host --network host \  -e PORT=18082 \  -e MODEL_ID=/models/model \  -e SERVED_MODEL_NAME=xyz-aquila-nvfp4 \  -e QUANTIZATION=modelopt_mixed \  -e KV_CACHE_DTYPE=fp8 \  -e ATTENTION_BACKEND=flashinfer \  -e MOE_BACKEND=flashinfer_b12x \  -e LINEAR_BACKEND=flashinfer_cutlass \  -e LANGUAGE_MODEL_ONLY=1 \  -e GPU_MEMORY_UTILIZATION=0.85 \  -e MAX_MODEL_LEN=32768 \  -e MAX_NUM_SEQS=16 \  -v $PWD:/models/model:ro \  ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814

Long-context tip: raise MAX_MODEL_LEN toward 262144 with MAX_NUM_SEQS=1 and chunked prefill (MAX_NUM_BATCHED_TOKENS can stay 16384). Host-admit before claiming full native ctx.

Native markers: FlashInferCutlassNvFp4LinearKernel, FLASHINFER_B12X MoE, FlashInfer attn arch=sm121, FP8 KV. No Marlin/emulation.

Quant recipe

Eligible MLP/experts/lm_head: NVFP4 W4A4
Attention: FP8
KV cache at serve: FP8
Excluded: vision, routers, norms, embeddings, GDN aux, MTP (MTP off in derived config only)
Calibration: 512 windowed rows
Export: ~21 GiB, 3 shards; hf_quant 291 layers (NVFP4 161 / FP8 130)

Quality results

Table with columns: Suite, Score, Notes
Suite	Score	Notes
BFCL v4 `multi_turn_base` (200)	64.00% campaign · 68.00% core-subset re-run	bfcl-eval==2025.12.17 · 0 infra errors
BFCL AST-600 micro	33.33% prior · 33.50% core-subset	multiple / parallel / parallel_multiple
AST multiple	35.0% / 35.5%	200 cases
AST parallel	46.5% / 48.5%	200 cases
AST parallel_multiple	18.5% /

Do not use diluted BFCL “Overall Acc” CSV columns when only a subset of categories was run.

Throughput (GB10, vllm bench serve)

1024 in / 256 out, ignore_eos, temp 0, 3 reps drop-first:

Table with columns: Conc., Output tok/s, Total tok/s, mean TTFT ms, mean TPOT ms, Failed
Conc.	Output tok/s	Total tok/s	mean TTFT ms	mean TPOT ms	Failed
1	64.6	325.4	694	12.8	0
2	82.4	415.3	2010	16.3	0
4	164.5	829.1

Conversion gates

Table with columns: Gate, Status
Gate	Status
B1 BF16 CPU real-residency baseline (~70.2 GiB)	PASS
C W4A4 PTQ	PASS
D packed NVFP4 export audit	PASS
E SM121 native serve	PASS

Runtime repo: https://github.com/r0b0tlab/xyz-aquila-mini-nvfp4-sm121-vllm
Bench client: https://github.com/r0b0tlab/r0b0bench (core / core-subset profiles)
Evidence HTML: see runtime repo docs/evidence-report.html

License

Weights follow upstream XYZ-Aquila-mini / Apache-2.0 where applicable. Quantization & packaging © r0b0tlab.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

r0b0tlab

Model Tree

Base

XYZAILab/XYZ-Aquila-mini

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Runtime

bash

docker pull ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814 docker run --gpus all --ipc=host --network host \  -e PORT=18082 \  -e MODEL_ID=/models/model \  -e SERVED_MODEL_NAME=xyz-aquila-nvfp4 \  -e QUANTIZATION=modelopt_mixed \  -e KV_CACHE_DTYPE=fp8 \  -e ATTENTION_BACKEND=flashinfer \  -e MOE_BACKEND=flashinfer_b12x \  -e LINEAR_BACKEND=flashinfer_cutlass \  -e LANGUAGE_MODEL_ONLY=1 \  -e GPU_MEMORY_UTILIZATION=0.85 \  -e MAX_MODEL_LEN=32768 \  -e MAX_NUM_SEQS=16 \  -v $PWD:/models/model:ro \  ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814

Long-context tip: raise MAX_MODEL_LEN toward 262144 with MAX_NUM_SEQS=1 and chunked prefill (MAX_NUM_BATCHED_TOKENS can stay 16384). Host-admit before claiming full native ctx.

Native markers: FlashInferCutlassNvFp4LinearKernel, FLASHINFER_B12X MoE, FlashInfer attn arch=sm121, FP8 KV. No Marlin/emulation.

Quant recipe

Eligible MLP/experts/lm_head: NVFP4 W4A4
Attention: FP8
KV cache at serve: FP8
Excluded: vision, routers, norms, embeddings, GDN aux, MTP (MTP off in derived config only)
Calibration: 512 windowed rows
Export: ~21 GiB, 3 shards; hf_quant 291 layers (NVFP4 161 / FP8 130)

Quality results

Table with columns: Suite, Score, Notes
Suite	Score	Notes
BFCL v4 `multi_turn_base` (200)	64.00% campaign · 68.00% core-subset re-run	bfcl-eval==2025.12.17 · 0 infra errors
BFCL AST-600 micro	33.33% prior · 33.50% core-subset	multiple / parallel / parallel_multiple
AST multiple	35.0% / 35.5%	200 cases
AST parallel	46.5% / 48.5%	200 cases
AST parallel_multiple	18.5% /

Do not use diluted BFCL “Overall Acc” CSV columns when only a subset of categories was run.

Throughput (GB10, vllm bench serve)

1024 in / 256 out, ignore_eos, temp 0, 3 reps drop-first:

Table with columns: Conc., Output tok/s, Total tok/s, mean TTFT ms, mean TPOT ms, Failed
Conc.	Output tok/s	Total tok/s	mean TTFT ms	mean TPOT ms	Failed
1	64.6	325.4	694	12.8	0
2	82.4	415.3	2010	16.3	0
4	164.5	829.1

Conversion gates

Table with columns: Gate, Status
Gate	Status
B1 BF16 CPU real-residency baseline (~70.2 GiB)	PASS
C W4A4 PTQ	PASS
D packed NVFP4 export audit	PASS
E SM121 native serve	PASS

Runtime repo: https://github.com/r0b0tlab/xyz-aquila-mini-nvfp4-sm121-vllm
Bench client: https://github.com/r0b0tlab/r0b0bench (core / core-subset profiles)
Evidence HTML: see runtime repo docs/evidence-report.html

XYZ-Aquila-mini-NVFP4

README

Runtime

Quant recipe

Quality results

Throughput (GB10, vllm bench serve)

Conversion gates

Related

License

Explore FriendliAI today

README

Runtime

Quant recipe

Quality results

Throughput (GB10, vllm bench serve)

Conversion gates

Related

License