InfiniCloud

Qwen3.5-9B-FP8-Dynamic

README

License: apache-2.0

Intended VRAM: one L4 24GB

The deployment target is one dedicated NVIDIA L4 with 24GB VRAM, tensor parallel size 1, with no CPU weight offload. For initial L4 validation, plan for an 8,192-token total context and one concurrent request, retaining BF16 KV, prefix caching disabled, and at most one image / no video per prompt. L4 has not yet been tested; this is a capacity-planning target, not a verified minimum VRAM requirement or a guarantee of compatibility.

The existing Blackwell startup logs provide the following reference values. These are GPU-memory measurements, separate from weight file sizes.

Table with columns: Model, GPU memory reported for model loading, Sum of profiled non-cache components
Model	GPU memory reported for model loading	Sum of profiled non-cache components
FP8	12.76 GiB	approximately 15.10 GiB
INT4 G128	10.35 GiB	approximately 12.70 GiB

The last column adds weights plus non-PyTorch consumption, peak activation, and actual CUDA graph allocation: 13.06 + 1.98 + 0.06 GiB for FP8, 10.66 + 1.98 + 0.06 GiB for INT4. It excludes the attention KV / hybrid-state cache pool. These sums use rounded startup diagnostics; they are neither whole-run peak measurements nor minimum requirements. The reference setup used vLLM 0.29.0, context 16,384, maximum 8 sequences, batched-token limit 8,192, non-eager execution, and at most one image / no video per prompt. Encoder profiling included one image at the maximum feature size; the performance requests themselves were text-only.

Cache demand also depends on context length and concurrency. From the stored model configuration, the eight full-attention layers use 32 KiB per token of BF16 KV content across the model. The 24 GatedDeltaNet layers add 49.125 MiB per active request of raw state with BF16 convolution state and FP32 recurrent state. Under tensor parallel size 1, prefix caching disabled, and no speculative decoding:

Table with columns: Total tokens per request, Concurrent requests, Attention KV content, KV plus GatedDeltaNet state content
Total tokens per request	Concurrent requests	Attention KV content	KV plus GatedDeltaNet state content
8,192	1	0.250 GiB	approximately 0.298 GiB
8,192	8	2.000 GiB	approximately 2.384 GiB
16,384	1	0.500 GiB	approximately 0.548 GiB
16,384	8	4.000 GiB

Total context includes input, generated tokens, and any image tokens. This table describes tensor contents, not the allocated cache pool or total VRAM; block rounding, padding, workspaces, and runtime headroom are additional. Prefix caching can also retain additional states. FP8/INT4 weight quantization does not reduce the BF16 KV sizes in this table.

For the intended dedicated L4 setup, use --gpu-memory-utilization 0.90 as an initial planning setting: the executor budget is 0.90 × the total memory reported by CUDA. For illustration only, if CUDA reports 22.5 GiB, the budget is 20.25 GiB, leaving approximately 5.15 GiB for FP8 or 7.55 GiB for INT4 after subtracting the Blackwell non-cache reference. This is an allowance for the cache and further overhead, not a proven usable KV capacity. L4's advertised 24GB must not be substituted as 24 GiB; check its actual reported capacity. Different L4 kernels, image sizes, and serving settings can change these figures.

vLLM normally uses the remaining executor budget to preallocate cache. Consequently, nvidia-smi usage can be much larger than the weights and live request contents. The 60.92 / 63.33 GiB cache allocations in our 96GB Blackwell runs are automatic reservations, not model requirements. See vLLM memory tuning. Re-profile on L4; start with --max-model-len 8192 --max-num-seqs 1, then increase context or concurrency after checking startup and peak memory with representative text and images. The 16K × 8 rows above are sizing examples, not validated L4 operating points.

The log excerpts, source hashes, assumptions, and calculations are included in VRAM planning evidence. This guidance covers inference, not quantization or fine-tuning.

Artifact and runtime verification

Weight file: 13,520,203,008 bytes (13.52 GB; 12.59 GiB).
SHA256: a928657692dbbad12dc248c51f82db70c6ccf868a486b25395028d223eb14768.
All 888 floating tensors are finite; all 128 quantization scales are positive and finite. See numerical evidence.
vLLM loaded each checkpoint and returned HTTP 200 responses with normal completion for the three fixed smoke requests. Per-case correctness and formatting outcomes are reported below, including failures.
Generated using llmcompressor 0.13.0, compressed-tensors 0.18.0, Transformers 5.14.1, PyTorch 2.13.0+cu130 on one RTX PRO 6000 Blackwell Max-Q 96GB.
Inference/evaluation used vLLM 0.29.0 with compressed-tensors 0.17.0, Transformers 5.14.1, and driver 580.159.04. The generation and inference environments are separate.

Smoke outcomes and limitations

Table with columns: Model, Japanese text, Arithmetic JSON, Image shapes / strict JSON
Model	Japanese text	Arithmetic JSON	Image shapes / strict JSON
BF16	Pass	Pass	Pass
FP8	Pass	Pass	Pass
INT4	Pass	Pass	Fail (strict_json, expected_answer)

INT4 image_shapes returned Markdown-fenced output and failed the strict JSON requirement. Raw content and the original checks are preserved in results.json; no Markdown stripping or outcome reclassification is applied. These three cases do not establish quantitative image or application-quality performance. Smoke servers were explicitly launched with enforce_eager=True.

Quality comparison

Measured on 2026-09-12 using lm-eval 0.4.13: full GSM8K test set (1,319 questions) and Japanese MGSM native chain-of-thought test set (250 questions), both 5-shot. The original tokenizer/revision and chat template were shared across all three models, with multi-turn few-shot formatting, thinking disabled, temperature 0, and at most 2,048 generated tokens. Context limit was 8,192; all formatted prompts plus the output budget fit. Task prompts, stop sequences, filters, and dataset revisions are bundled in evaluation.

Table with columns: Model, GSM8K strict / flexible (%), Japanese MGSM strict / flexible (%)
Model	GSM8K strict / flexible (%)	Japanese MGSM strict / flexible (%)
BF16	93.40 / 93.56	2.80 / 79.60
FP8	93.10 / 93.78	2.80 / 78.80
INT4	92.27 / 92.95	2.40 / 76.80

Strict and flexible answer extraction are both shown. The upstream Japanese strict extractor requires the exact answer-marker format: spaces or intervening units, such as 答えは 3 です。 or 答えは18ドルです。, can fail strict extraction despite containing a correct numeric answer. A low strict score therefore combines answer correctness with formatting sensitivity; inspect the flexible metric and sample outputs alongside it. The metrics, standard errors, sample counts, and protocol are in results.json; no statistical significance or automatic acceptance threshold is asserted. Japanese MGSM is a translated mathematics benchmark and does not establish broad Japanese application quality.

Exploratory serving performance on Blackwell

These measurements use RTX PRO 6000 Blackwell Max-Q 96GB, not L4. Each cell contains 64 synthetic requests with fixed token lengths, seed 42, ignored EOS, and unlimited request arrival rate under the stated concurrency cap. The server used BF16 KV cache, a 16,384-token context limit, maximum 8 sequences, chunked prefill, prefix caching disabled, and GPU memory utilization 0.8. Recorded serving configuration used enforce_eager=False; this setting alone is not evidence of which CUDA graphs actually executed. Short-run p95 values are exploratory.

The benchmark used zero request warmups and disabled its initial readiness-test request (ready_check_timeout_sec=0). Server startup is outside measurement, but kernel JIT during measured requests is included; these are not warmed steady-state results. Source inspection indicates that performance requests inherit vLLM's temperature 1.0 with generation-config=vllm; request payloads were not captured. The quality evaluation above explicitly uses temperature 0.

Table with columns: Model, Input / output tokens, Concurrency, TTFT p50 / p95 (ms), TPOT p50 / p95 (ms), Output tok/s, Completed
Model	Input / output tokens	Concurrency	TTFT p50 / p95 (ms)	TPOT p50 / p95 (ms)	Output tok/s	Completed
BF16	512 / 256	1	66.03 / 70.38	12.16 / 12.24	81.33	64 / 64
BF16	512 / 256	8	291.90 / 302.90	12.42 / 13.11	593.28

No L4 hardware measurement has been performed. File size and the Blackwell results do not establish L4 runtime memory fit, throughput, latency, or concurrency limits.

Use and reproduce

Use a separate environment with vllm==0.29.0 and transformers==5.14.1. The tested checkpoint loads its stored compressed-tensors quantization configuration automatically. Follow evaluation commands and environment for the exact smoke, quality, and performance settings, and standalone quantization code to regenerate either recipe.

For INT4, the calibration dataset is HuggingFaceH4/ultrachat_200k, immutable revision 8049631c405ae6576f93f445c6b8166f76f5505a. Selected row IDs and token hashes are in the INT4 quantization manifest. The corpus is English-focused; Japanese-specific calibration, quantitative vision evaluation, tool calling, MTP, and L4 deployment are not validated here.

A normalized exact-text check found none of the 1,569 complete evaluation questions within the 512 actual truncated INT4 calibration samples. The check reproduced the manifest token hash and used Unicode NFKC, casefolding, and whitespace normalization. It does not assess paraphrases, semantic overlap, or base-model pretraining contamination; its scope and evidence are recorded in results.json.

The quantization manifest records the original structural validation and generation provenance; later inference results are recorded separately. GPU reruns are not promised to be bit-identical. Original model weights retain the Qwen Apache-2.0 license; bundled tools and evaluation sources retain their included licenses and attribution.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

InfiniCloud

Model Tree

Base

Qwen/Qwen3.5-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality