InfiniCloud

Qwen3.5-9B-GPTQ-INT4-G128

README

License: apache-2.0

Important: use FP8 for JSON-based workflows

Use FP8 Dynamic for workflows that exchange results in JSON or another specified format. We do not recommend this INT4 model for InfiniCloud AI (Shiraito). In this INT4 comparison, outputs violated formatting instructions by wrapping JSON in Markdown fences, using different keys, or returning an array instead of an object. Even when the answer content is correct, these outputs can cause problems for integrations that expect fixed keys, types, and structures.

Consider INT4 only when memory constraints prevent the use of FP8, such as insufficient VRAM to start FP8. In that case, validate JSON syntax, required keys, value types, and object/array structure using your actual prompts and integration before deployment.

This recommendation is a usage policy based on the limited evaluation below. It is not a measured failure rate from production InfiniCloud AI (Shiraito), and FP8 is not guaranteed to follow every JSON format perfectly either.

G64 and MLP-only INT4 did not resolve the JSON issues

On 2026-09-12, we also generated and evaluated INT4 with group size 64 and G128 with only the language MLP quantized, using the same source BF16 model and calibration data. G64 retains the original G128 quantization scope; the MLP-only variant also retains ordinary self-attention in BF16. Vision components and Gated DeltaNet linear_attn, among other exclusions, remain BF16 in all three INT4 variants.

Table with columns: Model, GSM8K flexible: % (correct/total), Japanese MGSM flexible: % (correct/total), Fixed smoke tests passed
Model	GSM8K flexible: % (correct/total)	Japanese MGSM flexible: % (correct/total)	Fixed smoke tests passed
BF16	93.56 (1234/1319)	79.60 (199/250)	3/3
FP8 Dynamic	93.78 (1237/1319)	78.80 (197/250)	3/3
INT4 G128 (this model)	92.95 (1226/1319)	76.80 (192/250)	2/3
INT4 G64	92.57 (1221/1319)	75.60 (189/250)	2/3
INT4 G128 MLP-only	93.25 (1230/1319)	76.80 (192/250)	1/3

Arithmetic evaluation used the same 5-shot setup, thinking disabled, temperature 0, at most 2048 generated tokens, and context 8192 across all models. The three smoke cases cover short Japanese text, arithmetic JSON, and image JSON; their pass counts are not a general JSON compliance rate. Formatting was requested through prompts only; JSON Schema constrained decoding and application-side repair or retry were not evaluated.

G64: flexible extraction produced 5 fewer correct English answers and 3 fewer correct Japanese answers than G128. For arithmetic JSON, it returned {"key":"answer","value":12} instead of the requested {"answer":12}.
MLP-only: flexible extraction produced 4 more correct English answers and the same number of correct Japanese answers. Arithmetic JSON used {"key_answer":12}; image JSON returned the array directly, omitting the outer object required by {"shapes":[...]}. The image colors and shapes were correct; the structural violation is distinct from an image-recognition error.
Original G128: it identified the image colors and shapes correctly but wrapped its JSON in Markdown fences, violating the JSON-only requirement.

Within this evaluation, switching to G64 or MLP-only INT4 did not resolve the JSON formatting issues or consistently improve quality. We retain and report the small English arithmetic gain for MLP-only. Weight files grew from 10.95 GB for G128 to 11.03 GB for G64 and 11.65 GB for MLP-only, providing no additional weight-memory saving. These are weight file sizes, not runtime VRAM measurements.

Japanese MGSM strict extraction is sensitive to answer formatting such as spaces and units; it is separate from testing JSON structure. Both strict and flexible metrics, standard errors, the raw smoke answers, and source hashes are included in INT4 variant comparison data. No JSON repair or rescoring was applied. These few smoke cases and arithmetic benchmarks do not establish statistical superiority, broad Japanese application quality, or the limits of other 4-bit methods.

Model and quantization

Quantized from Qwen/Qwen3.5-9B, revision c202236235762e1c871ad0ccb60c8ee5ba337b9a, by InfiniCloud.

GPTQ INT4 W4A16, symmetric group size 128, actorder=weight, BF16 activations. Calibration used 512 text conversations from UltraChat train_sft, seed 42, at most 2048 tokens per conversation.

This is a mixed-precision multimodal checkpoint. Quantization covers 128 Linear modules, containing 5,301,600,256 weight elements out of 9,409,813,744 loaded model parameters. Vision modules, Gated DeltaNet linear_attn projections, lm_head, embeddings, and other non-Linear parameters remain unquantized. This exclusion is a conservative recipe choice. The original vision processor is retained; calibration for INT4 used text only.

MTP weights are not included because the quantization model class does not load them. KV-cache quantization is not part of this recipe. Source weights include MTP tensors, so source/candidate file-size reductions also include that omission.

Intended VRAM: one L4 24GB

The deployment target is one dedicated NVIDIA L4 with 24GB VRAM, tensor parallel size 1, with no CPU weight offload. For initial L4 validation, plan for an 8,192-token total context and one concurrent request, retaining BF16 KV, prefix caching disabled, and at most one image / no video per prompt. L4 has not yet been tested; this is a capacity-planning target, not a verified minimum VRAM requirement or a guarantee of compatibility.

The existing Blackwell startup logs provide the following reference values. These are GPU-memory measurements, separate from weight file sizes.

Table with columns: Model, GPU memory reported for model loading, Sum of profiled non-cache components
Model	GPU memory reported for model loading	Sum of profiled non-cache components
FP8	12.76 GiB	approximately 15.10 GiB
INT4 G128	10.35 GiB	approximately 12.70 GiB

The last column adds weights plus non-PyTorch consumption, peak activation, and actual CUDA graph allocation: 13.06 + 1.98 + 0.06 GiB for FP8, 10.66 + 1.98 + 0.06 GiB for INT4. It excludes the attention KV / hybrid-state cache pool. These sums use rounded startup diagnostics; they are neither whole-run peak measurements nor minimum requirements. The reference setup used vLLM 0.29.0, context 16,384, maximum 8 sequences, batched-token limit 8,192, non-eager execution, and at most one image / no video per prompt. Encoder profiling included one image at the maximum feature size; the performance requests themselves were text-only.

Cache demand also depends on context length and concurrency. From the stored model configuration, the eight full-attention layers use 32 KiB per token of BF16 KV content across the model. The 24 GatedDeltaNet layers add 49.125 MiB per active request of raw state with BF16 convolution state and FP32 recurrent state. Under tensor parallel size 1, prefix caching disabled, and no speculative decoding:

Table with columns: Total tokens per request, Concurrent requests, Attention KV content, KV plus GatedDeltaNet state content
Total tokens per request	Concurrent requests	Attention KV content	KV plus GatedDeltaNet state content
8,192	1	0.250 GiB	approximately 0.298 GiB
8,192	8	2.000 GiB	approximately 2.384 GiB
16,384	1	0.500 GiB	approximately 0.548 GiB
16,384	8	4.000 GiB

Total context includes input, generated tokens, and any image tokens. This table describes tensor contents, not the allocated cache pool or total VRAM; block rounding, padding, workspaces, and runtime headroom are additional. Prefix caching can also retain additional states. FP8/INT4 weight quantization does not reduce the BF16 KV sizes in this table.

For the intended dedicated L4 setup, use --gpu-memory-utilization 0.90 as an initial planning setting: the executor budget is 0.90 × the total memory reported by CUDA. For illustration only, if CUDA reports 22.5 GiB, the budget is 20.25 GiB, leaving approximately 5.15 GiB for FP8 or 7.55 GiB for INT4 after subtracting the Blackwell non-cache reference. This is an allowance for the cache and further overhead, not a proven usable KV capacity. L4's advertised 24GB must not be substituted as 24 GiB; check its actual reported capacity. Different L4 kernels, image sizes, and serving settings can change these figures.

vLLM normally uses the remaining executor budget to preallocate cache. Consequently, nvidia-smi usage can be much larger than the weights and live request contents. The 60.92 / 63.33 GiB cache allocations in our 96GB Blackwell runs are automatic reservations, not model requirements. See vLLM memory tuning. Re-profile on L4; start with --max-model-len 8192 --max-num-seqs 1, then increase context or concurrency after checking startup and peak memory with representative text and images. The 16K × 8 rows above are sizing examples, not validated L4 operating points.

The log excerpts, source hashes, assumptions, and calculations are included in VRAM planning evidence. This guidance covers inference, not quantization or fine-tuning.

Artifact and runtime verification

Weight file: 10,950,193,416 bytes (10.95 GB; 10.20 GiB).
SHA256: e980f522852a9e09f57c0e86360fe8a37325b6d043e6e648e22a35f34243038e.
All 760 floating tensors are finite; all 128 quantization scales are positive and finite. See numerical evidence.
vLLM loaded each checkpoint and returned HTTP 200 responses with normal completion for the three fixed smoke requests. Per-case correctness and formatting outcomes are reported below, including failures.
Generated using llmcompressor 0.13.0, compressed-tensors 0.18.0, Transformers 5.14.1, PyTorch 2.13.0+cu130 on one RTX PRO 6000 Blackwell Max-Q 96GB.
Inference/evaluation used vLLM 0.29.0 with compressed-tensors 0.17.0, Transformers 5.14.1, and driver 580.159.04. The generation and inference environments are separate.

Smoke outcomes and limitations

Table with columns: Model, Japanese text, Arithmetic JSON, Image shapes / strict JSON
Model	Japanese text	Arithmetic JSON	Image shapes / strict JSON
BF16	Pass	Pass	Pass
FP8	Pass	Pass	Pass
INT4	Pass	Pass	Fail (strict_json, expected_answer)

INT4 image_shapes returned Markdown-fenced output and failed the strict JSON requirement. Raw content and the original checks are preserved in results.json; no Markdown stripping or outcome reclassification is applied. These three cases do not establish quantitative image or application-quality performance. Smoke servers were explicitly launched with enforce_eager=True.

Quality comparison

Measured on 2026-09-12 using lm-eval 0.4.13: full GSM8K test set (1,319 questions) and Japanese MGSM native chain-of-thought test set (250 questions), both 5-shot. The original tokenizer/revision and chat template were shared across all three models, with multi-turn few-shot formatting, thinking disabled, temperature 0, and at most 2,048 generated tokens. Context limit was 8,192; all formatted prompts plus the output budget fit. Task prompts, stop sequences, filters, and dataset revisions are bundled in evaluation.

Table with columns: Model, GSM8K strict / flexible (%), Japanese MGSM strict / flexible (%)
Model	GSM8K strict / flexible (%)	Japanese MGSM strict / flexible (%)
BF16	93.40 / 93.56	2.80 / 79.60
FP8	93.10 / 93.78	2.80 / 78.80
INT4	92.27 / 92.95	2.40 / 76.80

Strict and flexible answer extraction are both shown. The upstream Japanese strict extractor requires the exact answer-marker format: spaces or intervening units, such as 答えは 3 です。 or 答えは18ドルです。, can fail strict extraction despite containing a correct numeric answer. A low strict score therefore combines answer correctness with formatting sensitivity; inspect the flexible metric and sample outputs alongside it. The metrics, standard errors, sample counts, and protocol are in results.json; no statistical significance or automatic acceptance threshold is asserted. Japanese MGSM is a translated mathematics benchmark and does not establish broad Japanese application quality.

Exploratory serving performance on Blackwell

These measurements use RTX PRO 6000 Blackwell Max-Q 96GB, not L4. Each cell contains 64 synthetic requests with fixed token lengths, seed 42, ignored EOS, and unlimited request arrival rate under the stated concurrency cap. The server used BF16 KV cache, a 16,384-token context limit, maximum 8 sequences, chunked prefill, prefix caching disabled, and GPU memory utilization 0.8. Recorded serving configuration used enforce_eager=False; this setting alone is not evidence of which CUDA graphs actually executed. Short-run p95 values are exploratory.

The benchmark used zero request warmups and disabled its initial readiness-test request (ready_check_timeout_sec=0). Server startup is outside measurement, but kernel JIT during measured requests is included; these are not warmed steady-state results. Source inspection indicates that performance requests inherit vLLM's temperature 1.0 with generation-config=vllm; request payloads were not captured. The quality evaluation above explicitly uses temperature 0.

Table with columns: Model, Input / output tokens, Concurrency, TTFT p50 / p95 (ms), TPOT p50 / p95 (ms), Output tok/s, Completed
Model	Input / output tokens	Concurrency	TTFT p50 / p95 (ms)	TPOT p50 / p95 (ms)	Output tok/s	Completed
BF16	512 / 256	1	66.03 / 70.38	12.16 / 12.24	81.33	64 / 64
BF16	512 / 256	8	291.90 / 302.90	12.42 / 13.11	593.28

No L4 hardware measurement has been performed. File size and the Blackwell results do not establish L4 runtime memory fit, throughput, latency, or concurrency limits.

Use and reproduce

Use a separate environment with vllm==0.29.0 and transformers==5.14.1. The tested checkpoint loads its stored compressed-tensors quantization configuration automatically. Follow evaluation commands and environment for the exact smoke, quality, and performance settings, and standalone quantization code to regenerate either recipe.

For INT4, the calibration dataset is HuggingFaceH4/ultrachat_200k, immutable revision 8049631c405ae6576f93f445c6b8166f76f5505a. Selected row IDs and token hashes are in the INT4 quantization manifest. The corpus is English-focused; Japanese-specific calibration, quantitative vision evaluation, tool calling, MTP, and L4 deployment are not validated here.

A normalized exact-text check found none of the 1,569 complete evaluation questions within the 512 actual truncated INT4 calibration samples. The check reproduced the manifest token hash and used Unicode NFKC, casefolding, and whitespace normalization. It does not assess paraphrases, semantic overlap, or base-model pretraining contamination; its scope and evidence are recorded in results.json.

The quantization manifest records the original structural validation and generation provenance; later inference results are recorded separately. GPU reruns are not promised to be bit-identical. Original model weights retain the Qwen Apache-2.0 license; bundled tools and evaluation sources retain their included licenses and attribution.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

InfiniCloud

Model Tree

Base

Qwen/Qwen3.5-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality