CodeStreet

Qwen-3.5-4B-NVFP4A16

README

License: apache-2.0

Scheme

4-bit NVFP4 weights (E2M1 with an FP8 scale per block of 16), bf16 activations. input_activations is null, so vLLM selects CompressedTensorsW4A16Fp4 → MarlinNvFp4LinearKernel, not the W4A4 path that most public NVFP4 checkpoints take.

Left in bf16 via ignore: lm_head, embed_tokens, the full visual tower, every linear_attn (GDN) projection, and mtp. Only self_attn and mlp in the full-attention layers are quantized.

yaml
GPTQModifier:
  targets: [Linear]
  scheme: NVFP4A16
  block_size: 128
  dampening_frac: 0.01
  actorder: static
  ignore: ['re:.*lm_head', 're:.*embed_tokens', 're:.*visual.*',
           're:.*linear_attn.*', 're:^mtp.*']

Calibration

512 prompts sampled from a 2342-prompt pool of short real-world chat messages wrapped in a few-shot language-identification prompt — not C4 or Pile. Locale mix of the pool: es 1360, en 471, und 131, ar 93, de 68, uk 47, ru 36, and a long tail. Length buckets skew short (860 samples at 11-20 chars, 541 at 4-10). Sequence length 512.

Evaluation

Language identification, top-1 accuracy, greedy, scores read from first-token logprobs. Compared against the same base model served with vLLM's online per-tensor FP8 forced into Marlin W8A16 (VLLM_TEST_FORCE_FP8_MARLIN=1).

Table with columns: checkpoint, uk/ru bench (n=347), hard-cases bench (n=1309)
checkpoint	uk/ru bench (n=347)	hard-cases bench (n=1309)
this (NVFP4A16)	330 (95.10%)	1146 (87.55%)
base, online FP8 W8A16	333 (95.97%)	1144 (87.39%)
int8 W8A16 group-32	329 (94.81%)	1144 (87.39%)
int4 W4A16 group-128	331 (95.39%)	1135 (86.71%)
NVFP4 W4A4	326 (93.95%)	1132 (86.48%)
bf16, no quantization

No pair differs significantly (McNemar exact, all p > 0.4; vs the FP8 baseline p=0.45 and p=0.86). Notably bf16 does not top the table — on this task the spread across schemes is measurement noise on the uk↔ru boundary, so treat the ordering above as ties, not a ranking.

Both benches are disjoint from the calibration pool.

Serving

bash
vllm serve CodeStreet/Qwen-3.5-4B-NVFP4A16 \
  --language-model-only --max-model-len 512 --max-logprobs 128 \
  --gpu-memory-utilization 0.12 --trust-remote-code

gpu-memory-utilization needs more headroom than a fully-quantized NVFP4 checkpoint: the conservative ignore list keeps ~3.3 GB in bf16 (GDN projections 2.0 GB, embed_tokens 1.3 GB). At 0.07 on a 96 GB card the engine fails with max_num_seqs exceeds available Mamba cache blocks. Requires Blackwell (sm_120) or newer for the NVFP4 kernels.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

CodeStreet

Model Tree

Base

Qwen/Qwen3.5-4B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality