crushleorey

Qwopus3.8-27B-Flash-V2-NVFP4

README

License: apache-2.0

Model lineage

Table with columns: Field, Value
Field	Value
Base model	`Jackrong/Qwopus3.8-27B-Flash-V2`
Pinned source revision	`13f92e09a46fa364f8de1edb85684d57bda01126`
Relationship	Post-training quantization of the complete BF16 checkpoint
Architecture	Qwen3.8, 64 language-model layers, visual encoder, one MTP layer
Native context	262,144 tokens
Output format	`compressed-tensors`, mixed precision, safetensors
Quantized weight size	24,688,489,816 bytes across safetensors files
Safetensors shard count	6
Tensor count	1,967
Quantization date	`2026-09-28`
Quantization script SHA-256	`d3f5f10c8f2feb335a65d474d5c1172fbf6941fb3522c63e296031847ffe4026`

This repository changes the storage and execution precision of the upstream checkpoint. It does not add further supervised or reinforcement training, and it does not claim to improve the upstream model's quality.

License

The upstream model card declares Apache-2.0, and this derived checkpoint is published under Apache-2.0. Users must retain the upstream license and notices and independently review the licenses and usage terms of any data or software used in their own downstream work.

Quantization environment

Table with columns: Component, Version or immutable reference
Component	Version or immutable reference
Quantization container ID	`sha256:4d34e8fe542bc420cdb4ff5c35e0b7ac1f37ef4c5da81f23c70452981f2866ac`
Validated serving container	vLLM 0.30.0, `vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90`
PyTorch	`2.13.0+cu130`
Transformers	`5.16.1`
llmcompressor	`0.14.0`
compressed-tensors

The quantization container removes the vLLM Python package inherited from its base image before installing the pinned quantization stack. The serving image is the immutable vLLM 0.30.0 image shown above, not the quantization image.

Quantization method

The recipe is a quality-first Candidate B variant of the llmcompressor 0.14.0 Qwen3.8 W4A4 FP4 example. Its only intentional precision-map deviation from that example is removing lm_head from the FP8 targets and adding it to the BF16 ignore set. The source checkpoint is loaded as BF16 through Qwen3_5ForConditionalGeneration, with sequential CPU/disk offload to bound host and GPU memory use.

Calibration

text

dataset alias: perfectblendresolved dataset: mlabonne/open-perfectblendpinned snapshot/ref: af60f3c18201652a83a93f46fcfee1b646ba3df7split: train[:512]number of samples: 512maximum sequence length: 4096pipeline: sequentialsubgraphs per sequential step: 1AWQ activation-cache offload: CPUcompleted-subgraph offload: CPUmoe_calibrate_all_experts: trueshuffle_calibration_samples: falsetorch seed: 20260928

The calibration contract is therefore 512 samples × at most 4,096 tokens. This does not validate inference at 262K context length.

The runner passes the immutable dataset revision mlabonne/open-perfectblend@af60f3c18201652a83a93f46fcfee1b646ba3df7 directly to the datasets loader, disables calibration-sample shuffling, and sets the Torch seed to 20260928. The released artifact includes the same revision in CALIBRATION_DATASET_REVISION, covered by the SHA-256 manifest.

AWQ followed by GPTQ

The recipe order is fixed:

AWQModifier(duo_scaling="both", offload_device="cpu") applies activation- and weight-side scaling to reduce the effect of outliers.
GPTQModifier performs layer-wise second-order error compensation and emits the mixed-precision compressed-tensors checkpoint.

Precision map

Table with columns: Module, Stored/execution precision, Scope
Module	Stored/execution precision	Scope
MLP layers 0–55	NVFP4 W4A4	`gate_proj`, `up_proj`, and `down_proj`; 4-bit float, group size 16
MLP layers 56–63	Dynamic FP8	`gate_proj`, `up_proj`, and `down_proj`
Self-attention	Dynamic FP8	Every layer's `q_proj`, , , and

The model is exported as safe-serialized shards with a maximum shard size of 5 GB. Tokenizer, processor, chat template, source revision, and recipe metadata are preserved with the weights.

Artifact validation

Promotion validation is fail-closed. A release must verify all of the following:

The recorded source revision exactly matches the pinned SHA above.
CALIBRATION_DATASET_REVISION records mlabonne/open-perfectblend@af60f3c18201652a83a93f46fcfee1b646ba3df7.
max_position_embeddings is 262,144 and the language model has 64 layers.
quant_method=compressed-tensors, quantization_status=compressed, and format=mixed-precision are present.
Exactly one expected dynamic-FP8 group and one expected NVFP4 group exist.
The FP8 KV-cache scheme matches the recipe.
NVFP4 packed tensors cover only layers 0–55 MLP projections.
Layers 56–63 MLP projections are FP8, not NVFP4-packed.
lm_head remains a plain BF16 weight and has no packed or scale tensors.

Release metadata:

Table with columns: Check, Result
Check	Result
Structural and tensor validation	Passed: 6 safetensors shards, 1,967 indexed tensors, 24,688,489,816 bytes
Manifest filename	`QUANTIZED_MODEL.sha256`
Total tensor count	1,967
Total quantized weight bytes	24,688,489,816

The release manifest and validation marker are distributed with the artifact. Their own content hashes are deliberately not embedded in this model card, because this file is itself covered by the release manifest.

vLLM example: two GPUs, TP2

The following is the validated production launch configuration for two RTX 5090 32 GB GPUs. Set MODEL_DIR to the downloaded model directory.

bash

MODEL_DIR=/absolute/path/to/Qwopus3.8-27B-Flash-V2-NVFP4 docker run --rm \  --gpus all \  --shm-size=32g \  -p 8000:8000 \  -e CUDA_VISIBLE_DEVICES=0,1 \  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \  -e NCCL_P2P_DISABLE=1 \  -e NCCL_CUMEM_ENABLE=0 \  -v "${MODEL_DIR}:/model:ro" \  vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90 \  /model \  --served-model-name Qwen3.8-27B-TP2 \  --trust-remote-code \  --dtype auto \  --tensor-parallel-size 2 \  --disable-custom-all-reduce \  --gpu-memory-utilization 0.95 \  --max-model-len 262144 \  --max-num-seqs 2 \  --max-num-batched-tokens 4096 \  --kv-cache-dtype fp8 \  --enable-prefix-caching \  --enable-chunked-prefill \  --scheduler-reserve-full-isl \  --no-async-scheduling \  --enable-auto-tool-choice \  --exclude-tools-when-tool-choice-none \  --tool-call-parser hermes \  --reasoning-parser qwen3 \  --default-chat-template-kwargs '{"enable_thinking": false}' \  --host 0.0.0.0 \  --port 8000

This example intentionally does not enable MTP/speculative decoding. MTP-on is a separate candidate and requires its own correctness, acceptance-rate, latency, and stability evaluation.

The model's bundled chat template emits Hermes-style JSON tool calls. The qwen3_coder tool parser is incompatible with this template and must not be used for this artifact; production uses --tool-call-parser hermes.

Security warning

The example does not configure an API key. Binding to 0.0.0.0 therefore creates an unauthenticated API. Use it only on a trusted, access-controlled network. Do not expose this command directly to the public internet. Add an authentication and authorization layer before use outside a trusted network.

Validation status

Table with columns: Evaluation, Configuration, Result
Evaluation	Configuration	Result
Artifact loading	vLLM 0.30.0 pinned image, TP2	Passed
Short generation	Streaming chat and streaming tool calls	Passed
Tool-choice behavior	256 definitions; `auto`, `required`, named, `none` direct, and `none` after tool-result continuation	All passed
Multi-turn tool workflow	120 tool definitions, approximately 57K prompt tokens, strict five-call chain with call-ID/result correspondence and final completion state

Known limitations

The dataset snapshot, sample order, and Torch seed are pinned. Bit-for-bit reproducibility across machines is still not claimed because underlying CUDA kernels and libraries may be nondeterministic.
The 512×4096 calibration distribution does not cover 196K–262K histories, very large tool schemas, or every visual-input distribution.
max-num-seqs=2 is a scheduling limit. The reported C2 checks separately verified actual overlap through server metrics, but other workloads still require their own capacity testing.
Long-context acceptance preflights each chat request through vLLM /tokenize, derived from the same completion endpoint origin, requires the server to report the non-overridable production contract max_model_len=262144, rejects prompt_tokens + max_tokens overflow before GPU prefill, and checks that the completion response reports the same prompt-token count.
FP8 KV cache and mixed W4A4/FP8 weights can change quality relative to BF16.
The visual encoder remains BF16, but the quantized language model can still change image-conditioned text output.
MTP weights remain BF16, but the launch example keeps MTP disabled. No MTP-on performance or quality claim is made.
The intended 262,144-token context is the native model limit. This card does not recommend RoPE scaling beyond that value.

Intended use

This checkpoint is intended for evaluation and controlled deployment of the upstream instruction model where reduced memory use is needed. Users should evaluate their own languages, tools, visual inputs, safety requirements, and long-context distributions before deployment.

The upstream model's limitations, training-data caveats, acceptable-use requirements, and license obligations continue to apply.

References

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

crushleorey

Model Tree

Base

Jackrong/Qwopus3.8-27B-Flash-V2

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality