ThinkingCap-Qwen3.6-27B-Abliterix-NVFP4 API & Inference Endpoint

Quantization

ModelOpt: 0.45.0
PyTorch / CUDA: 2.11.0 / CUDA 13.0
Transformers: 5.9.0
Format: ModelOpt NVFP4, group size 16, activation-aware clipping calibration
Calibration: 256 instruction conversations from HuggingFaceH4/ultrachat_200k and 128 image-question examples from the AI2D subset of HuggingFaceM4/the_cauldron
Text calibration length: 2,048 tokens; calibration images are resized to at most 1,500,000 pixels without truncating multimodal placeholder tokens
KV cache: BF16/default (kv_cache_quant_algo: null)
Execution hardware: two NVIDIA B200 GPUs; intended runtime: vLLM on Blackwell hardware, including DGX Spark / GB10-class deployments

Features

Text, image and video processor assets are included.
The vision/video tower and merger remain BF16, as do embeddings, normalizations, LM head, hybrid linear-attention modules, and MTP-sensitive paths.
The source configuration does not contain an audio tower or audio weights; audio capability is therefore not claimed.
The underlying Qwen MTP configuration is preserved. Enable self-speculation only when supported by your installed vLLM build.

No benchmark or quality claims are made here; evaluate the quantized checkpoint on your own representative text and multimodal workloads.

vLLM

bash
vllm serve edougawa/ThinkingCap-Qwen3.6-27B-Abliterix-NVFP4 \
  --quantization modelopt_fp4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For a non-speculative launch, omit --speculative-config.

Quantization

ModelOpt: 0.45.0
PyTorch / CUDA: 2.11.0 / CUDA 13.0
Transformers: 5.9.0
Format: ModelOpt NVFP4, group size 16, activation-aware clipping calibration
Calibration: 256 instruction conversations from HuggingFaceH4/ultrachat_200k and 128 image-question examples from the AI2D subset of HuggingFaceM4/the_cauldron
Text calibration length: 2,048 tokens; calibration images are resized to at most 1,500,000 pixels without truncating multimodal placeholder tokens
KV cache: BF16/default (kv_cache_quant_algo: null)
Execution hardware: two NVIDIA B200 GPUs; intended runtime: vLLM on Blackwell hardware, including DGX Spark / GB10-class deployments

Features

Text, image and video processor assets are included.
The vision/video tower and merger remain BF16, as do embeddings, normalizations, LM head, hybrid linear-attention modules, and MTP-sensitive paths.
The source configuration does not contain an audio tower or audio weights; audio capability is therefore not claimed.
The underlying Qwen MTP configuration is preserved. Enable self-speculation only when supported by your installed vLLM build.

No benchmark or quality claims are made here; evaluate the quantized checkpoint on your own representative text and multimodal workloads.

vLLM

bash
vllm serve edougawa/ThinkingCap-Qwen3.6-27B-Abliterix-NVFP4 \
  --quantization modelopt_fp4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For a non-speculative launch, omit --speculative-config.

ThinkingCap-Qwen3.6-27B-Abliterix-NVFP4

README

Quantization

Features

vLLM

Explore FriendliAI today

README

Quantization

Features

vLLM