gemma-4-12B-it-qat-NVFP4 API & Inference Endpoint

Model Details

Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
Architecture: Gemma4UnifiedForConditionalGeneration
Quantization: NVFP4 W4A4
Serialization format: nvfp4-pack-quantized
Compression backend: compressed-tensors
Calibration samples: 1,024
Maximum calibration sequence length: 8,192 tokens

Calibration Data

The following datasets were used for calibration:

The calibration dataset consists of:

512 general multilingual instruction samples from Aya
512 reasoning and tool-use conversation samples from Hermes

Quantization Configuration

yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
      scheme: NVFP4
      kv_cache_scheme:
        num_bits: 8
        type: float
        symmetric: true
        group_size: null
        strategy: tensor
        block_structure: null
        dynamic: false
        actorder: null
        scale_dtype: null
        zp_dtype: null
        observer: static_minmax
        observer_kwargs: {}
      bypass_divisibility_checks: false

Reproducibility

The quantization code and calibration preprocessing are available at:

github.com/yasu-oh/gemma-4-31b-nvfp4-quantization

The quantization run used:

text
transformers:        5.12.1
compressed-tensors:  0.17.2.a20260623
llmcompressor:       0.12.1.dev63+g57ed2be4

Model Details

Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
Architecture: Gemma4UnifiedForConditionalGeneration
Quantization: NVFP4 W4A4
Serialization format: nvfp4-pack-quantized
Compression backend: compressed-tensors
Calibration samples: 1,024
Maximum calibration sequence length: 8,192 tokens

Calibration Data

The following datasets were used for calibration:

The calibration dataset consists of:

512 general multilingual instruction samples from Aya
512 reasoning and tool-use conversation samples from Hermes

Quantization Configuration

yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
      scheme: NVFP4
      kv_cache_scheme:
        num_bits: 8
        type: float
        symmetric: true
        group_size: null
        strategy: tensor
        block_structure: null
        dynamic: false
        actorder: null
        scale_dtype: null
        zp_dtype: null
        observer: static_minmax
        observer_kwargs: {}
      bypass_divisibility_checks: false

Reproducibility

The quantization code and calibration preprocessing are available at:

github.com/yasu-oh/gemma-4-31b-nvfp4-quantization

The quantization run used:

text
transformers:        5.12.1
compressed-tensors:  0.17.2.a20260623
llmcompressor:       0.12.1.dev63+g57ed2be4

gemma-4-12B-it-qat-NVFP4

README

Model Details

Calibration Data

Quantization Configuration

Reproducibility

Explore FriendliAI today

README

Model Details

Calibration Data

Quantization Configuration

Reproducibility