yasu-oh

gemma-4-26B-A4B-it-qat-NVFP4

Deploy Dedicated

README

License: apache-2.0

Model Details

Base model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
Architecture: Gemma4ForConditionalGeneration
Quantization: NVFP4 W4A4
Serialization format: nvfp4-pack-quantized
Compression backend: compressed-tensors
Calibration samples: 1,024
Maximum calibration sequence length: 8,192 tokens

Calibration Data

The following datasets were used for calibration:

The calibration dataset consists of:

512 general multilingual instruction samples from Aya
512 reasoning and tool-use conversation samples from Hermes

Quantization Configuration

yaml
default_stage:
  default_modifiers:
    IMatrixGatherer:
      requires_calibration_data: true
      targets: [Linear]
      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
      weight_observer: imatrix_mse
    QuantizationModifier:
      config_groups:
        group_0:
          targets: [Linear]
          weights:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            block_structure: null
            dynamic: false
            actorder: null
            scale_dtype: torch.float8_e4m3fn
            zp_dtype: null
            observer: imatrix_mse
            observer_kwargs: {strict: true}
          input_activations:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            block_structure: null
            dynamic: local
            actorder: null
            scale_dtype: torch.float8_e4m3fn
            zp_dtype: null
            observer: static_minmax
            observer_kwargs: {}
          output_activations: null
          format: null
      targets: [Linear]
      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
      kv_cache_scheme:
        num_bits: 8
        type: float
        symmetric: true
        group_size: null
        strategy: tensor
        block_structure: null
        dynamic: false
        actorder: null
        scale_dtype: null
        zp_dtype: null
        observer: static_minmax
        observer_kwargs: {}
      bypass_divisibility_checks: false
      requires_calibration_data: true

Reproducibility

The quantization code and calibration preprocessing are available at:

gemma4-nvfp4-quantization

The quantization run used:

text
transformers:        5.12.1
llmcompressor:       0.12.1.dev89+g9e1fcd014

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

yasu-oh

Model Tree

Base

google/gemma-4-26B-A4B-it-qat-q4_0-unquantized

Quantized

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Base model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
Architecture: Gemma4ForConditionalGeneration
Quantization: NVFP4 W4A4
Serialization format: nvfp4-pack-quantized
Compression backend: compressed-tensors
Calibration samples: 1,024
Maximum calibration sequence length: 8,192 tokens

Calibration Data

The following datasets were used for calibration:

The calibration dataset consists of:

512 general multilingual instruction samples from Aya
512 reasoning and tool-use conversation samples from Hermes

Quantization Configuration

yaml
default_stage:
  default_modifiers:
    IMatrixGatherer:
      requires_calibration_data: true
      targets: [Linear]
      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
      weight_observer: imatrix_mse
    QuantizationModifier:
      config_groups:
        group_0:
          targets: [Linear]
          weights:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            block_structure: null
            dynamic: false
            actorder: null
            scale_dtype: torch.float8_e4m3fn
            zp_dtype: null
            observer: imatrix_mse
            observer_kwargs: {strict: true}
          input_activations:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            block_structure: null
            dynamic: local
            actorder: null
            scale_dtype: torch.float8_e4m3fn
            zp_dtype: null
            observer: static_minmax
            observer_kwargs: {}
          output_activations: null
          format: null
      targets: [Linear]
      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
      kv_cache_scheme:
        num_bits: 8
        type: float
        symmetric: true
        group_size: null
        strategy: tensor
        block_structure: null
        dynamic: false
        actorder: null
        scale_dtype: null
        zp_dtype: null
        observer: static_minmax
        observer_kwargs: {}
      bypass_divisibility_checks: false
      requires_calibration_data: true

Reproducibility

The quantization code and calibration preprocessing are available at:

gemma4-nvfp4-quantization

The quantization run used:

text
transformers:        5.12.1
llmcompressor:       0.12.1.dev89+g9e1fcd014