MiniCPM-V-4_5-abliterated-int8 API & Inference Endpoint

About this repository

An 8-bit (bitsandbytes LLM.int8()) quantization of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated, published by WaveSpeed AI.

Note what the base model is: abliterated means the upstream author ablated the refusal direction out of MiniCPM-V-4.5. This model will therefore answer prompts the original declines, and it is on you to put your own safety layer in front of it. If you want the original behaviour, quantize openbmb/MiniCPM-V-4_5 instead.

MiniCPM-V-4.5-abliterated-int8

This is an 8-bit quantized version of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated using bitsandbytes int8 quantization.

Model Details

Base Model: huihui-ai/Huihui-MiniCPM-V-4_5-abliterated
Quantization: 8-bit integer using bitsandbytes
Model Size: ~9.35 GB (79.4% reduction from original 45.28 GB)
Compute dtype: float16
Quantization method: LLM.int8() with mixed-precision decomposition

Quantization Configuration

json
{
  "load_in_8bit": true,
  "bnb_8bit_compute_dtype": "float16",
  "bnb_8bit_quant_type": "int8",
  "llm_int8_skip_modules": ["lm_head", "vision"],
  "llm_int8_threshold": 6.0,
  "quant_method": "bitsandbytes"
}

Key Features

Mixed Precision: Uses int8 for weights with fp16 for activations
Outlier Management: Automatically handles outliers in fp16 for better accuracy
Selective Quantization: Skips critical modules (lm_head, vision) to preserve quality
Better accuracy than int4: While larger than 4-bit, provides significantly better quality

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    trust_remote_code=True
)

# For inference
# The model will automatically use int8 weights with fp16 compute

Requirements

transformers>=4.35.0
bitsandbytes>=0.41.0
torch>=2.0.0
accelerate>=0.20.0
CUDA-capable GPU (int8 quantization requires CUDA)

Performance Notes

Memory Usage: ~9.35 GB VRAM required
Speed: Slightly slower than fp16 due to dequantization overhead
Quality: Better preservation of model quality compared to 4-bit quantization
Best for: Users who need better quality than 4-bit but still want memory savings

Comparison with Other Quantizations

Table with columns: Version, Size, Relative Quality, Use Case
Version	Size	Relative Quality	Use Case
Original (fp16)	45.28 GB	Best	Maximum quality, high VRAM
int8 (this)	9.35 GB	Very Good	Balanced quality/memory
int4	6.09 GB	Good	Maximum memory savings

License

Same as the original model - please refer to the base model's license.

Acknowledgments

Original model by huihui-ai
Quantization using bitsandbytes LLM.int8() method

About this repository

An 8-bit (bitsandbytes LLM.int8()) quantization of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated, published by WaveSpeed AI.

MiniCPM-V-4.5-abliterated-int8

This is an 8-bit quantized version of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated using bitsandbytes int8 quantization.

Model Details

Base Model: huihui-ai/Huihui-MiniCPM-V-4_5-abliterated
Quantization: 8-bit integer using bitsandbytes
Model Size: ~9.35 GB (79.4% reduction from original 45.28 GB)
Compute dtype: float16
Quantization method: LLM.int8() with mixed-precision decomposition

Quantization Configuration

json
{
  "load_in_8bit": true,
  "bnb_8bit_compute_dtype": "float16",
  "bnb_8bit_quant_type": "int8",
  "llm_int8_skip_modules": ["lm_head", "vision"],
  "llm_int8_threshold": 6.0,
  "quant_method": "bitsandbytes"
}

Key Features

Mixed Precision: Uses int8 for weights with fp16 for activations
Outlier Management: Automatically handles outliers in fp16 for better accuracy
Selective Quantization: Skips critical modules (lm_head, vision) to preserve quality
Better accuracy than int4: While larger than 4-bit, provides significantly better quality

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    trust_remote_code=True
)

# For inference
# The model will automatically use int8 weights with fp16 compute

Requirements

transformers>=4.35.0
bitsandbytes>=0.41.0
torch>=2.0.0
accelerate>=0.20.0
CUDA-capable GPU (int8 quantization requires CUDA)

Performance Notes

Memory Usage: ~9.35 GB VRAM required
Speed: Slightly slower than fp16 due to dequantization overhead
Quality: Better preservation of model quality compared to 4-bit quantization
Best for: Users who need better quality than 4-bit but still want memory savings

Comparison with Other Quantizations

Table with columns: Version, Size, Relative Quality, Use Case
Version	Size	Relative Quality	Use Case
Original (fp16)	45.28 GB	Best	Maximum quality, high VRAM
int8 (this)	9.35 GB	Very Good	Balanced quality/memory
int4	6.09 GB	Good	Maximum memory savings

License

Same as the original model - please refer to the base model's license.

Acknowledgments

Original model by huihui-ai
Quantization using bitsandbytes LLM.int8() method

MiniCPM-V-4_5-abliterated-int8

README

About this repository

MiniCPM-V-4.5-abliterated-int8

Model Details

Quantization Configuration

Key Features

Usage

Requirements

Performance Notes

Comparison with Other Quantizations

License

Acknowledgments

Explore FriendliAI today

README

About this repository

MiniCPM-V-4.5-abliterated-int8

Model Details

Quantization Configuration

Key Features

Usage

Requirements

Performance Notes

Comparison with Other Quantizations

License

Acknowledgments