Quantization summary

• Quantization backend: TorchAO
• Quantization type: INT4 weight-only
• Packing/layout used during working export: Int4TilePackedTo4dTensor
• Targeted modules: language model attention and MLP Linear layers
• Kept dense: embeddings, lm_head, vision tower, multimodal projector
• Final checkpoint size: ~2.656 GiB
• Original BF16 checkpoint size measured locally: ~14.339 GiB
• Checkpoint size reduction: ~81.48%

Smoke test result

Test problem:

Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?

Expected answer: 72

Results:

Local benchmark summary

Notes

This checkpoint is intended as a TorchAO quantized research artifact. The first validation target was successful reload, reduced checkpoint size, lower VRAM allocation, and correctness on a smoke arithmetic task.

The quantized model may not be faster than BF16 on every GPU/backend. For this checkpoint, the main gain is storage and allocated VRAM reduction.