tomaarsen/tiny-random-paligemma-lora-key-mapping
Tiny-random PaliGemma checkpoint bundling a LoRA adapter that requires a key_mapping to load onto the
underlying PaliGemmaModel.
It mirrors vidore/colpali at tiny scale: the adapter's text weights
are stored under the old language_model.model.layers.* layout, so loading them onto today's
PaliGemmaModel (language_model.layers.*) needs:
from transformers import PaliGemmaModel
model = PaliGemmaModel.from_pretrained(
"tomaarsen/tiny-random-paligemma-lora-key-mapping",
key_mapping={r"language_model\.model\.": "language_model."},
)
PaliGemmaForConditionalGeneration auto-bridges this (via the llava conversion) and does not need the
mapping; the bare PaliGemmaModel does. Every lora_A weight is filled with 0.0234 and every
lora_B weight with 0.0567, so a test can assert the adapter was restored from the checkpoint.