nmslmgl

SSVAL-7B

README

License: apache-2.0

Files

Table with columns: File, Size, What it is
File	Size	What it is
`adapter_model.safetensors`	~620 MB	LoRA adapter (rank 128, alpha 256) over the LLM's attention and MLP projections
`non_lora_trainables.bin`	~170 MB	Multimodal projector and any other non-LoRA trainables saved separately
`adapter_config.json`	<1 KB	PEFT adapter configuration
`config.json`	~4 KB	Full model configuration including vision-side and CPD settings
`trainer_state.json`	~3 MB	Training-step / loss history (optional, kept for reproducibility)

Required base dependencies

python
# pip install: transformers==4.37.2, torch>=2.1, peft>=0.10,
#              Pillow, accelerate, einops

See the companion code repository msls38/SSVAL_code for the load_pretrained_model helper, evaluation scripts, and full dependency pins.

Vision encoder and VFM teacher

Table with columns: Component, Choice
Component	Choice
Vision tower (`mm_vision_tower`)	`clip-vit-large-patch14-336`
Patch selection layer (`mm_vision_select_layer`)	-2 (second-to-last)
Feature type (`mm_vision_select_feature`)	`patch`
Image aspect ratio	`pad`
Patch merge	`flat`

VFM teacher (training-time only, not needed at inference).

Table with columns: Field, Value
Field	Value
`vfm_target`	`dinov2-vit-b`
`num_vision_prompts`	4
`num_vfm_prompts`	4
`cpd_target_layers`	`[14]`
Multi-scale factors	`[1, 2, 4]`

How to load and use

The companion repo's evaluation entry points expect the following on-disk layout:

markdown
checkpoints/
├── Qwen2.5-7B-Instruct/         # base LLM, from Qwen/Qwen2.5-7B-Instruct
└── clip-vit-large-patch14-336/  # vision tower, from openai/clip-vit-large-patch14-336

Then, for single-image inference:

bash
python SSVAL/src/inference.py \
    --model-path ./SSVAL-7B \
    --model-base ./checkpoints/Qwen2.5-7B-Instruct \
    --image-path ./my_image.jpg \
    --prompt "Describe this image."

For benchmark evaluation, e.g. MME:

bash
python SSVAL/src/eval_mme.py \
    --model-path ./SSVAL-7B \
    --model-base ./checkpoints/Qwen2.5-7B-Instruct \
    --mme-path ./playground/benchmarks/MME/MME_Benchmark

The benchmark scripts run end-to-end without modification once the companion code is installed and the dataset / benchmark directories are populated as described in the repository README.

Citation

If you use this checkpoint, please cite the SSVAL paper (full bibtex to be added once the paper is publicly available).

License

The LoRA weights and configuration in this repository are released under the Apache License 2.0 (matching the SSVAL code repository).
The base LLM Qwen2.5-7B-Instruct is subject to its own license (Qwen2.5 LICENSE).
The vision tower is subject to the OpenAI CLIP LICENSE.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

nmslmgl

Model Tree

Base

Qwen/Qwen2.5-7B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities