prithivMLmods

OpenCaption-4B-VL-SFT-v1.0

Deploy Dedicated

README

License: apache-2.0

Key Highlights

Qwen 3-VL Foundation: Built on top of Qwen/Qwen3-VL-4B-Instruct.
Dense Visual Captioning: Focused on detailed image understanding and long-form image description.
Fine-Grained Descriptions: Captures object relationships, spatial layout, attributes, actions, lighting, and context.
Captioning-Oriented Training: Optimized for detailed image caption generation rather than generic short descriptions.
Research-Focused Release: Intended for experimentation, evaluation, and captioning research.
Efficient 4B Deployment: Suitable for local inference and research workflows.

Quick Start with Transformers

Install the required packages:

bash
pip install transformers accelerate qwen-vl-utils

Use the model with the following example:

python
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

# Load the model
model = Qwen3VLForConditionalGeneration.from_pretrained(
    "prithivMLmods/OpenCaption-4B-VL-SFT-v1.0",
    torch_dtype="auto",
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    "prithivMLmods/OpenCaption-4B-VL-SFT-v1.0"
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {
                "type": "text",
                "text": "Provide a detailed caption for this image with fine-grained visual details."
            },
        ],
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=512
)

generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)

print(output_text[0])

System Prompt

OpenCaption-4B-VL-SFT-v1.0 is trained with a specific system prompt. For the best results, use it verbatim.

text
You are a detailed image captioning assistant.

Structure every caption as follows:

1. Open with one sentence naming the shot type (e.g., eye-level, wide-angle, close-up), the overall setting, and the time of day or lighting condition.

2. Break the rest of the description into thematic sections, each introduced by a bold markdown header ending in a colon (e.g., **The Subject:**, **The Background:**, **Atmosphere & Lighting:**), chosen to fit what is actually in the image.

3. Within each section, use bullet points to list specific, concrete details, including positions, colors, textures, materials, actions, spatial relationships, and any legible text or fine-grained visual elements. If a section covers multiple distinct areas of the frame, introduce each with its own nested bold sub-label ending in a colon (e.g., **Foreground Right:**, **Background:**) before its bullet points.

4. Close with a short, unheaded paragraph beginning with "In summary," that ties the scene together and conveys its overall mood or narrative.

Use precise, sensory, and fluent language throughout. Describe only what is visible in the image, avoid speculation or unsupported inferences, and do not use emojis.

Training Details

Table with columns: Setting, Value
Setting	Value
Base Model	Qwen/Qwen3-VL-4B-Instruct
Training Method	Single-stage supervised fine-tuning (SFT)
Training Objective	Fine-grained image captioning and dense visual description
Training Precision	BF16 (Full Precision)
Training Data	10K Dense Image Caption Traces

Intended Use

Image Captioning: Generating clear and detailed captions for images.
Dense Image Understanding: Describing scenes with richer visual context and fine-grained detail.
Visual Reasoning: Explaining object placement, interactions, and relationships in an image.
Research and Evaluation: Studying caption quality, visual grounding, and multimodal fine-tuning.
Local Deployment: Running a compact 4B vision-language captioning model for experimentation.

Limitations

Experimental Model: Output quality may vary across domains and image styles.
Caption Biases: Performance depends on the diversity and quality of the training traces.
Hallucinations: Like other multimodal models, the model may occasionally describe details that are not present.
Not a Ground-Truth Detector: The model is designed for captioning and visual description, not strict object verification.

Acknowledgements

Qwen/Qwen3-VL-4B-Instruct: Base model used for this project.
Transformers: Transformers provides state-of-the-art machine learning models for text, computer vision, audio, video, and multimodal tasks, supporting both inference and training.
TRL - Transformers Reinforcement Learning: TRL is a full stack library providing tools to train transformer language models with methods including Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Reward Modeling, and more.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

prithivMLmods

Model Tree

Base

Qwen/Qwen3-VL-4B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer