TK17250

game-comment-vlm

README

License: apache-2.0

Supported Games

The model is trained specifically on gameplay screenshots from three titles:

Table with columns: Game, Genre
Game	Genre
Fortnite	Battle Royale
Rocket League	Vehicular Soccer
VALORANT	Tactical FPS

Commentary style and vocabulary (e.g. weapon names, map callouts, in-game terminology) are tailored to these three games. Performance on other games or genres outside this scope is not guaranteed, since the training data does not cover them.

Training Data

Fine-tuned on a synthetic dataset of ~1,600+ gameplay screenshots paired with commentator-style captions.
Captions were generated using Minimax-M3 as a teacher model, then quality-filtered using a 0–5 relevance/quality score before training.
Each training example consists of: a gameplay screenshot, a system persona prompt ("passionate live game commentator"), and a target caption written in a natural, spoken commentary style.

How to Use

python
from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
import torch

processor = AutoProcessor.from_pretrained("TK17250/game-comment-vlm")
model = AutoModelForVision2Seq.from_pretrained(
    "TK17250/game-comment-vlm",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

image = Image.open("your_gameplay_screenshot.jpg").convert("RGB")

system_prompt = (
    "You are a passionate live game commentator. You watch a screenshot from a "
    "video game and give a short, natural spoken-style comment on the gameplay "
    "moment, reacting to how good or bad the play looks."
)
user_prompt = "Comment on this gameplay moment."

messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": f"{system_prompt}\n\n{user_prompt}"}
    ]}
]

input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image, input_text, add_special_tokens=False, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=128, temperature=0.7, min_p=0.1)
print(processor.decode(output[0], skip_special_tokens=True))

Note: This model's chat template does not support a separate system role alongside image inputs. Combine the persona/system instruction directly into the user message text, as shown above.

Using Unsloth (faster inference)

python
from unsloth import FastVisionModel
from transformers import TextStreamer

model, tokenizer = FastVisionModel.from_pretrained("TK17250/game-comment-vlm", load_in_4bit=False)
FastVisionModel.for_inference(model)

# ... build messages/inputs as above using `tokenizer` instead of `processor`
_ = model.generate(**inputs, streamer=TextStreamer(tokenizer, skip_prompt=True), max_new_tokens=128)

Intended Use

Generating live-style commentary/captions for gameplay clips or screenshots (Fortnite, Rocket League, VALORANT).
Powering auto-commentary features for streaming tools, highlight reels, or gameplay analysis apps.

Limitations

Trained only on the three games listed above; output quality and terminology accuracy drop significantly for other games.
As a fine-tune of an 11B vision-language model on a relatively small synthetic dataset (~1,600 examples), the model may occasionally produce generic or repetitive commentary, or misread fine-grained HUD/UI details (e.g. exact kill counts, health values).
Captions are stylistic/entertainment-oriented and should not be relied on for precise, factual game-state reporting.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

TK17250

Model Tree

Base

unsloth/Llama-3.2-11B-Vision-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality