yinggzhang

WeGenBench-Consistency-COT

Deploy Dedicated

README

License: apache-2.0

Highlights

CoT-style deduction grading: The model first gives an overall assessment, then lists each deduction category and reason.
Prompt-faithfulness first: It evaluates prompt-image consistency rather than aesthetic quality, realism, or artistic preference.
Fine-grained error taxonomy: It supports categories such as entity, appearance, activity, counting, shape, material, text, and composition.
Long-prompt ready: Training uses max_length=32768, making the model suitable for complex prompts with many visual constraints.
Built with ms-swift: The model is trained with swift sft on top of Qwen3-VL-8B-Instruct for reproducible fine-tuning and evaluation.

What It Does

Input:

A generated image to be evaluated
The original prompt used to generate the image

Output:

text

Score: 7/10, Total deduction: 3Overall assessment: The main subject, composition, and overall style are mostly aligned with the prompt, but the biological details of the rear body do not fully match the requested insect abdomen.Deduction details:1. appearance: The prompt asks for a bee-like abdomen and tail, but the generated rear body looks more like a furry cat tail instead of a typical insect abdomen. Deduct 3 points.

The model follows a deduction-based consistency evaluation scheme. It starts from a maximum score of 10 and deducts points for explicit, observable prompt constraints that are not correctly reflected in the generated image.

Recommended Prompt

The recommended evaluation prompt is:

text

Your task is to evaluate text-to-image consistency. Based on the given image and prompt, judge how well the generated image matches the prompt. Give a score from 1 to 10. First provide an overall assessment, then list the deduction details and error reasons. The prompt is:

Append the original generation prompt after this instruction and pass the image together with the text.

Output Format

For automatic parsing and downstream evaluation, we recommend the following format:

text

Score: <1-10>/10, Total deduction: <0-10>Overall assessment: <overall consistency judgment>Deduction details:1. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.2. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.

Common deduction categories:

Table with columns: Category, Meaning
Category	Meaning
`entity`	Missing entity, wrong entity, or mismatched subject
`appearance`	Incorrect appearance, color, clothing, or local visual attribute
`activity`	Wrong action, pose, gesture, or interaction
`counting`	Incorrect number of objects or people
`shape`	Incorrect shape, structure, or geometry
`material`

Inference with ms-swift

We recommend using TransformersEngine from ms-swift:

python

from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest MODEL_PATH = "yinggzhang/WeGenBench-Consistency-COT"IMAGE_PATH = "example.png"PROMPT = "A classic BMW sedan is parked in an indoor environment, with smooth body lines..." judge_prompt = (    "Your task is to evaluate text-to-image consistency. Based on the given image and prompt, "    "judge how well the generated image matches the prompt. Give a score from 1 to 10. "    "First provide an overall assessment, then list the deduction details and error reasons. "    "The prompt is:") engine = TransformersEngine(MODEL_PATH, max_batch_size=1)request_config = RequestConfig(max_tokens=1024, temperature=0) request = InferRequest(    messages=[{"role": "user", "content": judge_prompt + PROMPT}],    images=[IMAGE_PATH],) response = engine.infer([request], request_config)[0]print(response.choices[0].message.content)

Best For

Automatic text-to-image consistency scoring
Cross-model comparison and leaderboard pre-screening
Constraint-following analysis for complex prompts
Badcase attribution and error distribution analysis
Image-text data cleaning, filtering, and quality regression
Human annotation assistance

Limitations

This model evaluates prompt consistency, not general aesthetics.
It should not be used as the final authority for highly specialized domains such as medical, legal, or safety-critical imagery.
Scores may still be affected by visual understanding errors, prompt ambiguity, and annotation style.
CoT-style explanations are useful for review and debugging, but production systems should still parse structured fields and keep human spot checks.
If the input prompt contains subjective, vague, or conflicting constraints, the model may produce conservative judgments.

Citation

If you find this model useful, please cite:

bibtex

@misc{liang2026wegenbenchmultidimensionaldiagnosticbenchmark,      title={WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization},      author={Qian Liang and Xiaomin Li and Ying Zhang and Jia Xu and Lihao Ni and Hongrui Li and Jingjing Li and Jing Lyu and Chen Li},      year={2026},      eprint={2606.20100},      archivePrefix={arXiv},      primaryClass={cs.CV},      url={https://arxiv.org/abs/2606.20100},}

Contact Us

For questions, feedback, or collaboration, please contact yinggzhang@tencent.com.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

yinggzhang

Model Tree

Base

Qwen/Qwen3-VL-8B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Highlights

CoT-style deduction grading: The model first gives an overall assessment, then lists each deduction category and reason.
Prompt-faithfulness first: It evaluates prompt-image consistency rather than aesthetic quality, realism, or artistic preference.
Fine-grained error taxonomy: It supports categories such as entity, appearance, activity, counting, shape, material, text, and composition.
Long-prompt ready: Training uses max_length=32768, making the model suitable for complex prompts with many visual constraints.
Built with ms-swift: The model is trained with swift sft on top of Qwen3-VL-8B-Instruct for reproducible fine-tuning and evaluation.

What It Does

Input:

A generated image to be evaluated
The original prompt used to generate the image

Output:

text

Score: 7/10, Total deduction: 3Overall assessment: The main subject, composition, and overall style are mostly aligned with the prompt, but the biological details of the rear body do not fully match the requested insect abdomen.Deduction details:1. appearance: The prompt asks for a bee-like abdomen and tail, but the generated rear body looks more like a furry cat tail instead of a typical insect abdomen. Deduct 3 points.

Recommended Prompt

The recommended evaluation prompt is:

text

Your task is to evaluate text-to-image consistency. Based on the given image and prompt, judge how well the generated image matches the prompt. Give a score from 1 to 10. First provide an overall assessment, then list the deduction details and error reasons. The prompt is:

Append the original generation prompt after this instruction and pass the image together with the text.

Output Format

For automatic parsing and downstream evaluation, we recommend the following format:

text

Score: <1-10>/10, Total deduction: <0-10>Overall assessment: <overall consistency judgment>Deduction details:1. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.2. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.

Common deduction categories:

Table with columns: Category, Meaning
Category	Meaning
`entity`	Missing entity, wrong entity, or mismatched subject
`appearance`	Incorrect appearance, color, clothing, or local visual attribute
`activity`	Wrong action, pose, gesture, or interaction
`counting`	Incorrect number of objects or people
`shape`	Incorrect shape, structure, or geometry
`material`

Inference with ms-swift

We recommend using TransformersEngine from ms-swift:

python

from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest MODEL_PATH = "yinggzhang/WeGenBench-Consistency-COT"IMAGE_PATH = "example.png"PROMPT = "A classic BMW sedan is parked in an indoor environment, with smooth body lines..." judge_prompt = (    "Your task is to evaluate text-to-image consistency. Based on the given image and prompt, "    "judge how well the generated image matches the prompt. Give a score from 1 to 10. "    "First provide an overall assessment, then list the deduction details and error reasons. "    "The prompt is:") engine = TransformersEngine(MODEL_PATH, max_batch_size=1)request_config = RequestConfig(max_tokens=1024, temperature=0) request = InferRequest(    messages=[{"role": "user", "content": judge_prompt + PROMPT}],    images=[IMAGE_PATH],) response = engine.infer([request], request_config)[0]print(response.choices[0].message.content)

Best For

Automatic text-to-image consistency scoring
Cross-model comparison and leaderboard pre-screening
Constraint-following analysis for complex prompts
Badcase attribution and error distribution analysis
Image-text data cleaning, filtering, and quality regression
Human annotation assistance

Limitations

This model evaluates prompt consistency, not general aesthetics.
It should not be used as the final authority for highly specialized domains such as medical, legal, or safety-critical imagery.
Scores may still be affected by visual understanding errors, prompt ambiguity, and annotation style.
CoT-style explanations are useful for review and debugging, but production systems should still parse structured fields and keep human spot checks.
If the input prompt contains subjective, vague, or conflicting constraints, the model may produce conservative judgments.

Citation

If you find this model useful, please cite:

bibtex

@misc{liang2026wegenbenchmultidimensionaldiagnosticbenchmark,      title={WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization},      author={Qian Liang and Xiaomin Li and Ying Zhang and Jia Xu and Lihao Ni and Hongrui Li and Jingjing Li and Jing Lyu and Chen Li},      year={2026},      eprint={2606.20100},      archivePrefix={arXiv},      primaryClass={cs.CV},      url={https://arxiv.org/abs/2606.20100},}

Contact Us

For questions, feedback, or collaboration, please contact yinggzhang@tencent.com.