WeGenBench-Consistency-COT

Highlights

• CoT-style deduction grading: The model first gives an overall assessment, then lists each deduction category and reason.
• Prompt-faithfulness first: It evaluates prompt-image consistency rather than aesthetic quality, realism, or artistic preference.
• Fine-grained error taxonomy: It supports categories such as entity, appearance, activity, counting, shape, material, text, and composition.
• Long-prompt ready: Training uses max_length=32768, making the model suitable for complex prompts with many visual constraints.
• Built with ms-swift: The model is trained with swift sft on top of Qwen3-VL-8B-Instruct for reproducible fine-tuning and evaluation.

What It Does

Input:

1. A generated image to be evaluated
2. The original prompt used to generate the image

Output:

text
Copy code
Score: 7/10, Total deduction: 3
Overall assessment: The main subject, composition, and overall style are mostly aligned with the prompt, but the biological details of the rear body do not fully match the requested insect abdomen.
Deduction details:
1. appearance: The prompt asks for a bee-like abdomen and tail, but the generated rear body looks more like a furry cat tail instead of a typical insect abdomen. Deduct 3 points.

The model follows a deduction-based consistency evaluation scheme. It starts from a maximum score of 10 and deducts points for explicit, observable prompt constraints that are not correctly reflected in the generated image.

Recommended Prompt

The recommended evaluation prompt is:

text
Copy code
Your task is to evaluate text-to-image consistency. Based on the given image and prompt, judge how well the generated image matches the prompt. Give a score from 1 to 10. First provide an overall assessment, then list the deduction details and error reasons. The prompt is:

Append the original generation prompt after this instruction and pass the image together with the text.

Output Format

For automatic parsing and downstream evaluation, we recommend the following format:

text
Copy code
Score: <1-10>/10, Total deduction: <0-10>
Overall assessment: <overall consistency judgment>
Deduction details:
1. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.
2. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.

Common deduction categories:

Inference with ms-swift

We recommend using TransformersEngine from ms-swift:

python
Copy code
from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest

MODEL_PATH = "yinggzhang/WeGenBench-Consistency-COT"
IMAGE_PATH = "example.png"
PROMPT = "A classic BMW sedan is parked in an indoor environment, with smooth body lines..."

judge_prompt = (
    "Your task is to evaluate text-to-image consistency. Based on the given image and prompt, "
    "judge how well the generated image matches the prompt. Give a score from 1 to 10. "
    "First provide an overall assessment, then list the deduction details and error reasons. "
    "The prompt is:"
)

engine = TransformersEngine(MODEL_PATH, max_batch_size=1)
request_config = RequestConfig(max_tokens=1024, temperature=0)

request = InferRequest(
    messages=[{"role": "user", "content": judge_prompt + PROMPT}],
    images=[IMAGE_PATH],
)

response = engine.infer([request], request_config)[0]
print(response.choices[0].message.content)

Best For

• Automatic text-to-image consistency scoring
• Cross-model comparison and leaderboard pre-screening
• Constraint-following analysis for complex prompts
• Badcase attribution and error distribution analysis
• Image-text data cleaning, filtering, and quality regression
• Human annotation assistance

Limitations

• This model evaluates prompt consistency, not general aesthetics.
• It should not be used as the final authority for highly specialized domains such as medical, legal, or safety-critical imagery.
• Scores may still be affected by visual understanding errors, prompt ambiguity, and annotation style.
• CoT-style explanations are useful for review and debugging, but production systems should still parse structured fields and keep human spot checks.
• If the input prompt contains subjective, vague, or conflicting constraints, the model may produce conservative judgments.

Citation

If you find this model useful, please cite:

bibtex
Copy code
@misc{liang2026wegenbenchmultidimensionaldiagnosticbenchmark,
      title={WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization},
      author={Qian Liang and Xiaomin Li and Ying Zhang and Jia Xu and Lihao Ni and Hongrui Li and Jingjing Li and Jing Lyu and Chen Li},
      year={2026},
      eprint={2606.20100},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.20100},
}

Contact Us

For questions, feedback, or collaboration, please contact yinggzhang@tencent.com.