Output
{"mentions_genai": true, "confidence": "high", "reasoning": "Calls the portraits 'AI slop'."}
Table with columns: Field, Values| Field | Values |
|---|
mentions_genai | true if the review discusses generative AI in the game's production, else false |
confidence | "low", "medium", "high" |
reasoning | one or two sentences quoting the decisive phrase |
true covers explicit terms ("AI art", "AI-generated", "AI slop"), tool and
vendor names, their equivalents in other languages, reactions to a
developer's AI disclosure, and speculative claims when the context supports
a genuine suspicion. false covers gameplay AI (enemy or NPC behaviour),
procedural generation, the reviewer's own use of AI tools, and rhetorical
comparisons ("an AI could have written this"). The full rubric is the
system prompt in prompt_system.md.
Usage
The model expects the system prompt from prompt_system.md and a user
message in the format of prompt_user_template.md
(game name, genres, tags, release date, the developer's AI disclosure if
any, review date, language, recommendation, and the review text). Both files
are in this repository.
import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "LRieser/steam-ai-mention-qwen3-8b-merged"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto").eval()
system_prompt = open(hf_hub_download(repo, "prompt_system.md"), encoding="utf-8").read()
user_message = """=== GAME ===
name: Example Quest
genres: RPG, Indie
top_tags: Pixel Graphics, Story Rich, 2D
release_date: 2024-09-12
early_access: False
=== DISCLOSURE ===
flag: no
=== REVIEW ===
written_on: 2025-03-02
regime: post_mandate_2024 (computed from effective date = max(written, edited))
language: english
voted_up: False
text: |
The portraits are AI slop, every face has that same plastic look."""
messages = [{"role": "system", "content": system_prompt},
{"role": "user", "content": user_message}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=128, do_sample=False)
result = json.loads(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
result["mentions_genai"]
For corpus-scale inference serve the weights with vLLM (about 11 reviews per
second on an RTX 4090, against about one per second with plain generate()):
vllm serve LRieser/steam-ai-mention-qwen3-8b-merged --dtype bfloat16
Training
- Base model:
Qwen/Qwen3-8B.
- Method: QLoRA with DoRA (
use_dora=True), rank 16, alpha 32, dropout 0,
on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj,
down_proj. NF4-quantised base, bf16 compute. The adapter was then merged
into the bf16 base weights.
- Data: 19,152 Steam reviews, 4,585 positive. Two sampling strata: reviews
matching a multilingual generative-AI keyword list (14,379) and a uniform
random sample of reviews (4,773). Labels were produced by DeepSeek V4-Pro
in thinking mode under the rubric in
prompt_system.md, after a
250-review audit that refined the rubric. Twenty languages; the largest
are English (55%), Simplified Chinese (12%), Russian (10%), Italian (6%),
and German (4%).
- Optimisation: paged AdamW 8-bit, learning rate 2e-4, cosine schedule,
warmup ratio 0.05, weight decay 0.01, gradient clipping 1.0, batch size 1
with gradient accumulation 4, gradient checkpointing, maximum sequence
length 4096. Loss on the assistant JSON only.

Evaluation
Held-out test split of 960 reviews (seed 42), never used in training, scored
with these weights under vLLM. The keyword-matched stratum (n=720) contains
almost all positives; the random stratum (n=240) checks for false positives
on ordinary reviews.
Table with columns: Metric, Value| Metric | Value |
|---|
| Macro-F1, keyword-matched stratum | 0.953 |
| Positive-class precision / recall / F1 | 0.959 / 0.912 / 0.935 |
| Macro-F1, English / non-English | 0.968 / 0.940 |
| False positives, random stratum (n=240) | 0 |
| Throughput (RTX 4090, vLLM bf16) | about 11 reviews/s |
The companion encoder
(LRieser/steam-ai-mention-mmbert-base)
reaches macro-F1 0.925 on the same split at about 430 reviews per second.
Citation
@misc{rieser2026steamaimentionqwen3merged,
author = {Rieser, Lars and Ohlrogge, Fynn and Joshi, Anant and Sethi, Navneet},
title = {Steam AI-mention classifier (Qwen3-8B, merged)},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/LRieser/steam-ai-mention-qwen3-8b-merged}
}