pblrvo

Qwen3-8B-Game-semantic-IDs-v3

README

License: apache-2.0

Training procedure

Two-stage fine-tune of Qwen/Qwen3-8B, run on a RunPod A100 80GB:

Embedding warmup — freezes every parameter except embed_tokens/lm_head, initializes the ~1,026 new semantic-ID tokens via codebook-grounded initialization (projecting the trained RQ-VAE codebook into the LLM's embedding space, informed by STAR, arXiv 2604.02324), then trains those two matrices for 750 steps at a high learning rate on grounding-only examples.
Full-parameter fine-tune — every parameter trainable (unlike v2's rank-8 LoRA adapter), 2 full epochs, learning rate 2e-5, 8-bit AdamW (bitsandbytes), effective batch size 128, max_seq_length=192.

Full fine-tuning was chosen over QLoRA for this run specifically: teaching ~1,000 new semantic-ID tokens real grounding across every layer is exactly the kind of representation shift a low-rank adapter is limited in reshaping, and the token budget (~48M tokens for 2 epochs) is small enough to make full fine-tuning affordable on a single 80GB GPU.

Dataset

Trained on pblrvo/steam-games-semanticIds-instructions-v3: ~396,000 examples across 8 tasks (up from 7 in v2 — adds a semantic-ID relatedness task), rebuilt with several ground-truth quality fixes over v2's dataset:

Sequences filtered to non-zero-playtime items only (v2's data included "owned but never played" items as valid history/targets)
similar_item/nl_similar_item ranked by PMI instead of raw co-occurrence count (raw counts favored mutually-popular-but-unrelated pairs)
New relatedness task: given two semantic IDs, predict whether they share a broad family (ground truth: shared RQ-VAE codebook level-0 code) — teaches the model the ID's internal compositional structure, which none of the other 7 tasks test directly
Synthetic sequential examples (k-NN walks over item embeddings) topping up under-exposed items, kept strictly out of the validation split
Instruction/query phrasing diversity expanded from 2-5 templates per task to 6-12

Evaluation results

Recall@K / NDCG@K via constrained beam search (num_beams=10), matching TIGER (Rajput et al. 2023) and LC-Rec's evaluation methodology for generative recommenders. nl_preference is scored on criteria (genre/category) satisfaction instead of exact-match recall, since many items validly satisfy an open-ended query. These are the final, corrected numbers after fixing two real eval-pipeline bugs (below) — n=500 per task (n=218 for nl_preference, its val split's actual size), deterministic beam search, catalog restricted to the ~8,563 interacted items.

Table with columns: Task, Recall@5, Recall@10, NDCG@10, v2 Recall@10
Task	Recall@5	Recall@10	NDCG@10	v2 Recall@10
`grounding_name2id`	95.40%	96.80%	0.9032	41.2%
`nl_preference`	91.28%	94.95%	0.8235	87.3%
`grounding_id2name`	0.80%

For scale: on Amazon Beauty/Sports/Toys, TIGER (arXiv 2305.05065, Table 1) — the paper this evaluation methodology is drawn from — reports Recall@10 of 4.0–7.1% as state-of-the-art for generative sequential recommendation, against SASRec/S³-Rec baselines in the same range. sequential at 1.0% and similar_item/nl_similar_item at 1.0–2.0% should be read against that ceiling, not against an intuition that "correct" should look like 50%+ — this whole task family tops out much lower than grounding/lookup-style tasks because the target space is a full item catalog with only one right answer, ranked from context alone.

relatedness isn't in the table above because it isn't a ranking question — its output space is exactly {"Yes", "No"}, so it's scored as binary classification accuracy (constrained decoding restricted to those two tokens) rather than Recall@K:

Table with columns: Task, n, Accuracy, Precision, Recall, F1
Task	n	Accuracy	Precision	Recall	F1
`relatedness`	500 (of 2,568, exactly class-balanced)	100.00%	100.00%	100.00%	100.00%

Take this at face value, but with its actual difficulty in mind, not as evidence of deep semantic understanding: both items' level-0 code is a literal, visible token in the prompt (<|sid_L0_{code}|>), so the task reduces to comparing one visible substring between two strings, not recalling or reasoning about hidden structure. A perfect score here says the model reliably reads its own vocabulary correctly, which is a real and useful property (grounding_id2name/asy's weaknesses above are partly about exactly this kind of internal consistency) — it isn't evidence the model has learned anything richer about why those items share a family.

Two real eval-pipeline bugs, found by investigating the v2→v3 regression

grounding_id2name and asy scored near-zero in v2 and stayed near-zero after this run's much larger training budget, which prompted an investigation rather than accepting the number. Two bugs surfaced:

Pad-token not stripped from beam candidates. constrained_beam_search decoded candidates and stripped only tokenizer.eos_token (<|im_end|>), but beams that finish early are right-padded with a different token, pad_token (<|endoftext|>). Variable-length name-trie candidates kept trailing padding and failed exact-match lookup against the real catalog description — even when the generated text was a correct, real game. Fixed by stripping both terminators. Verified directly: manual inspection of raw generations showed the model producing correct catalog entries that were being discarded purely by this string mismatch. This alone moved grounding_id2name from 0.2%/4.4% (buggy) to 7.20% (fixed, n=500) — a real ~3x gain, not just noise.
Beam collapse on the name trie. Across every sampled example, Recall@5 == Recall@10 exactly on the name-trie tasks — beams 6–10 contributed nothing. All 10 beams return near-identical variants of one franchise (observed: three consecutive "Tales of Monkey Island Complete Pack: Chapter N" entries), because beam search shares long prefixes and can't diverge on a trie this branchy. A fix was attempted (HF's diverse/group beam search, num_beam_groups=10, diversity_penalty=1.0) and made things categorically worse — generation degenerated into a single repeated character on every example. ( ∈ {2, 5}, ∈ {0.2, 0.4}) specifically to check whether a gentler setting avoided the breakdown — it didn't; all four produced the identical degenerate output on every example (0/80 valid candidates each). Diversity strength isn't the variable that matters: HF's Hub-hosted group-beam-search generation strategy is structurally incompatible with this project's trie-constrained decoding at any setting tried. Closed, not open — left at plain beam search (). A secondary finding from the same probe: on a small sample, plain beam search showed collapse at all (9.6/10 mean distinct candidates), suggesting the collapse concentrates in specific franchise-heavy queries rather than happening uniformly — deduplicating near-identical catalog entries within a franchise, not further beam-search tuning, is the more promising next lever. Full grid and evidence: .

The relational-task regression: cold-start is not the explanation

sequential/similar_item/nl_similar_item scoring worse than the undertrained v2, despite a full 2 epochs of full-parameter fine-tuning vs. v2's ~0.9 epochs, looked like it could be an artifact of the evaluation protocol — these tasks are held out by item (train/val split by target, not by example), so every similar_item/ nl_similar_item val target has 0% overlap with training targets. That hypothesis was tested directly and refuted: splitting sequential's val set by whether the target item appeared in training at all showed items seen in training scoring worse (0.50% Recall@10) than items never seen (1.50%) — the opposite of what an eval-only explanation predicts. (Caveat: "seen in training" for sequential specifically means the item's only training exposure was synthetic k-NN-walk sequences, which are also, by construction, the rarest items in the catalog — rarity and synthetic-data quality are confounded in this split and weren't separated.)

Ruled out: naive mode collapse (manual inspection shows the model producing different, structurally valid semantic IDs per input, not a fixed default answer regardless of context) and a popularity-baseline explanation (a "always recommend the top-10 globally popular items" baseline scores 0.00% on this same split, by construction, since popular items can't be held-out targets — uninformative here).

The more likely explanation, given the dataset: sequences are ordered by playtime, not chronology (no timestamps exist in the source data). "Predict the next-lower- playtime game from the higher-playtime ones" is a much weaker behavioral signal than genuine purchase/play order, which is what TIGER and comparable benchmarks train and evaluate on. More training epochs, more parameters, and a bigger model all failed to move this number — consistent with a data ceiling rather than a model-capacity problem.

Confirming the diagnosis: multi-target re-scoring

If sequences are ordered by playtime rather than chronology, the model shouldn't be scored on hitting one arbitrary position in that ordering — it should be scored on whether it recommends games the user actually played. Re-scoring the same beam-search candidates (same n=500, same seed, same decode settings — only the metric changes) under "did the top-k contain ANY item this user played" instead of "did it hit the one stored target":

Table with columns: Task, Exact-target Recall@10, Any-played Hit@10, Random baseline, Lift
Task	Exact-target Recall@10	Any-played Hit@10	Random baseline	Lift
`sequential`	0.80%	58.00%	9.70%	5.98x
`asy`	0.00%	36.20%	10.53%	3.44x

Restricted to the below-median-library half of users, where the metric actually discriminates (a user who's played most of the catalog is nearly impossible to miss, which drags the pooled lift down, not up, versus this stratified view):

Table with columns: Task, Any-played Hit@10, Random baseline, Lift
Task	Any-played Hit@10	Random baseline	Lift
`sequential` (n=259, ≤40 relevant items)	43.24%	2.13%	20.31x
`asy` (n=252, ≤45 relevant items)	21.83%	2.53%	8.62x

The model was retrieving genuinely relevant games well above chance the whole time — the near-zero exact-target score was measuring the arbitrariness of the playtime-sorted position, not an absence of learned signal. asy lags sequential on identical (history, target) pairs (36.2% vs 58.0%) because of the beam-collapse bug on the name trie noted above, not a difference in what either task learned; they'd be expected to converge once that's fixed. This does not make these numbers comparable to TIGER's 4–7% Recall@10 — that remains an exact-target metric on timestamped data. It reframes what sequential/asy can honestly claim to measure given source data with no timestamps; it does not turn them into a benchmark-comparable sequential recommender. Full methodology, including the per-example random-baseline derivation: src/evaluate_multi_target.py in the project repository.

Quick start

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("pblrvo/Qwen3-8B-Game-semantic-IDs-v3", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("pblrvo/Qwen3-8B-Game-semantic-IDs-v3")

messages = [{"role": "user", "content": "What is the semantic ID for this game?\nHalf-Life 2"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

For constrained decoding (guarantees a valid catalog item) and evaluation code, see the project repository.

Framework versions

Transformers: 5.5.0
TRL: 0.24.0
PyTorch: 2.13.0+cu126
Datasets: 4.3.0

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

pblrvo

Model Tree

Base

Qwen/Qwen3-8B

Fine-tuned

this model

Input Modalities

Text

Output Modalities