pblrvo

Qwen3-4B-Game-semantic-IDs-v2

README

License: apache-2.0

What changed from v1

Grounding train/val split fixed. v1 held out entire items from grounding's val set, so grounding was tested on items whose name<->SID mapping was structurally never in training -- not a fair test. v2 splits within each item's repeated examples instead, so every item is trained on and val tests recall under an unseen phrasing.
Two new task types: nl_preference (open-ended natural-language queries, e.g. "I want a racing game") and nl_similar_item (natural-language "recommend something like X" queries), both grounded in the same catalog/co-occurrence data as the existing tasks.
Cross-task exposure cap. An item could independently hit the ceiling in several recommendation tasks at once (sequential + asy + similar_item + nl_similar_item), compounding to ~140 total exposures for popular items vs. a dozen for a typical one -- this is what was driving the v1 model's tendency to over-recommend a handful of popular titles regardless of input. v2 caps combined exposure at 40, deliberately excluding grounding (which stays exactly uniform per item).
Stage 1 (embedding warmup) redesigned per STAR (arXiv, "Semantic-ID Token-Embedding Alignment for Generative Recommenders"): gradient-masked so only the new SID tokens update (the pretrained vocabulary stays frozen, previously it was inadvertently trainable too), and restricted to grounding-only text<->SID pairs instead of a mix of all task types.
grounding_id2name enriched with a short "About the game" snippet alongside name+genres, so the model grounds SIDs against real content, not just a name/genre tag.

Pipeline

Game catalog items -> embedded (Qwen3-0.6B) -> compressed into 4-level semantic IDs via RQ-VAE -> those ID tokens become new vocabulary the LLM is fine-tuned to reason over. Two-stage fine-tuning:

Embedding warmup: only the new semantic-ID token embeddings are trained (codebook-grounded initialization + gradient-masked, grounding-only high-LR run) so the new tokens carry meaningful, linguistically grounded structure before task-specific training begins.
QLoRA (this checkpoint): a real LoRA adapter (rank 8) across all attention/MLP projections, trained on top of stage 1's warmed-up embeddings, via Unsloth.

Both stages run through 4-bit quantization -- full-parameter fine-tuning of a ~4B model doesn't fit a single 12GB consumer GPU alongside gradients/optimizer state.

Tasks

Sequential recommendation: predict the next item's semantic ID from a user's play history
Grounding: map a semantic ID <-> item name/genres(+short description), both directions
Similar item: given an item, suggest another one real users also engaged with
NL preference: open-ended natural-language preference queries ("I want a racing game") -> a matching item's semantic ID
NL similar item: natural-language "recommend something like X" queries, same ground truth as Similar item
ASY (asymmetric item prediction, LC-Rec, arXiv 2311.09049): same history/target pairs as sequential, rendered as the target's name instead of its semantic ID

Evaluation

Recall@K / NDCG@K via constrained beam search (candidates restricted to real catalog items through a trie over valid semantic IDs / descriptions), the same methodology TIGER (Rajput et al. 2023) and LC-Rec use for semantic-ID generative recommenders. n=500 per task (n=71-277 where a task's val split has fewer examples than that), temperature=0.8, beam=10; catalog size ~8,563 items, so random-chance Recall@10 is ~0.12%. grounding_id2name and asy (both name+genres(+blurb)-output tasks) are scored by collapsing each beam candidate and the target down to just the item's plain Name before scoring, so the metric asks "predicted the correct game" rather than requiring an exact match on genre/blurb text too.

Table with columns: Task, v1 Recall@5, v2 Recall@5, v1 Recall@10, v2 Recall@10, v1 NDCG@10, v2 NDCG@10
Task	v1 Recall@5	v2 Recall@5	v1 Recall@10	v2 Recall@10	v1 NDCG@10	v2 NDCG@10
grounding_name2id	2%	32.4%	3%	41.2%	0.012	0.275
grounding_id2name	0%	0.2%	0%	0.2%	0.000

This is a real tradeoff, not a clean win. Grounding (name<->SID) improved dramatically. But sequential/similar_item regressed substantially under strict exact-match Recall@K -- most likely because the cross-task exposure cap cut raw training volume for those tasks by ~40%, trading some ability to memorize the specific recorded co-occurrence pairing for eliminating the popularity- collapse shortcut (v1's tendency to recommend the same few popular items regardless of input).

Qualitative spot-checks (10-15 examples each, decoded to game names) support that read only partly. v2's predictions for sequential/similar_item are often genre/franchise-coherent and fully diverse (no repeated defaults) even when they miss the specific recorded target -- e.g. a racing-game seed predicts another racing game, a Saints Row 2 history predicts Saints Row: The Third. Recall@K can't credit a plausible-but-different answer, so the metric understates v2's real recommendation quality on these tasks to some degree. But a second check complicates the "just needs more training data" story: for a batch of similar_item misses, the specific recorded target's own training exposure was solidly mid-range (6-22 repeats, vs. a dataset median of 12 post-cap) -- not the low-exposure outliers a pure data-starvation explanation would predict. And not every miss is genre-coherent either: a

markdown

Time Mysteries 3

seed (same series as its recorded Time Mysteries 2 target) predicted an unrelated Medieval: Total War, no genre or franchise logic behind it. The more accurate picture: similar_item's ground truth itself is inherently ambiguous (a seed item can have up to 10 valid co-occurring partners in the raw data, e.g. from bundles or sales rather than genre; only one is credited per eval example), so some of the gap is metric-can't-credit-a-different-valid-answer, but some is a genuine specific-recall gap unrelated to how often that particular item was seen. Checkpoint-by-checkpoint Recall@K was flat across the entire back half of training (no late-training improvement), consistent with the model having already reached whatever ceiling this recipe supports rather than needing more steps.

grounding_id2name/asy are the one place a real evaluation bug was found and fixed mid-analysis: constrained beam search defaulted to max_new_tokens=32, plenty for the 6-token semantic-ID outputs but far too small once grounding_id2name's target could include a ~90-token blurb -- generation was being hard-cut before reaching any valid stopping point, so candidates never matched anything, correct or not. Fixing that (raised to 96 for these two tasks) moved grounding_id2name from a measured 0.00% to 0.20% -- genuinely small, not an artifact. The likely real explanation: a semantic ID's coarser RQ-VAE codes are deliberately shared across many similar items (that's what makes the hierarchy useful for generalization elsewhere), so recovering the one specific name behind a shared/coarse code is close to an arbitrary lookup problem, and a rank-8 LoRA adapter over ~8,500 items may not have the capacity to memorize it precisely.

Update (2026-08-10): fair comparison against v3, and a metric correction

Qwen3-8B-Game-semantic-IDs-v3 (bigger model, full-parameter fine-tune, redesigned dataset) initially looked like it regressed on sequential/asy relative to this model's numbers above -- but those numbers were measured on a different, pre-redesign val set with temperature=0.8 sampling, not v3's deterministic eval on the redesigned data. That was never a controlled comparison. Two things were checked to make it one:

Is the semantic-ID vocabulary even comparable? If the RQ-VAE had been retrained for v3, this model's learned token embeddings would encode a different item mapping than v3's, making any comparison invalid outright. Checked directly by joining this model's own training data (8,093 grounding_name2id name->SID pairs) against the current codebook by item name: 8,011/8,093 (99.0%) identical assignments -- the 82 mismatches are ambiguous duplicate catalog names (e.g. "Portal", "Contract"), not a re-embedding. The codebook is unchanged, so a fair comparison is valid.

Re-run under identical conditions. This model was re-evaluated with v3's exact eval code, val set, and settings (n=500/task, seed=0, deterministic beam search):

Table with columns: Task, This model (v2), fair comparison, v3
Task	This model (v2), fair comparison	v3
grounding_name2id	37.00%	96.80%
sequential	3.60%	1.00%
similar_item	1.00%	1.00%
nl_similar_item	4.40%	2.00%
grounding_id2name	0.20%	7.20%
asy	0.00%

The sequential edge over v3 holds up under fair conditions -- not a dataset artifact. But re-scored under a metric that asks "did the model recommend any game this user actually played" instead of the one exact next-lowest-playtime item (justification: these sequences are ordered by playtime, not real chronology -- see v3's README for the full argument), the picture changes substantially:

Table with columns: Task, Metric, This model (v2), v3
Task	Metric	This model (v2)	v3
sequential	Any-played Hit@10	59.20%	58.00%
sequential	Small-library Hit@10 (lift over random)	49.03% (23.03x)	43.24% (20.31x)
asy	Any-played Hit@10	21.80%	36.20%
asy	Small-library Hit@10 (lift over random)	7.54% (2.98x)

sequential is a near-tie once measured this way (the apparent 3.6x gap was mostly an artifact of exact-match Recall@K's sensitivity to an arbitrary playtime-rank position, not a real capability gap). asy is not a tie -- v3 wins clearly (1.7x pooled, 2.9x on the harder small-library stratum), a real improvement the exact-match metric couldn't see because both models scored a flat 0.00% under it. Full writeup, including the codebook-compatibility check and multi-target methodology: SEQUENTIAL_ASY_REGRESSION_ANALYSIS.md in the project repository.

Known limitations

Choosing between v1 and v2 depends on the task you care about. v1 is meaningfully better at sequential/similar_item's strict exact-match recall; v2 is dramatically better at grounding and adds two new natural-language query tasks.
grounding_id2name/asy (SID/history -> name) are weak in both versions, most likely because recovering one specific name from a semantic ID's shared/coarse codes is close to an arbitrary lookup problem -- see above.
Compute-constrained training. QLoRA rank 8, ~0.9 epochs of the fine-tuning stage (wall-clock capped on a single 12GB GPU, not run to convergence).
No classical-recommender baseline (e.g. SASRec) has been run against the same data -- these numbers show the model beats random chance substantially, not that the semantic-ID + LLM approach outperforms a much simpler sequential recommender.
Single training run, single seed, throughout.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

ADAPTER = "pblrvo/Qwen3-4B-Game-semantic-IDs-v2"
BASE_MODEL = "Qwen/Qwen3-4B"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True, llm_int8_skip_modules=["lm_head"],
)
base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, dtype=torch.bfloat16, quantization_config=quantization_config)
base_model.resize_token_embeddings(len(tokenizer))
model = PeftModel.from_pretrained(base_model, ADAPTER)

The semantic-ID vocabulary (<|sid_start|>, <|sid_L{level}_{code}|>, <|sid_end|>) is only meaningful relative to the specific RQ-VAE codebook trained in the source project -- this model isn't usable standalone without that catalog/codebook context.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

pblrvo

Model Tree

Base

Qwen/Qwen3-4B

Adapter

this model

Input Modalities

Text

Output Modalities