pblrvo

Qwen3-4B-Game-semantic-IDs

README

License: apache-2.0

Pipeline

Game catalog items -> embedded (Qwen3-0.6B) -> compressed into 4-level semantic IDs via RQ-VAE -> those ID tokens become new vocabulary the LLM is fine-tuned to reason over. Two-stage fine-tuning:

Embedding warmup: only embed_tokens/lm_head trained (codebook-grounded initialization + short high-LR run) so the new ID tokens carry meaningful structure before task-specific training begins.
QLoRA (this checkpoint): a real LoRA adapter (rank 8) across all attention/MLP projections, trained on top of stage 1's warmed-up embeddings, via Unsloth.

Both stages run through 4-bit quantization -- full-parameter fine-tuning of a ~4B model doesn't fit a single 12GB consumer GPU alongside gradients/optimizer state.

Tasks

Sequential recommendation: predict the next item's semantic ID from a user's play history
Grounding: map a semantic ID <-> item name/genres, both directions
Similar item: given an item, suggest another one real users also engaged with
ASY (asymmetric item prediction, LC-Rec, arXiv 2311.09049): same history/target pairs as sequential, rendered as the target's name instead of its semantic ID

Evaluation

Recall@K / NDCG@K via constrained beam search (candidates restricted to real catalog items through a trie over valid semantic IDs / descriptions), the same methodology TIGER (Rajput et al. 2023) and LC-Rec use for semantic-ID generative recommenders -- a single greedy decode is a top-1 prediction, not a ranking, so beam search stands in for the ranking step a classic recommender gets for free from a dot-product over all items.

Table with columns: Task, Recall@5, NDCG@5, Recall@10, NDCG@10
Task	Recall@5	NDCG@5	Recall@10	NDCG@10
sequential	12%	0.091	13%	0.094
similar_item	10%	0.072	13%	0.082
grounding_name2id	2%	0.009	3%	0.012
grounding_id2name

(n=100 per task, temperature=0.8, beam=10; catalog size ~8,563 items, so random-chance Recall@10 is ~0.12% -- the recommendation-shaped tasks land well above chance, the two grounding directions remain a clear, unresolved weak point.)

Known limitations

Grounding (exact ID<->name lookup) is weak. grounding_id2name produces plausible, valid catalog descriptions (~85-90% valid-format) but essentially never the specific correct one, even given a 10-candidate retry budget. Unlike the relational tasks (sequential/similar item), the ID<->name mapping has no exploitable structure to generalize from -- it's closer to an arbitrary lookup table than a learnable pattern, and a rank-8 LoRA adapter may simply not have enough capacity to memorize it precisely for ~8,500 items. Some of this may also be a property of the semantic ID space itself: items that collide on their coarser RQ-VAE codes are only disambiguated by a tiebreaker digit that carries no learnable signal connecting it to the item's name.
Compute-constrained training. QLoRA rank 8, ~0.74 epochs of the fine-tuning stage (wall-clock capped on a single 12GB GPU, not run to convergence). Results should be read as "what's achievable under this specific hardware budget," not a ceiling on the approach.
No classical-recommender baseline (e.g. SASRec) has been run against the same data -- these numbers show the model beats random chance substantially, not that the semantic-ID + LLM approach outperforms a much simpler sequential recommender.
Single training run, single seed, throughout.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

ADAPTER = "pblrvo/Qwen3-4B-Game-semantic-IDs"
BASE_MODEL = "Qwen/Qwen3-4B"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True, llm_int8_skip_modules=["lm_head"],
)
base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, dtype=torch.bfloat16, quantization_config=quantization_config)
base_model.resize_token_embeddings(len(tokenizer))
model = PeftModel.from_pretrained(base_model, ADAPTER)

The semantic-ID vocabulary (<|sid_start|>, <|sid_L{level}_{code}|>, <|sid_end|>) is only meaningful relative to the specific RQ-VAE codebook trained in the source project -- this model isn't usable standalone without that catalog/codebook context.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

pblrvo

Model Tree

Base

Qwen/Qwen3-4B

Adapter

this model

Input Modalities

Text

Output Modalities