laskar-ks

qwen-1.5b-product-category-classifier

README

License: apache-2.0

Results

Table with columns: Method, Accuracy
Method	Accuracy
Random (1 of ~8)	12.0%
Embedding cosine (`multilingual-e5-small`)	35.8%
Qwen2.5-1.5B zero-shot	34.3%
Qwen2.5-1.5B + LoRA	66.0% ± 0.9%

95% CI on n=9,652. Differences below ~1.9 points are not distinguishable from noise.

The zero-shot baseline had a 0.0% parse failure rate, so the entire 31.7-point gain is accuracy — none of it comes from the model learning to follow the output format.

Both the fine-tuned model and the zero-shot baseline are evaluated with constrained scoring: the logits for digit tokens 1–n are compared directly, so an out-of-range answer is structurally impossible for either.

Accuracy by difficulty

difficulty = how many of the candidates share the same top-level category as the correct answer. 1 = the answer stands alone; 9 = all candidates are siblings.

Table with columns: difficulty, n, zero-shot, LoRA, delta
difficulty	n	zero-shot	LoRA	delta
1	998	29.6%	83.1%	+53.5
2	651	30.4%	70.0%	+39.6
3	655	30.7%	68.5%	+37.9
4

The largest gain is at difficulty 1, which is counterintuitive — those should be the easiest cases. The likely explanation is that these rows encode Shopify's internal taxonomy conventions rather than semantic similarity. For example, a Dutch scrub brush (Alpina schrobborstel) is labelled Shaving Brushes even though Scrub Brushes is among the candidates. That is not noise: it is a consistent, non-obvious convention, and the model learns it. Embedding similarity cannot.

Effect of product descriptions

Table with columns: n, zero-shot, LoRA
	n	zero-shot	LoRA
Description present (≥20 chars)	8,861	34.7%	66.4%
Description absent	791	30.1%	61.1%

Titles carry most of the signal. Catalogues with missing or poor descriptions lose about 5 points, not the task.

Generalization to unseen categories

Three ablation models, all with hyperparameters identical to the main run. Only the training data differs. Runs B2 and C are evaluated on the same 1,494-row slice.

Table with columns: Run, Training data, Held-out categories seen?, Accuracy
Run	Training data	Held-out categories seen?	Accuracy
A	38,511 (100%)	yes	66.0%
C	10,537 (27%)	yes	60.0% ± 2.5%
B2	10,537 (27%)	never	59.1% ± 2.5%
B	~33,000 (86%)	only as wrong answers	14.9% ± 2.2%

C vs B2 — the main finding. These two differ by 0.9 points, well inside the margin of error. Having seen a category during training gives essentially no advantage. The entire drop from 66.0% to 59.1% is explained by training on 27% of the data, not by facing categories the model had never encountered.

The model is not memorising a category list. It learns how to read a product and match it against taxonomy structure, and that ability transfers intact to leaves absent from training. Practically: Shopify's taxonomy grows, and this model handles new leaves without retraining.

B vs B2 — the interesting failure. Removing a category from the answers while leaving it among the candidates is not an ablation. In Run B, 15% of training rows still contained the held-out categories as distractors, so the model saw them thousands of times and always as the wrong choice. It learned an active negative bias: those categories scored 17 points below the untuned base model.

Takeaway for anyone training a reranker: candidate composition in the training data matters as much as label distribution. Ignorance is far better than a learned negative bias.

Caveat: Run B used a different held-out set (701 leaves, n=1,008) than B2 and C (1,566 leaves, n=1,494), so its magnitude is not directly comparable. The 44-point gap far exceeds any plausible set-to-set variation, but the sets are not identical.

Inference cost

Benchmarked on A100 40GB, bfloat16, median prompt 368 tokens, constrained scoring (one forward pass per product, no generation loop).

Table with columns: Metric, Value
Metric	Value
Best throughput	47.5 products/sec (batch 8)
Time per 1,000 products	21.1 sec
Cost per 1,000 products	$0.0084
100,000 products	$0.84, ~35 minutes

At $1.43/hr for an A100 40GB (Spheron, 2026-08-23).

Comparison to a hosted API, with an important limitation. At $0.15 p er 1 M in p u tt o k e n s, t h es am e 1, 000 p r o d u c t s w o u l d cos t ab o u t$ 0.0736 in input tokens alone — roughly 8.8× more.

This compares cost, not accuracy. No API model was evaluated on this test set. A larger hosted model would very likely score above 66.0%. The case for a local 1.5B model rests on cost, data residency, and throughput predictability — not on matching frontier models on accuracy. Anyone choosing between the two should measure both.

Two things that do not appear in the cost table but often decide the question:

Catalogue data never leaves your infrastructure. Product naming and category structure are competitive assets.
No rate limits. Batching 100,000 products through an API means retries and queueing; on your own GPU, throughput is bounded only by hardware.

Usage

python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
from huggingface_hub import snapshot_download

BASE = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
tok.padding_side = "left"
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

base = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="cuda"
)
path = snapshot_download("laskar-ks/product-category-classifier-qwen-1.5b")
model = PeftModel.from_pretrained(base, path).eval()

SYSTEM = (
    "You are a product taxonomy classifier. "
    "Given a product and a numbered list of candidate categories, "
    "reply with ONLY the number of the correct category. No explanation."
)

def classify(title, description, candidates):
    listing = "\n".join(f"{i+1}. {c}" for i, c in enumerate(candidates))
    user = (
        f"Product title: {title}\n"
        f"Description: {description or '(no description)'}\n\n"
        f"Candidate categories:\n{listing}\n\n"
        f"Answer with the number only (1-{len(candidates)}):"
    )
    prompt = tok.apply_chat_template(
        [{"role": "system", "content": SYSTEM},
         {"role": "user", "content": user}],
        tokenize=False, add_generation_prompt=True,
    )
    enc = tok(prompt, return_tensors="pt").to("cuda")

    with torch.no_grad():
        logits = model(**enc).logits[0, -1, :]

    # Constrained scoring: only digits 1..n are considered
    digit_ids = [tok(str(i + 1), add_special_tokens=False)["input_ids"][0]
                 for i in range(len(candidates))]
    return candidates[int(torch.argmax(logits[digit_ids]).item())]


print(classify(
    "Alpina schrobborstel | 2 stuks",
    "Ruige schrobborstel voor hardnekkige schoonmaak. Materiaal: kunststof.",
    ["Home & Garden > Household Supplies > Household Cleaning Supplies > Scrub Brushes",
     "Health & Beauty > Personal Care > Shaving & Grooming > Shaving Brushes"],
))

The model expects candidates to come from a retrieval step. Feeding it arbitrary or unrelated candidate lists is outside what it was trained for.

Training

Table

Base model	Qwen/Qwen2.5-1.5B-Instruct
Method	LoRA, r=16, alpha=32, dropout=0.05
Target modules	q/k/v/o_proj, gate/up/down_proj
Trainable params	18,464,768 (1.18%)
Epochs	1
Effective batch	16 (4 × grad accum 4)
Learning rate	2e-4, cosine, 75 warmup steps
Precision	bfloat16, gradient checkpointing on
Hardware

Candidate order is randomly permuted during both training and evaluation, so the model cannot learn a positional shortcut.

Loss is computed only on the answer token (prompt tokens masked with -100). Without this, LoRA capacity is spent memorising prompt text that is always supplied anyway.

Data preparation

Derived from Shopify/product-catalogue (Apache 2.0). Official train/test splits preserved.

Dropped product_image (text-only task)
Removed rows with an empty title, or where ground_truth_category was absent from potential_product_categories
Deduplicated on product_title within each split (81 removed from train, 6 from test)
39 titles appeared in both the official train and test splits. Removed from train only; the test split was left untouched so results stay comparable to other work on this dataset
Truncated titles to 200 chars, descriptions to 1,000 chars (one description ran to 46,773 chars)
Added difficulty and has_desc columns for the analyses above

Final: 38,511 train / 9,652 test.

Reproducibility

Hardware: A100 40GB (Colab Pro). Seed 42 throughout.

Package versions matter here — the HF ecosystem moved quickly around this release and several combinations fail:

markdown
torch 2.11.0+cu128
transformers 5.15.0
peft 0.13.2

Known issue. peft==0.13.2 calls hf_hub_download(use_auth_token=...), which newer huggingface_hub releases removed. Loading the adapter directly by repo ID raises a misleading Can't find 'adapter_config.json' error. Download first, then load from the local path (as in the usage example above), or pin huggingface_hub<0.26.

transformers 5.x also removed warmup_ratio from TrainingArguments; warmup_steps works on both 4.x and 5.x.

Limitations

Rerank only. Requires a retrieval step to produce 8–9 candidates. It does not classify into the full Shopify taxonomy from scratch.
Cost comparison is not an accuracy comparison. No API model was benchmarked on this test set.
Ceiling unknown. The dataset's own labels were not audited. Some appear to encode non-obvious conventions rather than errors, but no manual audit was run, so the maximum achievable accuracy on this test set is not established.
English-dominant. The training data includes Spanish, Dutch, Russian, and German products, but the distribution was not measured and per-language accuracy was not broken out.
Single epoch, no validation set. Training loss was still declining at the end of epoch 1. Whether a second epoch helps was not tested, because no validation split was held out to detect overfitting.

License

Apache 2.0, matching the source dataset.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.