Intended use
Research and experimentation with multilingual query/product relevance in
US English, Spanish, and Japanese. This release is a causal-language-model
adapter used as a four-choice decision model, not a separately trained
classification head. Use the supplied scoring code: a generic text-classification
pipeline or chat-generation workflow does not implement this experiment's decision rule.
How inference works
The exact prompt is frozen in src/jev.py. It contains a fixed instruction,
JSON product fields, numbered criteria, and an unfinished Selection: [ prefix.
There is no chat template. The four suffixes are 1], 2], 3], and 4], mapped
in order to E / S / C / I. Each suffix has two tokens with the pinned tokenizer.
The model sums the autoregressive log probabilities of both tokens in each
suffix and applies softmax over the four sums. Argmax selects the label. Training
uses cross-entropy over these candidate scores, rather than ordinary token-level
supervised fine-tuning. Probabilities are conditional on this candidate set and
should not be interpreted as calibrated confidence estimates.
Query, title, brand, color, locale, bullets, and description are accepted. The
prompt plus answer suffix is limited to 256 tokens. Field-aware truncation
removes description/bullets and auxiliary fields before query/title. The exact
prompt, truncation, and candidate scoring are needed to reproduce the results.
Use the adapter
The following downloads this release and runs its bundled frozen scoring code.
The GitHub project provides the pinned Python environment. Install uv first;
the default inference configuration requires a CUDA GPU supporting bf16.
git clone https://github.com/tocab/esci-qwen.gitcd esci-qwenuv sync --locked --extra gpu-kernelsexport HF_HOME="$PWD/.cache/huggingface" .venv/bin/python - <<'PY'from huggingface_hub import snapshot_downloadsnapshot_download("tcabanski/Qwen3.5-0.8B-ESCI-LoRA", local_dir="runs/hf-model")PY .venv/bin/python runs/hf-model/inference.py \ --input runs/hf-model/example.jsonl --output runs/hf-predictions.jsonl
Output contains the E/S/C/I label and all four probabilities. --device cpu
uses float32. Input is UTF-8 JSONL, with a nonempty query and a product_locale
of us, es, or jp; omitted product fields become empty strings. The sample
record is synthetic, not a held-out evaluation example.
For direct PEFT loading, the standard adapter files are at the repository root.
Use AutoModelForImageTextToText for the pinned Qwen base and
PeftModel.from_pretrained(base, adapter_path); run the supplied PromptEncoder
and candidate-scoring code rather than generation. The base revision is
f75361505deaead5ce6a4a25bd2f433cb1cd1ca3.
Training
Table with columns: Setting, Value| Setting | Value |
|---|
| Training / validation examples | 202,359 / 19,458 |
| Dataset | Amazon ESCI Task 2, large_version == 1 |
| Dataset revision | 7916cdf6ab75a462e77f20ab40428a10923998d5 |
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Trainable parameters | 10,822,656 |
| Targets | Language-backbone linear layers, including DeltaNet projections |
| Effective batch | 32 (microbatch 4, accumulation 8) |
| Optimizer / peak LR | AdamW / 1e-4 |
| Schedule | Six-epoch cosine horizon; 3% warmup |
| Precision / hardware | bf16 / one RTX 3090, 24 GB |
| Completed training | 3 epochs, 34.5 training hours; stopped on validation plateau |
| Selected checkpoint | During epoch 2, global update 9,603 |
| Best validation macro F1 | 0.5570318283 |
The base model, vision components, embeddings, and LM output head remain frozen.
The adapter weights are the unchanged best-validation checkpoint, rather than
the final epoch's weights. Hourly saves and roughly six-hour validation enabled
checkpoint selection and early stopping. Validation timing depends on hardware,
so retraining is not guaranteed to be bit-for-bit identical.
Training and validation split connected query-ID / normalized-text groups across
locales. Any training group touching the official test set was quarantined.
Subset sampling used seed 42, 3,000 training groups and 300 validation groups per
locale, including all rows in selected connected groups. Semantic near duplicates
were not exhaustively detected; duplicate/conflicting pairs within partitions
were retained and audited. See the experiment record for the original settings.
Official test results
The adapter was selected using validation only, then evaluated on the complete
638,016-example official test split, spanning US / ES / JP.
Table with columns: Model, Macro F1, Accuracy| Model | Macro F1 | Accuracy |
|---|
| This Qwen 0.8B adapter | 54.69% | 70.05% |
| CLEF Flash 9B | 57.01% | 67.86% |
CLEF leads macro F1 by 2.32 percentage points. Higher Qwen accuracy reflects
its better results on Exact matches, which comprise approximately 62% of test
examples; macro F1 gives each class equal weight.
Table with columns: Class F1, Qwen adapter, CLEF Flash| Class F1 | Qwen adapter | CLEF Flash |
|---|
| Exact | 0.8191 | 0.7973 |
| Substitute | 0.5330 | 0.5784 |
| Complement | 0.3815 | 0.3768 |
| Irrelevant | 0.4538 | 0.5277 |
Table with columns: Locale macro F1, Qwen adapter, CLEF Flash| Locale macro F1 | Qwen adapter | CLEF Flash |
|---|
| US | 0.5380 | 0.5480 |
| ES | 0.5773 | 0.6335 |
| JP | 0.5295 | 0.5793 |
Qwen's query-connected-group bootstrap 95% interval is 0.5426–0.5514
(200 samples, seed 42). Full metrics and the frozen comparison plan are in
comparison.json.
Both models received identical retained product information after the Qwen
checkpoint's field-aware truncation, using their native interfaces. CLEF used
its native joint-schema head, pinned to revision
8b2e5fd17c09fd49bc2880b805d5323ec7ff4ff3.
This is not a reproduction of the published Decision Index benchmark, nor
an evaluation of CLEF with unrestricted product context. CLEF's upstream
training-data overlap was not audited here.
Limitations
The model did not beat CLEF Flash on the primary balanced metric. Complement
classification is weak, and Substitute/Irrelevant performance accounts for much
of the comparison gap. The 256-token budget discards substantial product detail.
Results apply to this dataset, prompt, and label scheme; production latency,
cost, robustness, other languages, and general-domain performance were not
established. Product text may contain misleading instructions despite being
rendered as JSON. Review failure modes before relying on automated decisions.
Release contents and license
This repository contains adapter-only safetensors, pinned adapter metadata,
inference configuration/code, aggregate metrics, a synthetic example, and the
model card. Optimizer state, checkpoint pickle files, base weights, tokenizer
copies, and dataset rows are not distributed. The pinned base tokenizer is
loaded from Qwen's repository.
Released under Apache-2.0; see LICENSE and NOTICE. The upstream
Qwen model and
Amazon ESCI dataset retain their
respective licenses and notices.