Model Details
Table | |
|---|
| Developed by | President Information Corp. (統一資訊, PIC) and Uni-President Information Philippines Corp. (UNI-PIPC) |
| Model type | Decoder-only causal language model (hybrid Gated DeltaNet–Gated Attention, as in Qwen3.5) |
| Base model | Qwen/Qwen3.5-4B-Base |
| Adaptation method | Continual pre-training (CPT) |
| Domain | Taiwan retail and e-commerce |
| Languages | Traditional Chinese (zh-TW, primary), Simplified Chinese (zh-CN), English (en) |
| Training framework | ms-swift |
| Training hardware | 4× NVIDIA H100 96 GB |
| License | Apache 2.0 |
Key Features
- 🎯 Retail-Domain Adaptation: Continual pre-training on 2.56B tokens, including 7.30M proprietary Taiwan product listings across 15 retail categories.
- 🇹🇼 Taiwan Retail Knowledge: Synthetic brand and contributor wikis, together with name variants (aliases, foreign-language names, Zhuyin and Pinyin spellings, and typographical errors), generated with NVIDIA NeMo Data Designer.
- 🌐 General-Language Retention: A replay mix of general-domain text (Wikipedia in zh-TW, zh-CN, and en, plus en↔zh translation pairs) preserves general and multilingual language modeling ability.
Training Data
Data Mixture
Token counts are computed with the base model tokenizer. A 0.1% split of each dataset is held out for validation.
Token Mix at a Glance
53.7% · 1.38B
Taiwan Retail Wiki26.3% · 674.4M
Replay Data20.0% · 512.9M
Replay Composition (% of Replay Tokens)
zh-TW Wikipedia40.0%
en↔zh Translation Pairs30.0%
en Wikipedia20.0%
zh-CN Wikipedia10.0%
Product Listings1.38Btokens7,300,108 samples · 53.7%
Taiwan Retail Wiki674.4Mtokens2,762,847 samples · 26.3%
Replay Data512.9Mtokens2,774,330 samples · 20.0%
Total2.56Btokens12,837,285 samples · 100%
Taiwan retail wiki (synthetic, NVIDIA NeMo Data Designer): brand and contributor wikis, plus name variants (aliases, foreign-language names, Simplified/Traditional script, Zhuyin and Pinyin spellings, typos).
Replay: general-domain text mixed into CPT to mitigate catastrophic forgetting. Sources: wikimedia/wikipedia (zh-TW, zh-CN, en) and en↔zh translation pairs from wmt/wmt19.
zh-TW = Chinese converted to Traditional Chinese with OpenCC (translation pairs: Chinese side converted to zh-TW).
Product Listings
The product listings are proprietary Taiwan retail data comprising title, brand, category, price, and description. The data was cleaned with NVIDIA NeMo Curator, which removed SEO filler text, emojis, contact details, hyperlinks, and policy terms, followed by exact and fuzzy deduplication.
Category distribution (15 categories, share of listings)
Evaluation
Evaluation Setup
The CPT model is compared against its base model, Qwen/Qwen3.5-4B-Base. Statistical significance is assessed with a 95% paired bootstrap over items (1,000 resamples):
🟢 significant improvement · 🔴 significant degradation · ⚪ no significant difference
Language Modeling
Metric: Bits per byte (BPB); lower is better.
Held-Out Domain Data
1.3920.359−1.032
Retail Wiki (Brand & Contributor)1.0110.574−0.437
Name Variants (Brand & Contributor)1.1430.348−0.795
General-Domain Replay Data
0.9290.854−0.074
Chinese (Simplified)0.9310.900−0.031
English0.6840.647−0.038
English–Chinese Translation0.8100.541−0.270
Knowledge Probes
Metric: PMI-normalized accuracy, which adjusts for the prior likelihood of each option; higher is better.
Setup: Zero-shot, five-option multiple choice (chance level: 20%). Each option is scored by its log-likelihood as a continuation of the prompt.
Data: Internal probe sets derived from the retail training data. Alias and wiki-fact labels are LLM-derived. Scores are not comparable with results reported for other models.
Summary by Probe Set
50.5%64.9%+14.4%
Retail Wiki Knowledge54.6%65.6%+10.9%
Name Variants (Brand & Contributor)78.6%83.6%+5.0%
Product and Work Knowledge
These probes test knowledge about products and about the works (books, music, and other media) that products represent.
61.4%69.9%+8.5%
Product Title → Brand (Brand Masked)31.6%34.5%+2.9%
Product → Category64.7%84.3%+19.7%
Brand → Main Category41.4%65.0%+23.6%
Work → Contributor53.7%70.8%+17.2%
Retail Wiki Knowledge
A contributor is a credited person or organization on a product or work, such as an author, singer, artist, illustrator, translator, editor, or publisher. Brand origin is the country or region where the brand was founded.
48.2%51.9%+3.7%
Brand: Founding Year57.4%66.5%+9.1%
Brand: Origin64.6%74.0%+9.4%
Brand: Parent Company60.6%71.5%+10.9%
Contributor: Nationality76.8%88.5%+11.8%
Contributor: Occupation53.0%64.5%+11.5%
Contributor: Credit Role21.8%42.0%+20.3%
Name Variants
A name variant is an alternative form of a brand or contributor name: a foreign-language alias, a different script, a phonetic spelling, or a typographical error. The canonical name is the standard form of the name.
Query Normalization (Variant → Canonical Name). Given a noisy variant of a brand or contributor name, the model selects the canonical name. A typographical error is either a Latin-letter typo or a same-sound Chinese character substitution.
66.7%70.9%+4.2%
Typographical Error90.2%86.3%−3.9%
Simplified Chinese Script98.7%91.0%−7.7%
Zhuyin (Bopomofo)66.1%69.8%+3.7%
Zhuyin Keystroke Sequence20.6%19.6%−1.0%
Pinyin (Toneless)92.4%84.3%−8.1%
Pinyin (Tone Numbers)89.5%80.4%−9.2%
Variant Generation (Canonical Name → Variant). Given a canonical brand or contributor name, the model selects the correct variant.
79.6%90.0%+10.4%
Simplified Chinese Script100.0%100.0%+0.0%
Zhuyin (Bopomofo)98.9%100.0%+1.1%
Zhuyin Keystroke Sequence20.4%95.0%+74.6%
Pinyin (Toneless)99.6%100.0%+0.4%
Pinyin (Tone Numbers)99.6%99.9%+0.3%
Citation
@misc{retailtwqwen354bbase, title = {Retail-TW-Qwen3.5-4B-Base}, author = {President Information Corp. (PIC) and Uni-President Information Philippines Corp. (UNI-PIPC)}, year = {2026}, publisher = {Hugging Face}, url = {https://huggingface.co/UNI-PIC/Retail-TW-Qwen3.5-4B-Base}}