anicka

qwen3-embedding-8b-tibetan-lora

README

License: apache-2.0

Results — dictionary headword retrieval (held-out terms)

R@1 over the full 26,787-headword haystack. Held-out terms (300 Sanskrit

500 English) contributed zero training pairs.

Table with columns: axis, base R@1, +LoRA R@1, +LoRA R@5
axis	base R@1	+LoRA R@1	+LoRA R@5
sanskrit → wylie	0.044	0.264	0.447
sanskrit → uchen	0.100	0.261	0.419
english → wylie	0.004	0.186	0.328
english → uchen	0.030	0.144	0.263
wylie → uchen	0.430	0.927	0.990

Results — passage retrieval (real practice-text corpus)

58 gold queries (14 dharma terms × up to 4 transliteration schemes + 4 end-to-end questions) against 32,607 passages (gold targets + 30k distractors) from a 866-document practice-text collection. hit@5 = any gold passage in top 5; cross-hit masks out every passage that contains the query string literally — the score string-matching cannot reach.

Table with columns: scheme, base hit@5, base cross, +LoRA hit@5, +LoRA cross
scheme	base hit@5	base cross	+LoRA hit@5	+LoRA cross
wylie	10/14	4/14	13/14	11/14
phonetic	4/14	2/14	12/14	10/14
sanskrit	7/12	2/12	4/12	2/12

Read this table honestly: it is a specialization trade-off, measured on small n (12-14 queries per scheme). Tibetan-side retrieval jumps — cross-hit (the case alias tables exist for) goes 4/14→11/14 on Wylie and 2/14→10/14 on phonetic. Phonetic renderings were learned transitively: the dictionary contains no phonetics at all, yet "tongpa nyi" now finds stong pa nyid passages. The price: general-semantic English retrieval regresses (5/14→2/14, end-to-end questions 0/4) — the LoRA pulls English glosses toward Tibetan headword space and away from general passage space. Dictionary-level English→headword improved (0.004→0.186), so the regression is specific to long-passage semantic matching.

Deployment recommendation: route queries. Adapter ON for Tibetan-script/Wylie/phonetic/Sanskrit-term queries, adapter OFF (base model) for general English questions — with PEFT this is enable_adapter_layers()/disable_adapter_layers() per query, no extra memory. In a hybrid stack (FTS + vector + reranker) the English regression is further absorbed by the lexical side.

Usage

Exactly like base Qwen3-Embedding-8B (last-token pooling, L2-normalize, instruction prefix on queries only) with the adapter attached:

python
from transformers import AutoModel, AutoTokenizer
from peft import PeftModel
import torch, torch.nn.functional as F

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-Embedding-8B", padding_side="right")
model = AutoModel.from_pretrained("Qwen/Qwen3-Embedding-8B", torch_dtype=torch.bfloat16).cuda()
model = PeftModel.from_pretrained(model, "anicka/qwen3-embedding-8b-tibetan-lora").eval()

INSTRUCT = ("Instruct: Given a term in any transliteration scheme "
            "(Wylie, Tibetan script, Sanskrit, English), retrieve the "
            "matching Tibetan dictionary headword\nQuery: ")

def embed(texts, prefix=""):
    enc = tok([prefix + t for t in texts], padding=True, truncation=True,
              max_length=256, return_tensors="pt").to("cuda")
    h = model(**enc).last_hidden_state
    idx = enc.attention_mask.sum(1) - 1
    v = h[torch.arange(h.size(0)), idx]
    return F.normalize(v.float(), dim=-1)

q = embed(["tongpa nyi"], prefix=INSTRUCT)          # phonetic query
d = embed(["stong pa nyid", "སྟོང་པ་ཉིད་", "śūnyatā"])  # any scheme
print(q @ d.T)

Training

Data: 95,135 contrastive pairs from 26,787 dictionary entries: wylie↔uchen (26k), english-gloss→wylie (36k), form→definition (22k), abbreviation/misspelling→canonical (3.6k), sanskrit→wylie/uchen (4k), verb-stem→lemma (3.2k). Held-out eval terms excluded from all pair types.
Objective: symmetric InfoNCE, in-batch negatives, temperature 0.05. Length-bucketed batches (terms vs definitions) — term batches give all-term in-batch negatives, which are the hard ones.
LoRA: r=16, α=32, all attention + MLP projections, 1 epoch (best val at end, no overfit), lr 1e-4 cosine, batch 48, bf16 + gradient checkpointing.
Trained with the query instruction prefix on anchors only, matching inference usage.

Limitations

General-English and question-style retrieval regress (see the passage table above) — route those to the base model. The natural v2 fix is mixing general-retrieval pairs into training to pin the English side in place.
English glosses collide ("wisdom" maps to shes rab, ye shes, …); english-axis R@1 undercounts real quality — check R@5.
Trained on dharma vocabulary; no claim about secular modern Tibetan.
Passage-eval n is small (12-14 queries/scheme); treat those fractions as strong signal with wide error bars, not precise rates.

Acknowledgements

The Illuminator Tibetan-English Encyclopaedic Dictionary (Lotsawa Tony Duff, Padma Karpo Translation Committee) — used under license as training data. Motivated by a Buddhist-RAG builder's observation that no neural embedder handles Tibetan transliteration schemes.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

anicka

Model Tree

Base

Qwen/Qwen3-Embedding-8B

Adapter

this model

Input Modalities

Text

Output Modalities