MoAI-Embedding-0.6B-LoRA API & Inference Endpoint

1.1. TL;DR

Base model: Qwen/Qwen3-Embedding-0.6B — 28 layers, hidden 1024, last-token pooling, instruction-aware
Domain / Language: Finance (BC Card — personal / merchant / corporate / VIP) / Korean
Task: Query-document retrieval (QA search, document similarity), RAG retriever
Method: PEFT (LoRA) + Multiple Negatives Ranking (contrastive)
Embedding dimension: 1024 · Max sequence length: 1024 · Similarity: cosine (outputs are L2-normalized)
Intended use
- In-house BC Card-domain RAG retriever (Top-K candidate retrieval)
- QA search, document-similarity scoring

1.2. Usage

Install sentence-transformers and peft (required to apply the LoRA adapter); loading also downloads the base model Qwen/Qwen3-Embedding-0.6B on first use.

bash

pip install -U sentence-transformers peft

Queries use an instruction prompt; documents use none (matching how the model was trained).

python

from sentence_transformers import SentenceTransformer model = SentenceTransformer("BCCard/MoAI-Embedding-0.6B") queries = ["BC카드 연회비는 어떻게 되나요?"]documents = [    "BC카드 연회비는 카드 종류와 혜택 구성에 따라 다르게 책정됩니다 ...",    "카드 분실 신고는 고객센터 또는 앱에서 즉시 가능합니다 ...",] # `prompt_name` selects the prompt stored in the model configq_emb = model.encode(queries, prompt_name="query")        # query instruction auto-appliedd_emb = model.encode(documents, prompt_name="document")   # document side (no instruction) scores = model.similarity(q_emb, d_emb)   # cosine; rank documents by scoreprint(scores)

Query prompt (instruction): Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:
Document prompt: none

1.3. Training Data

Table with columns: Dataset, Role, Size
Dataset	Role	Size
BCAI-Finance-Kor-Embedding-Triplet	Training (anchor / positive / negative)	43,394 triplets (train)
BCAI-Finance-Kor-Embedding-Pair	Corpus pool / evaluation	36,281 unique chunks

Sources: BC Card financial QA (BCAI) + website crawl + synthetic data (chunking + multi-query generation)
Triplets are constructed via hard-negative mining over the unified corpus.

1.4. Training Procedure

Table with columns: Item, Value
Item	Value
Method	LoRA (PEFT)
LoRA	r=64, alpha=128, dropout=0.05, targets = q,k,v,o,gate,up,down_proj
Loss	CachedMultipleNegativesRankingLoss (in-batch negatives)
Batch	per-device 256 (DDP) → 511 in-batch negatives per rank
LR / scheduler	1e-4 / cosine, warmup_ratio 0.1, weight_decay 0.01
Epochs	3, early stopping — best checkpoint selected by validation NDCG@10
Precision	bf16, gradient checkpointing
Hardware	6× NVIDIA L40S (DDP)

2. Evaluation

2.1. Training

Trained for 3 epochs (early-stopped) with a cosine schedule; training loss decreases steadily while validation NDCG@10 climbs early and plateaus, and the best checkpoint is selected at the peak. Curves (loss / learning rate / validation NDCG@10) are logged to Weights & Biases.

2.2. In-domain Retrieval Benchmark

(1) Setup

Queries: 1,000 (held-out test split) · Corpus: 36,281 unique chunks
Protocol: binary-relevance information retrieval; the same evaluator used during training
Metrics: NDCG@10 (primary), MRR@10, Recall@{1,10}, Accuracy@1, MAP@10
Models compared: base (Qwen3-Embedding-0.6B, no fine-tuning) vs. v1 (r32 / lr2e-4 / 4ep) vs. v2 (r64 / lr1e-4 / 3ep, released)

(2) Test set

Table with columns: Metric, base (Qwen3-0.6B), v1 (r32/2e-4/4ep), v2 (r64/1e-4/3ep), v2 Δ vs base
Metric	base (Qwen3-0.6B)	v1 (r32/2e-4/4ep)	v2 (r64/1e-4/3ep)	v2 Δ vs base
NDCG@10	0.6186	0.6665	0.6695	+0.051 (+8.2%)
MRR@10	0.6449	0.6993	0.7060	+0.061 (+9.5%)
Recall@10	0.7046	0.7512	0.7508

v2 is the released model (best across all metrics; Recall@10 is on par with v1). Fine-tuning lifts in-domain retrieval by roughly +10% over the base model, with the largest gains on top-rank precision (Accuracy@1, Recall@1).

2.3. Limitations

Domain-specific — tuned for BC Card Korean financial text; out-of-domain or non-Korean performance is not guaranteed.
Re-ranking recommended — as a 0.6B bi-encoder, it favors recall/throughput over fine-grained precision.
- Recommended pipeline: Bi-Encoder (this model) Top-K → Cross-Encoder re-ranking.
Sequence length — inputs are truncated at 1,024 tokens; content past that limit is not encoded, so very long documents should be chunked before indexing.
Exact-value matching — fine-grained numeric/tabular facts (fees, rates, dates, terms) are not reliably distinguished by dense similarity alone; pair with lexical (BM25) retrieval or a re-ranker when exactness matters.
Retrieval only — this is an embedding model, not a generator; it ranks passages and does not produce answers.
Synthetic data influence — part of the training set is LLM-synthesized (chunking + multi-query), which may carry the generator's stylistic/coverage biases.
PII — personal/card information was masked during preprocessing, but the model performs no PII protection at inference; apply your own masking/filtering on inputs and outputs.

3. Future Work

Data quality improvement & re-training
- Human-annotation labeling
- More rigorous hard-negative mining (iterative, mined with this model)
- Broader/higher-quality data (incl. general financial corpora)
System-level
- Cross-Encoder re-ranker for precision
- HyDE / dynamic instruction injection at query time

4. Meta Info

4.1. Citation

bibtex

@misc{bccard2026moaiembedding,  title        = {MoAI-Embedding-0.6B: A BC Card-Domain Korean Text Embedding Model},  author       = {BC Card},  year         = {2026},  howpublished = {\url{https://huggingface.co/BCCard/MoAI-Embedding-0.6B}},  note         = {LoRA fine-tune of Qwen3-Embedding-0.6B for BC Card-domain Korean retrieval}}

Corpus dataset: BCCard/BCAI-Finance-Kor-Embedding-Pair

1.1. TL;DR

Base model: Qwen/Qwen3-Embedding-0.6B — 28 layers, hidden 1024, last-token pooling, instruction-aware
Domain / Language: Finance (BC Card — personal / merchant / corporate / VIP) / Korean
Task: Query-document retrieval (QA search, document similarity), RAG retriever
Method: PEFT (LoRA) + Multiple Negatives Ranking (contrastive)
Embedding dimension: 1024 · Max sequence length: 1024 · Similarity: cosine (outputs are L2-normalized)
Intended use
- In-house BC Card-domain RAG retriever (Top-K candidate retrieval)
- QA search, document-similarity scoring

1.2. Usage

Install sentence-transformers and peft (required to apply the LoRA adapter); loading also downloads the base model Qwen/Qwen3-Embedding-0.6B on first use.

bash

pip install -U sentence-transformers peft

Queries use an instruction prompt; documents use none (matching how the model was trained).

python

from sentence_transformers import SentenceTransformer model = SentenceTransformer("BCCard/MoAI-Embedding-0.6B") queries = ["BC카드 연회비는 어떻게 되나요?"]documents = [    "BC카드 연회비는 카드 종류와 혜택 구성에 따라 다르게 책정됩니다 ...",    "카드 분실 신고는 고객센터 또는 앱에서 즉시 가능합니다 ...",] # `prompt_name` selects the prompt stored in the model configq_emb = model.encode(queries, prompt_name="query")        # query instruction auto-appliedd_emb = model.encode(documents, prompt_name="document")   # document side (no instruction) scores = model.similarity(q_emb, d_emb)   # cosine; rank documents by scoreprint(scores)

Query prompt (instruction): Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:
Document prompt: none

1.3. Training Data

Table with columns: Dataset, Role, Size
Dataset	Role	Size
BCAI-Finance-Kor-Embedding-Triplet	Training (anchor / positive / negative)	43,394 triplets (train)
BCAI-Finance-Kor-Embedding-Pair	Corpus pool / evaluation	36,281 unique chunks

Sources: BC Card financial QA (BCAI) + website crawl + synthetic data (chunking + multi-query generation)
Triplets are constructed via hard-negative mining over the unified corpus.

1.4. Training Procedure

Table with columns: Item, Value
Item	Value
Method	LoRA (PEFT)
LoRA	r=64, alpha=128, dropout=0.05, targets = q,k,v,o,gate,up,down_proj
Loss	CachedMultipleNegativesRankingLoss (in-batch negatives)
Batch	per-device 256 (DDP) → 511 in-batch negatives per rank
LR / scheduler	1e-4 / cosine, warmup_ratio 0.1, weight_decay 0.01
Epochs	3, early stopping — best checkpoint selected by validation NDCG@10
Precision	bf16, gradient checkpointing
Hardware	6× NVIDIA L40S (DDP)

2. Evaluation

2.1. Training

2.2. In-domain Retrieval Benchmark

(1) Setup

Queries: 1,000 (held-out test split) · Corpus: 36,281 unique chunks
Protocol: binary-relevance information retrieval; the same evaluator used during training
Metrics: NDCG@10 (primary), MRR@10, Recall@{1,10}, Accuracy@1, MAP@10
Models compared: base (Qwen3-Embedding-0.6B, no fine-tuning) vs. v1 (r32 / lr2e-4 / 4ep) vs. v2 (r64 / lr1e-4 / 3ep, released)

(2) Test set

Table with columns: Metric, base (Qwen3-0.6B), v1 (r32/2e-4/4ep), v2 (r64/1e-4/3ep), v2 Δ vs base
Metric	base (Qwen3-0.6B)	v1 (r32/2e-4/4ep)	v2 (r64/1e-4/3ep)	v2 Δ vs base
NDCG@10	0.6186	0.6665	0.6695	+0.051 (+8.2%)
MRR@10	0.6449	0.6993	0.7060	+0.061 (+9.5%)
Recall@10	0.7046	0.7512	0.7508

2.3. Limitations

Domain-specific — tuned for BC Card Korean financial text; out-of-domain or non-Korean performance is not guaranteed.
Re-ranking recommended — as a 0.6B bi-encoder, it favors recall/throughput over fine-grained precision.
- Recommended pipeline: Bi-Encoder (this model) Top-K → Cross-Encoder re-ranking.
Sequence length — inputs are truncated at 1,024 tokens; content past that limit is not encoded, so very long documents should be chunked before indexing.
Exact-value matching — fine-grained numeric/tabular facts (fees, rates, dates, terms) are not reliably distinguished by dense similarity alone; pair with lexical (BM25) retrieval or a re-ranker when exactness matters.
Retrieval only — this is an embedding model, not a generator; it ranks passages and does not produce answers.
Synthetic data influence — part of the training set is LLM-synthesized (chunking + multi-query), which may carry the generator's stylistic/coverage biases.
PII — personal/card information was masked during preprocessing, but the model performs no PII protection at inference; apply your own masking/filtering on inputs and outputs.

3. Future Work

Data quality improvement & re-training
- Human-annotation labeling
- More rigorous hard-negative mining (iterative, mined with this model)
- Broader/higher-quality data (incl. general financial corpora)
System-level
- Cross-Encoder re-ranker for precision
- HyDE / dynamic instruction injection at query time

4. Meta Info

4.1. Citation

bibtex

@misc{bccard2026moaiembedding,  title        = {MoAI-Embedding-0.6B: A BC Card-Domain Korean Text Embedding Model},  author       = {BC Card},  year         = {2026},  howpublished = {\url{https://huggingface.co/BCCard/MoAI-Embedding-0.6B}},  note         = {LoRA fine-tune of Qwen3-Embedding-0.6B for BC Card-domain Korean retrieval}}

Corpus dataset: BCCard/BCAI-Finance-Kor-Embedding-Pair

MoAI-Embedding-0.6B-LoRA

README

1.1. TL;DR

1.2. Usage

1.3. Training Data

1.4. Training Procedure

2. Evaluation

2.1. Training

2.2. In-domain Retrieval Benchmark

(1) Setup

(2) Test set

2.3. Limitations

3. Future Work

4. Meta Info

4.1. Citation

Explore FriendliAI today

README

1.1. TL;DR

1.2. Usage

1.3. Training Data

1.4. Training Procedure

2. Evaluation

2.1. Training

2.2. In-domain Retrieval Benchmark

(1) Setup

(2) Test set

2.3. Limitations

3. Future Work

4. Meta Info

4.1. Citation