papr-ai

papr-embed-code-1-0.6b

README

License: apache-2.0

What it is good at, and what it is not

Good at: code retrieval (natural-language to code, code to code, SQL). See CoIR below.
Not a general-purpose upgrade. On non-code retrieval it is flat to worse than the Qwen/Qwen3-Embedding-0.6B it starts from, with a clear drop on TREC-COVID (table below). If your workload is mostly prose, use the base model.

Usage

python

from sentence_transformers import SentenceTransformermodel = SentenceTransformer("papr-ai/papr-embed-code-1-0.6b")  # loads in fp32q = model.encode(["Instruct: Given a question, retrieve the code that answers it\nQuery:how do I reverse a list in python"], normalize_embeddings=True)d = model.encode(["def rev(xs): return xs[::-1]"], normalize_embeddings=True)print(q @ d.T)

Queries take an instruction in the form Instruct: {instruction}\nQuery:{query}; documents are embedded bare. The instructions behind every number below are in task_prompts.json.

Hosted: POST https://memory.papr.ai/v1/embeddings with "model": "papr-embed-code-1-0.6b".

Evaluation

Run with stock mteb 2.10.0, fp32, max length 8192, on 8x H100 80GB. The full protocol, including every deviation and when it was decided, is in PROTOCOL.md in this repo.

CoIR (nDCG@10). Mean over the 10 tasks: 77.19

Table with columns: Task, nDCG@10, Batch size
Task	nDCG@10	Batch size
Apps	79.31	16
CodeSearchNet	86.19	16
CodeFeedback-MT	94.22	16
CodeFeedback-ST	88.52	16
CodeSearchNet-CC	92.23	16
CodeTransOcean-Contest	90.93	32
CodeTransOcean-DL	34.47	32
CosQA	40.42	32
StackOverflowQA	93.57	16
SyntheticText2SQL	72.04	32

Batch size is mteb's default (32) wherever the task fit in 80 GB in fp32; tasks that ran out of memory were rerun at 16. Scores on most tasks barely move with batch size. SyntheticText2SQL does: the same weights scored between 65 and 72 depending on batching and hardware, because the corpus contains near-identical SQL answers and which one ranks first comes down to floating-point ties. Treat differences of a few points on that task as noise. For reference, stock Qwen3-Embedding-4B scores 68.44 on it in this same environment (its leaderboard entry, from an older mteb, is 78.21).

Held out from training (nDCG@10, fp32).

Table with columns: Task, papr-embed-code-1-0.6b, Qwen3-Embedding-0.6B, Δ
Task	papr-embed-code-1-0.6b	Qwen3-Embedding-0.6B	Δ
ArguAna	66.48	69.34	-2.86
NFCorpus	37.55	36.32	+1.23
SciDocs	24.03	24.39	-0.36
FiQA2018	47.76	47.25	+0.51
TRECCOVID	82.70	89.30	-6.60
CodeEditSearch	66.84	64.25	+2.59
CodeSearchNet	90.97	90.77	+0.20

SciFact: 82.42 vs 69.72 for the base. We trained on the SciFact train split, so this is an in-domain result, not zero-shot.

Training data and contamination

No test split was used anywhere; only train splits entered training. The model saw the train splits of all 10 CoIR tasks, plus the SciFact train split, and its MTEB metadata declares these as training data. CoIR is therefore mostly an in-domain result for this model, not a zero-shot one. In the held-out table, MTEB's CodeSearchNet shares its source with CoIR's CodeSearchNet, whose train split we did use, so read that row as near-domain. CodeEditSearch was not confirmed absent from our training mix either, so read it as indicative.

Limitations

Regresses on non-code retrieval (above). - Running in fp16 costs about 0.8 points. - English and code only; not evaluated on other languages.

Reproduce

replicate_mteb.py in this repo is the script behind every number above. With mteb==2.10.0, torch==2.10.0, transformers==4.57.6, sentence-transformers==5.2.3:

text

hf download papr-ai/papr-embed-code-1-0.6b --local-dir papr-embed-code-1-0.6bpython papr-embed-code-1-0.6b/replicate_mteb.py --model papr-embed-code-1-0.6b --tasks coir --out results --batch 32 --efficient-attn

--efficient-attn only changes which attention kernel runs (same math, ~1e-7 relative). On an 80 GB GPU in fp32, Apps, StackOverflowQA, CodeFeedback and both CodeSearchNet tasks need --batch 16.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

papr-ai

Model Tree

Base

Qwen/Qwen3-Embedding-0.6B

Fine-tuned

this model

Input Modalities

Text

Output Modalities