aozalevsky

search_slm_demo

README

License: apache-2.0

Quick start

Serve with vLLM (the speculative-decoding flag is optional; it uses the model's own multi-token-prediction head and roughly halves latency without changing greedy outputs):

bash

# on a 16 GB GPU also add: --max-num-seqs 8vllm serve aozalevsky/search_slm_demo --max-model-len 8192 \  --limit-mm-per-prompt '{"image":0,"video":0}' \  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

The model was trained with one fixed system prompt (with today's date, for "last 5 years" style requests) and with thinking turned off. Use exactly this format:

python

import datetimeimport json import requestsfrom openai import OpenAI MODEL = "aozalevsky/search_slm_demo"SYSTEM = (    "You convert requests for macromolecular structure data into RCSB PDB Search API v2 queries.\n"    f"Today's date: {datetime.date.today().isoformat()}.\n"    'Reply with exactly one JSON object and nothing else: either a Search API query with "query" and '    '"return_type", or {"error": {"type": "not_a_search" | "ambiguous" | "unsupported", "message": "<short reason>"}}.')client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")  def nl2query(text, previous=None):    if previous is not None:  # follow-up edit: the model rewrites the previous query        text = f"Previous query:\n{json.dumps(previous, separators=(',', ':'))}\n\nFollow-up: {text}"    reply = client.chat.completions.create(        model=MODEL, temperature=0, max_tokens=1536,        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}],        extra_body={"chat_template_kwargs": {"enable_thinking": False}},    )    return json.loads(reply.choices[0].message.content)  query = nl2query("find all heteroduplexes resolved by NMR")print(json.dumps(query, indent=1))query = nl2query("only cryo-EM, released after 2020", previous=query)hits = requests.post("https://search.rcsb.org/rcsbsearch/v2/query", json=query).json()print(hits.get("total_count"), [r["identifier"] for r in hits.get("result_set", [])][:5])

The output is one JSON object: either a complete Search API request body (query, return_type, and request_options when sorting, limits or computed models are asked for), ready to POST to https://search.rcsb.org/rcsbsearch/v2/query, or {"error": {"type": ..., "message": ...}}.

In the evaluation below the model ran behind a small service that validates each query against the Search API schema and attribute catalog and retries once with the validator's messages; for version 4.10, 99.4% of gold outputs were valid on the first try and 99.7% after the retry.

Evaluation

Execution match: the predicted query and the reference query are run on the live RCSB Search API and must return the same set of IDs (with the same sort and limit). Acceptable also counts an alternative reading listed in the reference (e.g. a taxonomy-ID filter instead of a name filter); strict uses the preferred reading only.

Gold set: 356 hand-checked requests in the style of real users, never trained on:

198 from version 1: "all human proteins", "all bacterial ribosomes", "find all nanobody complexes", every search service and return type, 35 follow-up edits, 12 requests that should be refused;
50 added for version 2: common named objects, integrative structures, full-text fallbacks, follow-ups in these areas;
55 added for version 3, 22 of them about metals: nine requests from the public demo log, verbatim, and 46 written independently of the training templates;
39 added for version 4: enzyme classes, logical operators and interaction partners. Three are demo requests, verbatim. The partner templates were written after their four gold items, so those four are less independent;
14 added in version 4.2: author names written in different ways.

Table with columns: model, v1 items (198), v2 items (50), v3 items (55), v4 items (39 / 53), valid, latency p50 / p95
model	v1 items (198)	v2 items (50)	v3 items (55)	v4 items (39 / 53)	valid	latency p50 / p95
Qwen3.5-4B, 8 few-shot examples (no fine-tuning)	20.2%	–	–	–	87.4%	2.3 s / 8.2 s
search_slm_demo v1	91.4%	14.0%	–	–	99.0%

Percentages are acceptable answers.

§ All 53 version-4 items. v4.7: 89.7% on the 39 that v4 was measured on (v4: 94.9%) and 100% on the 14 author-name items; v4.10: 100% on both.

† Version 2 was measured on the first 33 of the 55 version 3 items, before the 22 metal items were added.

‡ Version 3 was measured on the first 38 of the 39 version 4 items.

* Version 2 maps named objects to curated recipes; on 16 v1 items that name an object the recipe reading was added as an acceptable answer. Against the untouched v1 answers v2 scores 90.9% (v1: 91.4%).

On 300 requests generated from templates held out of training:

v4.10: 82.7% acceptable, 99.7% valid; 83.0% on its own held-out sample;
v4.7: 79.3% acceptable, 99.7% valid; 85.7% on its own held-out sample;
v4: 78.7% acceptable, 100% valid. Six of these targets use the earlier name recipes for beta-lactamases, isomerases and glycosyltransferases, which version 4 replaced with EC classes. With those six rewritten, v4 scores 80.7% and v3 79.7%. On v4's own held-out sample, v4 scores 83.3%;
v3: 81.7% acceptable, 99.7% valid;
v2: 78.7% acceptable, 99.3% valid;
v1: 44.7%, but it saw 60 of those 149 templates in training. Latency is measured end to end per request (generation, validation, retry) with vLLM 0.30, greedy decoding, schema-constrained output, MTP speculative decoding; v1 on an RTX A4000, v2 to v4 on an RTX PRO 6000.

Limitations

Trained against the RCSB schema and attribute catalog of 2026-09-30; attributes added or renamed later are unknown to it.
Named objects map to about 300 curated recipes. All of them are in the training data, and some are provisional (chosen by measurement and sampling, pending a curator's review). Unknown names should become a full-text search with the user's words, but the model still sometimes invents a recipe-like list for a named complex it does not know.
"With ligands" counts every non-polymer except water, including ions and crystallization additives. Ask for a "ligand of interest" or an "inhibitor" to leave those out. Fold-class wording only matches chains that CATH has classified.
Known misses of version 4.10 in its probe sets:
- "a Q-score better than 0.55" comes out as below 0.55;
- a space group written without spaces that looks like a UniProt accession ("P43212") is read as one;
- in very long requests a multi-word name is sometimes shortened or replaced with invented title words.
Only the crystallization pH is searchable: a pH with cryo-EM or NMR is refused as unsupported. "Physiological pH" is fixed at 6.5–7.5, although what counts as physiological depends on the compartment or organ.
"Approved drugs" follows DrugBank's approval status, minus a fixed list of water and 14 common additives and gases (glycerol, acetate, citrate, Tris, carbon monoxide, oxygen, …).
Not taught: binding affinities (Ki, Kd, IC50), and "non-metal ligands", which have no clean expression in the Search API.

Training

Method: LoRA (r 32, alpha 64, dropout 0.05) on all attention, MLP and linear-attention projections of the language model, bf16, effective batch 32. Loss on the JSON answer only; the adapter was merged into the base weights. Version 4.10 was trained from scratch on one RTX PRO 6000: 2 epochs over 58,263 pairs, learning rate 2e-4 with cosine decay, 2.2 h.
Data: 58,263 synthetic request → query pairs from 1,779 templates:
- the version 1 templates;
- about 600 derived from real PDB entry titles;
- 80 for integrative structures and 80 for named objects;
- 5 for full-text fallbacks;
- 53 added for version 3 and 58 for version 4;
- 135 added in versions 4.2–4.7 and 80 in versions 4.8–4.10.
Every named object and every full-text fallback concept appears in at least one template expansion. Requests come in several styles (terse, conversational, expert, title-like, messy with typos) and are filled with real values from the RCSB PDB.

The templates, request phrasings and object recipes were generated with Anthropic's Claude. Every object recipe was measured on the live Search API, and every target query was validated and executed against it; queries with no hits, or that the API rejected, were dropped. 3% of pairs are refusals and 15% are follow-up edits.

License and attribution

Released under the Apache License 2.0 (see LICENSE), the license of the base model.

This is a modified version of Qwen/Qwen3.5-4B by the Qwen team: the language-model weights were fine-tuned as described above. The vision encoder and multi-token-prediction weights are unchanged from the base model, and the tokenizer and chat template are the base model's. This project is not affiliated with or endorsed by the Qwen team, Alibaba Cloud or the RCSB PDB.

The values used to build the training data come from the RCSB Protein Data Bank, whose data are available under CC0 1.0. When you use structures found with this model, cite the PDB entries and the RCSB PDB as described in their usage policy.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

aozalevsky

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality