What it's for
Use it to measure, at scale, how medical and pharma content talks about specific drugs and therapies:
- Medical affairs and KOL insight: how conference abstracts, publications and physicians' posts present a product,
and how that changes over time.
- Competitive intelligence: sentiment toward your product and its competitors, side by side, in the same documents.
- Literature and news monitoring: a label for every named drug in a stream of publications, trial records and news.
It handles five kinds of content: conference abstracts, PubMed publications, clinical trial registry records,
physicians' social media posts, and press and news articles.
A made-up example:
Patients on drug A lived longer than those on standard care, with manageable side effects. Drug B, tested in an
earlier study, was stopped after reports of liver damage. All patients also received drug C as background therapy.
With the targets drug A, drug B and drug C, the expected answer is
{"drug A": "Positive", "drug B": "Negative", "drug C": "Neutral"}.
Not for:
- diagnosis, treatment or any other clinical decision
- adverse-event detection or regulatory safety reporting, which it wasn't tested for
- finding the drugs in a text: you list the targets, and it judges only those
- judging whether a drug works: a label says how the text presents the drug, not what is true about it
How well it works
Sentiment
We tested it on 475 held-out documents with 500 drug-level labels. No test document was seen in training. The
reference labels were made blind by Claude (Anthropic's model), not by human experts. The main figures use the 473
labels that Claude was confident about.
Table with columns: Model, Accuracy, Negative precision, Negative recall| Model | Accuracy | Negative precision | Negative recall |
|---|
| This adapter (one request per document) | 91.3% | 84% | 84% |
| Gemma 4 31B, the base model | 90.5% | 71% | 87% |
| Qwen3.8-27B | 83.5% | 100% | 13% |
| Qwen3-32B | 83.3% | 60% | 84% |
| MedGemma 27B | 82.9% | 79% | 58% |
| Llama 3.1 8B | 76.3% | 62% | 11% |
The other models got long labelling instructions and one request per drug, the setup that gave the base model its
best score. Qwen3.8-27B's 100% Negative precision comes from only 6 Negative calls.
What the adapter adds over the base model:
- One request per document instead of one per drug.
- Answers you can always parse: no format failures in 1,500 answers over three runs of the test. Asked for all
drugs at once in this format, the base model scored 32%, mostly because 59% of its answers couldn't be read.
- More reliable Negative calls: 7 of its 45 Negative calls were wrong, against 16 of the base model's 55.
More detail:
- The 95% range for the 91.3% is 88–94%. On all 500 labels, uncertain ones included, the adapter scores 88.6%.
- On each of the five content types, accuracy is between 87% and 95%.
- There are only 45 Negatives among the 473 labels, so the Negative figures are less certain: the 95% range for the
84% is 71–92%.
- Served in FP8 (vLLM's
--quantization fp8), the adapter scored 91.5–91.8% on the same 473 labels.
Medical knowledge
These are scores on standard medical benchmarks, mostly with thinking mode on. The sentiment task runs with thinking
mode off, so they show what the model knows about medicine, not results of the sentiment task.
The knowledge comes from Gemma 4. Run the same way, the base model scores within chance of the adapter: 85.9% against
86.5% on 3,091 exam questions with thinking mode on. Fine-tuning kept that knowledge but didn't add to it. The adapter
is meant for sentiment, not for answering medical questions.
Table with columns: Benchmark, What it tests, This adapter, Gemma 4 31B, the base model, Best frontier score we found| Benchmark | What it tests | This adapter | Gemma 4 31B, the base model | Best frontier score we found |
|---|
| MedQA | US medical licensing exam, thinking mode on | 94.2 | 94.1 | 96.5 (o1, top of the Vals.ai leaderboard) |
| MedMCQA | Indian medical entrance exams, a fixed 1,000-question sample, thinking mode on | 81.4 | 80.7 | 83.9 (o1-preview, 2024) |
| MMLU medical | Six medical subjects from MMLU, step by step (not thinking mode) | 94.2 | 94.2 | 96.0 (GPT-5) |
| PubMedQA | Yes/no/maybe questions on research abstracts, thinking mode on (79.4 step by step) | 80.2 | 78.5* | 82.0 (GPT-4 with Medprompt prompting, 2023) |
* Step by step: we didn't run the base model with thinking mode on PubMedQA.
We ran the adapter and the base model ourselves on the standard test questions. Other models' scores come from public
leaderboards and papers with slightly different setups, so treat differences of 1–2 points as a tie. On MedQA, the
adapter is about 2 points below the top of the Vals.ai leaderboard (96.5%, last updated April 2026).
Sources for other models' scores:
How to use it
The repo is gated. Accept the terms on this page, then log in with hf auth login (or set HF_TOKEN) so that vLLM or
Transformers can download it.
Turn thinking mode off and use greedy decoding (temperature 0). Send this system prompt, word for word:
You classify the sentiment that biomedical or pharmaceutical content expresses toward each given target topic (a drug, product, drug class, or therapy/procedure).Positive: the content presents the target favourably (efficacy, benefit, manageable safety, better than an alternative, approval or endorsement).Negative: the content presents the target unfavourably (lack of efficacy, worse than an alternative, toxicity or safety problems, harms, misuse, failure).Neutral: no evaluative judgement about the target (background, design or protocol without results, comparator or concomitant use, methods).Judge only from the content and only toward each target. Reply with a JSON object mapping each target topic to Positive, Neutral or Negative.
Then send the document in this format, listing every drug or therapy you want judged. Other names for a target go in
brackets after "also:".
Source type: Press / news article Content:<<<Patients on drug A lived longer than those on standard care, with manageable side effects. Drug B, tested in an earlier study, was stopped after reports of liver damage. All patients also received drug C as background therapy.>>> Target topics:1. drug A (also: DA-101)2. drug B3. drug C
For the source type, use whichever of these fits best:
Conference abstract
Publication (PubMed record)
Clinical trial registry record (ClinicalTrials.gov)
Social media post by a physician
Press / news article
The answer has one key per target, using its main name: {"drug A": "Positive", "drug B": "Negative", "drug C": "Neutral"}.
With vLLM
vllm serve google/gemma-4-31B-it --enable-lora --max-lora-rank 16 \ --lora-modules pharma-sentiment=empirischtech/gemma-4-31B-it-Pharma-Sentiment-LoRA \ --max-model-len 16384 --limit-mm-per-prompt '{"image":0}'
Then call the OpenAI-compatible endpoint:
import jsonfrom openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") targets = ["drug A (also: DA-101)", "drug B", "drug C"]user_message = ( f"Source type: Press / news article\n\nContent:\n<<<\n{document_text}\n>>>\n\nTarget topics:\n" + "\n".join(f"{i}. {t}" for i, t in enumerate(targets, 1)))response = client.chat.completions.create( model="pharma-sentiment", messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user_message}], temperature=0, max_tokens=32 + sum(16 + len(t) // 2 for t in targets), extra_body={"chat_template_kwargs": {"enable_thinking": False}},)labels = json.loads(response.choices[0].message.content)
import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer tok = AutoTokenizer.from_pretrained("google/gemma-4-31B-it")model = AutoModelForImageTextToText.from_pretrained("google/gemma-4-31B-it", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "empirischtech/gemma-4-31B-it-Pharma-Sentiment-LoRA") messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user_message}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=256, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Load the base model with AutoModelForImageTextToText, because the adapter's layer names follow that class.
Hardware
In bf16 the weights take about 58 GiB of GPU memory. We serve the model on one 96 GB NVIDIA RTX PRO 6000 Blackwell,
with vLLM 0.29. Adding --quantization fp8 --kv-cache-dtype fp8 to the vLLM command roughly halves the weights' memory
and leaves about four times as much room for documents, with the same accuracy (see above).
Training
Fine-tuned on ~50k curated pharma and medical documents, each with drug-level sentiment labels.
- LoRA rank 16, alpha 32 and dropout 0.05, on the attention and MLP projections of the language model (122M trainable
parameters)
- learning rate 1e-4 with cosine decay, about 32 examples per step
- loss on the answer only
- documents up to 4,096 tokens, in bf16
Limits
- The reference labels come from a model, not from human experts. A second, independent labelling pass agreed with the
first on 96% of a sample, so treat differences of a point or two as a tie.
- It was trained and tested on English text only. Gemma's image input wasn't trained or tested.
- Documents longer than 4,096 tokens weren't used in training, so very long documents are less well covered.
- It judges only the targets you list, and only from the text you give it.
License
Apache 2.0, the same licence as the base model. This adapter modifies
Gemma 4 31B by Google DeepMind: we fine-tuned it with LoRA. See
LICENSE and NOTICE. To get access, you also accept that you won't use the model for diagnosis or treatment
decisions.
Developed by Empirisch Tech GmbH.