What it's for
Use it to measure, at scale, how medical and pharma content talks about specific drugs and therapies:
- Medical affairs and KOL insight: how conference abstracts, publications and physicians' posts present a product,
and how that changes over time.
- Competitive intelligence: sentiment toward your product and its competitors, side by side, in the same documents.
- Literature and news monitoring: a label for every named drug in a stream of publications, trial records and news.
It handles five kinds of content: conference abstracts, PubMed publications, clinical trial registry records,
physicians' social media posts, and press and news articles.
Not for:
- diagnosis, treatment or any other clinical decision
- adverse-event detection or regulatory safety reporting, which it wasn't tested for
- finding the drugs in a text: you list the targets, and it judges only those
- judging whether a drug works: a label says how the text presents the drug, not what is true about it
How well it works
Sentiment
We tested it on 475 held-out documents with 500 drug-level labels. No test document was seen in training. The
reference labels were made blind by Claude (Anthropic's model), not by human experts.
- This FP8 build: 88.8% on all 500 labels, against 88.6% for the adapter in bf16. Every target got a valid label.
- The adapter in bf16, on the 473 labels Claude was confident about: 91.3% accuracy (95% range 88–94%), and 84%
precision and 84% recall on Negatives.
The adapter's card compares these results with Gemma 4 31B itself and with other open models.
Medical knowledge
These scores were measured with the adapter, which scored the same as this build on the sentiment test. They come from
standard medical benchmarks, mostly with thinking mode on. The sentiment task runs with thinking mode off, so they show
what the model knows about medicine, not results of the sentiment task.
The knowledge comes from Gemma 4. Run the same way, the base model scores within chance of the adapter: 85.9% against
86.5% on 3,091 exam questions with thinking mode on. Fine-tuning kept that knowledge but didn't add to it. The model is
meant for sentiment, not for answering medical questions.
Table with columns: Benchmark, What it tests, The adapter, Gemma 4 31B, the base model, Best frontier score we found| Benchmark | What it tests | The adapter | Gemma 4 31B, the base model | Best frontier score we found |
|---|
| MedQA | US medical licensing exam, thinking mode on | 94.2 | 94.1 | 96.5 (o1, top of the Vals.ai leaderboard) |
| MedMCQA | Indian medical entrance exams, a fixed 1,000-question sample, thinking mode on | 81.4 | 80.7 | 83.9 (o1-preview, 2024) |
| MMLU medical | Six medical subjects from MMLU, step by step (not thinking mode) | 94.2 | 94.2 | 96.0 (GPT-5) |
| PubMedQA | Yes/no/maybe questions on research abstracts, thinking mode on (79.4 step by step) | 80.2 | 78.5* | 82.0 (GPT-4 with Medprompt prompting, 2023) |
* Step by step: we didn't run the base model with thinking mode on PubMedQA.
We ran the adapter and the base model ourselves on the standard test questions. Other models' scores come from public
leaderboards and papers with slightly different setups, so treat differences of 1–2 points as a tie. On MedQA, the
adapter is about 2 points below the top of the Vals.ai leaderboard (96.5%, last updated April 2026).
Sources for other models' scores:
How to use it
The repo is gated. Accept the terms on this page, then log in with hf auth login (or set HF_TOKEN) so that vLLM can
download it.
vllm serve empirischtech/gemma-4-31B-it-Pharma-Sentiment-FP8 --served-model-name pharma-sentiment \ --language-model-only --max-model-len 16384
Turn thinking mode off and use greedy decoding (temperature 0). Send this system prompt, word for word:
You classify the sentiment that biomedical or pharmaceutical content expresses toward each given target topic (a drug, product, drug class, or therapy/procedure).Positive: the content presents the target favourably (efficacy, benefit, manageable safety, better than an alternative, approval or endorsement).Negative: the content presents the target unfavourably (lack of efficacy, worse than an alternative, toxicity or safety problems, harms, misuse, failure).Neutral: no evaluative judgement about the target (background, design or protocol without results, comparator or concomitant use, methods).Judge only from the content and only toward each target. Reply with a JSON object mapping each target topic to Positive, Neutral or Negative.
Then send the document in this format, listing every drug or therapy you want judged. Other names for a target go in
brackets after "also:".
Source type: Press / news article Content:<<<Patients on drug A lived longer than those on standard care, with manageable side effects. Drug B, tested in an earlier study, was stopped after reports of liver damage. All patients also received drug C as background therapy.>>> Target topics:1. drug A (also: DA-101)2. drug B3. drug C
For the source type, use whichever of these fits best:
Conference abstract
Publication (PubMed record)
Clinical trial registry record (ClinicalTrials.gov)
Social media post by a physician
Press / news article
The answer has one key per target, using its main name: {"drug A": "Positive", "drug B": "Negative", "drug C": "Neutral"}.
From Python, through vLLM's OpenAI-compatible endpoint:
import jsonfrom openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") targets = ["drug A (also: DA-101)", "drug B", "drug C"]user_message = ( f"Source type: Press / news article\n\nContent:\n<<<\n{document_text}\n>>>\n\nTarget topics:\n" + "\n".join(f"{i}. {t}" for i, t in enumerate(targets, 1)))response = client.chat.completions.create( model="pharma-sentiment", messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user_message}], temperature=0, max_tokens=32 + sum(16 + len(t) // 2 for t in targets), extra_body={"chat_template_kwargs": {"enable_thinking": False}},)labels = json.loads(response.choices[0].message.content)
Hardware
The weights take 30.7 GiB of GPU memory, and vLLM needs about 3.5 GiB more to run. We tested the build on one 96 GB
NVIDIA RTX PRO 6000 Blackwell with vLLM 0.29.
How it was made
We merged the adapter into the base model in bf16. Then we converted every linear layer of the language model to FP8
with llm-compressor's FP8_DYNAMIC scheme: the weights get per-channel scales, and the activations are scaled per token
at runtime. The output head and the vision encoder stay in bf16. The weights are in the compressed-tensors format, and
recipe.yaml holds the exact recipe.
Training
Fine-tuned on ~50k curated pharma and medical documents, each with drug-level sentiment labels. The adapter's card
lists the LoRA settings.
Limits
- The reference labels come from a model, not from human experts. A second, independent labelling pass agreed with the
first on 96% of a sample, so treat differences of a point or two as a tie.
- It was trained and tested on English text only. Gemma's image input wasn't trained or tested.
- Documents longer than 4,096 tokens weren't used in training, so very long documents are less well covered.
- It judges only the targets you list, and only from the text you give it.
License
Apache 2.0, the same licence as the base model. This model modifies
Gemma 4 31B by Google DeepMind. We fine-tuned it with LoRA, merged the
adapter into the weights, and converted the language model's linear layers to FP8. See LICENSE and NOTICE. To get
access, you also accept that you won't use the model for diagnosis or treatment decisions.
Developed by Empirisch Tech GmbH.