hiimivantang

medgen-ja-qwen3-8b-grounded-answering-lora

README

License: apache-2.0

Base model (required)

Table

Base model	`Qwen/Qwen3-8B`
Revision (commit)	`b968826d9c46dd6066d109eabc6255188de91218`
Chat template	`qwen3`, thinking disabled (`enable_thinking=False`)

The adapter was trained against this exact revision. adapter_config.json records it, and AutoPeftModelForCausalLM uses it; if you load the base model yourself, pass revision="b968826d9c46dd6066d109eabc6255188de91218" as below.

Usage

Transformers + PEFT

python

import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE, REV = "Qwen/Qwen3-8B", "b968826d9c46dd6066d109eabc6255188de91218"tok = AutoTokenizer.from_pretrained(BASE, revision=REV)model = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "hiimivantang/medgen-ja-qwen3-8b-grounded-answering-lora") messages = [{"role": "system", "content": SYSTEM_PROMPT},            {"role": "user", "content": USER_PROMPT}]  # see "Prompt format" belowinputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,                                 return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1536, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

vLLM (OpenAI-compatible server, several adapters on one base)

bash

vllm serve Qwen/Qwen3-8B --revision b968826d9c46dd6066d109eabc6255188de91218 --enable-lora --max-lora-rank 16 \  --lora-modules ga-lora=hiimivantang/medgen-ja-qwen3-8b-grounded-answering-lora

Then request "model": "ga-lora" with temperature 0 and the prompt below. Both adapters from this project can be served on one base model this way.

Prompt format

The adapter was trained on exactly this system prompt and user template (Japanese). Use them verbatim; other prompts will not reproduce the results below. Replace {question} and one numbered block per evidence passage.

System prompt

text

あなたは医学文献に基づいて日本語で回答するアシスタントです。提示された根拠文献の内容だけを根拠に答え、文献に書かれていない知識で補ってはいけません。根拠が不十分な場合は、その旨を明示して回答を控えてください。個別の患者への診断や治療の推奨は行いません。

User prompt template

text

次の質問に、下の根拠文献だけを用いて日本語で回答してください。規則:- answer: 根拠文献が示す内容を日本語で簡潔にまとめる。数値（効果量、信頼区間、割合など）は文献の値をそのまま用いる。研究デザインや結果の不一致があれば触れる。- citations: answer の各主張を支持する文献を、document_id と passage_id（根拠文献に表示されたもの）で示し、claim にその文献が支持する内容を日本語で書く。提示されていない文献は引用しない。- evidence_sufficient: 根拠文献だけで質問に答えられる場合は true。答えられない場合は false とし、answer で「提示された文献からは回答できない」旨とその理由を述べ、citations は空にする。- limitations: 根拠の限界（研究デザイン、対象集団、文献間の不一致など）を列挙する。出力はJSONオブジェクトのみとし、説明文やコードブロック記号は付けないでください。形式: {"answer": "...", "citations": [{"document_id": "...", "passage_id": "...", "claim": "..."}], "evidence_sufficient": true, "limitations": [...]} 質問:「{question}」 根拠文献:[1] document_id: {document_id} / passage_id: {passage_id}タイトル: {title}掲載: {journal} ({publication_date}) / セクション: {section}本文: {text}

The model replies with a single JSON object.

Evaluation

Held-out test set of 430 synthetic items (test_ga, 10 categories; 111 items whose reference an independent audit rejected are excluded from the primary analysis, leaving 319), identical evidence for every model, greedy decoding, thinking disabled.

Table with columns: Model, Grounded composite, Citation validity, Numeric preservation, Judge: acceptable, Judge: completeness (0-4)
Model	Grounded composite	Citation validity	Numeric preservation	Judge: acceptable	Judge: completeness (0-4)
Qwen3-8B (base, same prompt)	0.943	0.994	0.921	93.7%	3.15
This adapter	0.944	1.000	0.944	93.4%	3.61
gpt-6.1-sol (OpenAI reference)	0.976	1.000	0.980	99.7%	3.93

Its main effect is behavioural: more complete answers (judge completeness +0.46, 95% CI +0.33 to +0.58) and more precise citing, with no measurable loss in correctness. The pre-registered hypothesis that it improves grounded answering was not supported on the deterministic endpoints.
Do not drop it into an end-to-end RAG pipeline as is: with its own retrieval, a pipeline using this adapter scored lower than the base model (grounded composite −0.055, 95% CI −0.081 to −0.030), mainly through lower citation recall and more abstention.
Judge scores: GLM 5.3 (primary), Kimi K3 (adjudication, 88% agreement on acceptability); LLM judging is not clinical validation.

Training

Table

Method	LoRA (PEFT), LLaMA-Factory
Rank / alpha / dropout	16 / 32 / 0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`
Learning rate / epochs	1e-4 (cosine) / 2
Training examples / steps	944 / 118
Seed	42
Precision	bf16
Hardware	NVIDIA DGX Spark (GB10)

Training metrics are in all_results.json, train_results.json, eval_results.json and trainer_state.json.

Training data

Synthetic. Questions, structured labels and reference answers were generated by a local Apache-2.0 teacher model (Qwen3.5-122B) from open-access literature abstracts in Europe PMC, restricted to CC BY and CC0 licences. No OpenAI output was used as training data. Test documents were published after 2026-06-01, later than every training document, and test items share no documents with training.

Conclusions

Its effect is behavioural, not accuracy. Judge scores showed no correctness change but more complete answers (+0.46 on a 0-4 scale) and much more precise citing (gold-citation precision 0.97 vs 0.84 under fixed evidence). The deterministic composite was flat.
Do not drop it into an end-to-end RAG pipeline as is: with its own retrieval, the pipeline scored lower than the plain base model (−0.055), through lower citation recall and more abstention. If you want more complete, tightly-cited answers from fixed evidence, it helps; for full pipelines, fix retrieval first.
OpenAI is still ahead on answer quality (judged acceptable 99.7% vs 93.4%; every judged dimension p < 0.05), so it is not a frontier replacement for open-ended grounded answering.
It serves together with the question-understanding adapter on one shared base (never merged; see the companion question-understanding adapter and clinical-document adapter).

Limitations

Synthetic questions: cleaner and less ambiguous than real clinicians' questions; the intent mix was designed, not observed.
One teacher model wrote the training labels and references, so its wording and terminology biases carry over.
One training seed; run-to-run variance is not estimated.
Japanese questions over Europe PMC abstracts (mostly English); no dosing, insurance or package-insert questions.
Not clinically validated.

License

The adapter weights are released under Apache-2.0. The base model has its own licence; see Qwen/Qwen3-8B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

hiimivantang

Model Tree

Base

Qwen/Qwen3-8B

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Base model (required)

Table

Base model	`Qwen/Qwen3-8B`
Revision (commit)	`b968826d9c46dd6066d109eabc6255188de91218`
Chat template	`qwen3`, thinking disabled (`enable_thinking=False`)

Usage

Transformers + PEFT

python

import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE, REV = "Qwen/Qwen3-8B", "b968826d9c46dd6066d109eabc6255188de91218"tok = AutoTokenizer.from_pretrained(BASE, revision=REV)model = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "hiimivantang/medgen-ja-qwen3-8b-grounded-answering-lora") messages = [{"role": "system", "content": SYSTEM_PROMPT},            {"role": "user", "content": USER_PROMPT}]  # see "Prompt format" belowinputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,                                 return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1536, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE, REV = "Qwen/Qwen3-8B", "b968826d9c46dd6066d109eabc6255188de91218"tok = AutoTokenizer.from_pretrained(BASE, revision=REV)model = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "hiimivantang/medgen-ja-qwen3-8b-grounded-answering-lora") messages = [{"role": "system", "content": SYSTEM_PROMPT},            {"role": "user", "content": USER_PROMPT}]  # see "Prompt format" belowinputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,                                 return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1536, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

vLLM (OpenAI-compatible server, several adapters on one base)

bash

vllm serve Qwen/Qwen3-8B --revision b968826d9c46dd6066d109eabc6255188de91218 --enable-lora --max-lora-rank 16 \  --lora-modules ga-lora=hiimivantang/medgen-ja-qwen3-8b-grounded-answering-lora

vllm serve Qwen/Qwen3-8B --revision b968826d9c46dd6066d109eabc6255188de91218 --enable-lora --max-lora-rank 16 \  --lora-modules ga-lora=hiimivantang/medgen-ja-qwen3-8b-grounded-answering-lora

Then request "model": "ga-lora" with temperature 0 and the prompt below. Both adapters from this project can be served on one base model this way.

Prompt format

System prompt

text

あなたは医学文献に基づいて日本語で回答するアシスタントです。提示された根拠文献の内容だけを根拠に答え、文献に書かれていない知識で補ってはいけません。根拠が不十分な場合は、その旨を明示して回答を控えてください。個別の患者への診断や治療の推奨は行いません。

あなたは医学文献に基づいて日本語で回答するアシスタントです。提示された根拠文献の内容だけを根拠に答え、文献に書かれていない知識で補ってはいけません。根拠が不十分な場合は、その旨を明示して回答を控えてください。個別の患者への診断や治療の推奨は行いません。

User prompt template

text

次の質問に、下の根拠文献だけを用いて日本語で回答してください。規則:- answer: 根拠文献が示す内容を日本語で簡潔にまとめる。数値（効果量、信頼区間、割合など）は文献の値をそのまま用いる。研究デザインや結果の不一致があれば触れる。- citations: answer の各主張を支持する文献を、document_id と passage_id（根拠文献に表示されたもの）で示し、claim にその文献が支持する内容を日本語で書く。提示されていない文献は引用しない。- evidence_sufficient: 根拠文献だけで質問に答えられる場合は true。答えられない場合は false とし、answer で「提示された文献からは回答できない」旨とその理由を述べ、citations は空にする。- limitations: 根拠の限界（研究デザイン、対象集団、文献間の不一致など）を列挙する。出力はJSONオブジェクトのみとし、説明文やコードブロック記号は付けないでください。形式: {"answer": "...", "citations": [{"document_id": "...", "passage_id": "...", "claim": "..."}], "evidence_sufficient": true, "limitations": [...]}質問:「{question}」根拠文献:[1] document_id: {document_id} / passage_id: {passage_id}タイトル: {title}掲載: {journal} ({publication_date}) / セクション: {section}本文: {text}

次の質問に、下の根拠文献だけを用いて日本語で回答してください。規則:- answer: 根拠文献が示す内容を日本語で簡潔にまとめる。数値（効果量、信頼区間、割合など）は文献の値をそのまま用いる。研究デザインや結果の不一致があれば触れる。- citations: answer の各主張を支持する文献を、document_id と passage_id（根拠文献に表示されたもの）で示し、claim にその文献が支持する内容を日本語で書く。提示されていない文献は引用しない。- evidence_sufficient: 根拠文献だけで質問に答えられる場合は true。答えられない場合は false とし、answer で「提示された文献からは回答できない」旨とその理由を述べ、citations は空にする。- limitations: 根拠の限界（研究デザイン、対象集団、文献間の不一致など）を列挙する。出力はJSONオブジェクトのみとし、説明文やコードブロック記号は付けないでください。形式: {"answer": "...", "citations": [{"document_id": "...", "passage_id": "...", "claim": "..."}], "evidence_sufficient": true, "limitations": [...]} 質問:「{question}」 根拠文献:[1] document_id: {document_id} / passage_id: {passage_id}タイトル: {title}掲載: {journal} ({publication_date}) / セクション: {section}本文: {text}

The model replies with a single JSON object.

Evaluation

Table with columns: Model, Grounded composite, Citation validity, Numeric preservation, Judge: acceptable, Judge: completeness (0-4)
Model	Grounded composite	Citation validity	Numeric preservation	Judge: acceptable	Judge: completeness (0-4)
Qwen3-8B (base, same prompt)	0.943	0.994	0.921	93.7%	3.15
This adapter	0.944	1.000	0.944	93.4%	3.61
gpt-6.1-sol (OpenAI reference)	0.976	1.000	0.980	99.7%	3.93

Its main effect is behavioural: more complete answers (judge completeness +0.46, 95% CI +0.33 to +0.58) and more precise citing, with no measurable loss in correctness. The pre-registered hypothesis that it improves grounded answering was not supported on the deterministic endpoints.
Do not drop it into an end-to-end RAG pipeline as is: with its own retrieval, a pipeline using this adapter scored lower than the base model (grounded composite −0.055, 95% CI −0.081 to −0.030), mainly through lower citation recall and more abstention.
Judge scores: GLM 5.3 (primary), Kimi K3 (adjudication, 88% agreement on acceptability); LLM judging is not clinical validation.

Training

Table

Method	LoRA (PEFT), LLaMA-Factory
Rank / alpha / dropout	16 / 32 / 0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`
Learning rate / epochs	1e-4 (cosine) / 2
Training examples / steps	944 / 118
Seed	42
Precision	bf16
Hardware	NVIDIA DGX Spark (GB10)

Training metrics are in all_results.json, train_results.json, eval_results.json and trainer_state.json.

Training data

Conclusions

Its effect is behavioural, not accuracy. Judge scores showed no correctness change but more complete answers (+0.46 on a 0-4 scale) and much more precise citing (gold-citation precision 0.97 vs 0.84 under fixed evidence). The deterministic composite was flat.
Do not drop it into an end-to-end RAG pipeline as is: with its own retrieval, the pipeline scored lower than the plain base model (−0.055), through lower citation recall and more abstention. If you want more complete, tightly-cited answers from fixed evidence, it helps; for full pipelines, fix retrieval first.
OpenAI is still ahead on answer quality (judged acceptable 99.7% vs 93.4%; every judged dimension p < 0.05), so it is not a frontier replacement for open-ended grounded answering.
It serves together with the question-understanding adapter on one shared base (never merged; see the companion question-understanding adapter and clinical-document adapter).

Limitations

Synthetic questions: cleaner and less ambiguous than real clinicians' questions; the intent mix was designed, not observed.
One teacher model wrote the training labels and references, so its wording and terminology biases carry over.
One training seed; run-to-run variance is not estimated.
Japanese questions over Europe PMC abstracts (mostly English); no dosing, insurance or package-insert questions.
Not clinically validated.

License

The adapter weights are released under Apache-2.0. The base model has its own licence; see Qwen/Qwen3-8B.