jatshi

EvidenceAgent-MM-Qwen3-1.7B-GRPO-LoRA

README

License: apache-2.0

Training

The training chain was executed on one RTX 4090:

supervised fine-tuning on schema-valid grounded responses;
DPO on chosen/rejected evidence and refusal behavior;
100 GRPO steps with a shaped verifiability reward.

The native Qwen chat template was used with thinking disabled. The GRPO run used 80 training examples from the checked-in EAMM bronze benchmark. Mean training reward increased from 0.5532 over the first 20 steps to 0.7796 over the last 20 steps (0.7101 across all 100 steps).

Evaluation

The held-out benchmark is deliberately small: 120 questions across 12 synthetic meeting sessions, with session-level train/validation/test separation. Validation and test each contain 20 questions from two sessions. These numbers demonstrate the structured-output and evidence-contract behavior on this bronze benchmark; they are not evidence of broad real-world meeting generalization.

Table with columns: Split, Samples, Total reward, Valid JSON, Grounding, Citation, Abstention, Mean latency
Split	Samples	Total reward	Valid JSON	Grounding	Citation	Abstention	Mean latency
test	20	0.9200	1.0000	1.0000	0.8000	0.8000	5.38 s
validation	20	0.9200	1.0000	1.0000	0.8000	0.8000	5.29 s

Inference was measured with Transformers on an RTX 4090. Peak allocated VRAM was approximately 3.40 GiB.

Usage

python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-1.7B"
adapter_id = "jatshi/EvidenceAgent-MM-Qwen3-1.7B-GRPO-LoRA"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)

messages = [
    {
        "role": "system",
        "content": (
            "Return only the EvidenceAgent-MM JSON contract. Cite only evidence "
            "IDs present in the supplied context; clarify or abstain when evidence "
            "is insufficient."
        ),
    },
    {
        "role": "user",
        "content": (
            "Question: 谁提出了低延迟检索方案？\n"
            "Evidence:\n"
            "- [meeting:utt:01] 00:10-00:14 张同学：我建议采用方案 A。\n"
            "- [meeting:ocr:01] slide 4: 方案 A，P95=40ms"
        ),
    },
]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)
with torch.inference_mode():
    output = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))

Limitations

The benchmark sessions are synthetic and templated; only two sessions are held out for each evaluation split.
Reward components validate schema, status, citation overlap, grounding, and abstention behavior. They do not replace human factuality review.
The model can still omit secondary cross-modal citations, reflected in the citation score of 0.80.
Use the full EvidenceAgent-MM evidence gate in safety-sensitive settings. Do not treat model confidence as calibrated probability.

Reproducibility

Training code, dataset generator, reward implementation, DeepSpeed single-GPU comparison, evaluation scripts, and deterministic ablations are in the EvidenceAgent-MM repository.

Framework versions for this run: PyTorch 2.10.0+cu128, Transformers 5.14.1, TRL 0.29.1, PEFT 0.20.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

jatshi

Model Tree

Base

Qwen/Qwen3-1.7B

Adapter

this model

Input Modalities

Text

Output Modalities