Evaluation Results
The model was evaluated across a 4-task suite testing healthcare policies, financial policies, multimodal analysis, and targeting constraints (10 steps per episode).
Baseline (Pre-GRPO)
- Success Rate: 0/4 (0%)
- Mean Reward per Step: -0.185
Final (Post-GRPO)
- Success Rate: 0/4 (0%)
- Mean Reward per Step: -0.050
Note: The GRPO fine-tuning successfully improved the agent's intermediate signal gathering phase, significantly pulling the mean reward up from the baseline.
Limitations
While the model shows strong capability in the information-gathering phase (e.g., calling query_regulations and analyze_image), it currently struggles with strict terminal sequence chaining. Specifically, the agent frequently fails to explicitly call the submit_audit action before attempting to issue a final approve or reject decision, resulting in an incomplete task execution.
- Developed by: parth-1
- License: apache-2.0
- Finetuned from model : parth-1/metaguard-policy-agent-v1
This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.