⚠️ Critical Usage Warning: The 422 Schema Error
If you are evaluating this agent against the MetaGuard API server, you will likely encounter 422 Unprocessable Entity errors.
Why this happens: The environment enforces strict JSON validation with additionalProperties set to false. While this model correctly selects valid actions, it occasionally hallucinates helpful but invalid tracking fields (such as actions_already_taken or confidence) in its JSON output.
Required Parser Patch
To successfully step the environment without crashing the API, you must sanitize the model's output before passing it to the server. Use the following Python drop-in replacement for your parsing logic. This guarantees the required reasoning field is present, enforces strict enums, and strips out hallucinated keys to perfectly satisfy the environment schema.
import json
def parse_attempt(text):
candidates = []
if "```" in text:
for p in text.split("```"):
p = p.strip().lstrip("json").strip()
candidates.append(p)
candidates.append(text.strip())
s, e = text.find("{"), text.rfind("}") + 1
if s != -1 and e > s:
candidates.append(text[s:e])
for c in candidates:
try:
r = json.loads(c)
if isinstance(r, dict) and "action_type" in r:
r.setdefault("reasoning", "No reasoning provided.")
valid_cat = {"HEALTHCARE", "FINANCIAL", "NONE", None}
if r.get("violation_category") not in valid_cat:
r.pop("violation_category", None)
allowed = {"action_type", "reasoning", "violation_category", "metadata"}
return {k: v for k, v in r.items() if k in allowed}
except Exception:
continue
return None
Evaluation Results
The model was evaluated across a 4-task suite testing healthcare policies, financial policies, multimodal analysis, and targeting constraints (10 steps per episode).
Baseline (Pre-GRPO)
- Success Rate: 0/4 (0%)
- Mean Reward per Step: -0.185
Final (Post-GRPO)
- Success Rate: 0/4 (0%)
- Mean Reward per Step: -0.050
Note: The GRPO fine-tuning successfully improved the agent's intermediate signal gathering phase, significantly pulling the mean reward up from the baseline.
Limitations
While the model shows strong capability in the information-gathering phase (e.g., calling query_regulations and analyze_image), it currently struggles with strict terminal sequence chaining. Specifically, the agent frequently fails to explicitly call the submit_audit action before attempting to issue a final approve or reject decision, resulting in an incomplete task execution.
- Developed by: parth-1
- License: apache-2.0
- Finetuned from model : parth-1/metaguard-policy-agent-v1
This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.