Base Model
- Base model:
Qwen/Qwen3-8B
- Adapter type: LoRA
- Training objective: pairwise preference classification over repair candidates
- Output format: single-choice preference between repair A and repair B
This repository contains only the LoRA adapter, not the full base model. Users must load it together with the compatible base model and follow the base model license and terms.
Intended Use
This adapter is intended for research on language-agent debugging, repair selection, and counterfactual replay analysis.
It is designed for AgentBlackBox-style repair ranking:
failed agent trace + repair candidate A + repair candidate B -> preferred repair
Evaluation Summary
Evaluation is performed on replay-labeled RAG repair candidates from AgentBlackBox.
Table with columns: Split, Top-1, Top-2, MRR, NDCG, AUROC, Oracle upper bound| Split | Top-1 | Top-2 | MRR | NDCG | AUROC | Oracle upper bound |
|---|
| dev | 0.6080 | 0.6751 | 0.6533 | 0.6668 | 0.5454 | 0.7149 |
| test | 0.6873 | 0.7059 | 0.6937 | 0.7086 | 0.5624 | 0.7647 |
On the test split, the pairwise ranker improves over raw candidate order and repair confidence. Diagnosis-conditioned structural priors and XGBoost remain strong competitive baselines.
Example Usage
With AgentBlackBox installed:
python -m agentblackbox.cli rank \
examples/case_studies/rag_mixed_trace_qwen_success/trace_cli.json \
examples/case_studies/rag_mixed_trace_qwen_success/repairs_cli.jsonl \
--ranker qwen_pairwise \
--model-name-or-path Qwen/Qwen3-8B \
--adapter-dir Eyerf/agentblackbox-qwen3-8b-rag-repair-ranker-lora \
--aggregation log_odds \
--output ranking.json
Limitations
- The adapter was trained and evaluated primarily on RAG/document-recall repair traces.
- It should not be treated as a general-purpose ranking model for all agent failures.
- The adapter relies on the AgentBlackBox repair schema and prompt format.
- Absolute pairwise scores are less important than the induced repair ranking.
Citation
Please cite the AgentBlackBox project if this adapter is useful for your research.
The replay-labeled RAG repair outcome dataset is available on HuggingFace:
Eyerf/agentblackbox-rag-repair-outcomes
It includes failed RAG traces, candidate repairs, counterfactual replay labels, pairwise repair-ranking data, evaluation outputs, and final result tables.