Tasks
Given the ablation-free context of a research paper, this one model handles both:
- Task 1: Ablation Objective Generation — propose candidate ablation objectives, each
expressed as a Target Module (the component to ablate) paired with a Research Question
it is meant to answer.
- Task 2: Ablation Experiment Design — produce a concrete, executable ablation experiment
plan (variants, controls, datasets, metrics, expected outcomes) for a given objective.
Training data
Mixed-task SFT on train/sft_task1_45961.jsonl + train/sft_task2_37019.jsonl, then mixed-task
GRPO on train/RL_task1_30K.jsonl + train/RL_task2_30K.jsonl, from
SlowGuess/abforge-data
(derived from CC-licensed research papers). Both stages use a 1:1 task mixture, and during RL
each rollout is routed to its task-specific reward by data_source. Evaluation uses the
held-out AblationBench split of the same dataset (eval/ablationbench_1000.jsonl).
Results
Full AblationBench, automated rubric-based LLM-as-a-Judge evaluation:
Table with columns: Model, Task 1 (%), Task 2 (%)| Model | Task 1 (%) | Task 2 (%) |
|---|
Qwen/Qwen3-8B (base) | 44.4 | 43.4 |
| ABForge-Qwen3-8B-Combined (this model) | 55.9 | 62.4 |
The unified model surpasses the task-specific specialist on Task 2 while consolidating both
capabilities into a single checkpoint; see the paper's ablation table for the task-specific
comparison.
Task-specific specialists (ablation of task sharing):
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SlowGuess/ABForge-Qwen3-8B-Combined"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
Use the Task 1 / Task 2 prompt templates from the code release — the model is trained on those
exact formats and the rubric-based evaluator expects the corresponding output structure.
Evaluation
Reproduce AblationBench evaluation with the
SlowGuess/Abforge_1 code:
git clone https://github.com/SlowGuess/Abforge_1 && cd Abforge_1
huggingface-cli download SlowGuess/abforge-data --repo-type dataset --local-dir data