GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
SlowGuess
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
SlowGuess
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
This repository contains ABForge-Qwen3-8B, the model presented in ABForge: Post-Training for Paper-Grounded Ablation Design. Given a paper's methodology with its ablation content removed, it proposes the ablation objectives the paper should investigate and designs a rigorous experiment plan for each — both from a single checkpoint.
The model is obtained through an SFT → GRPO pipeline on Qwen3-8B, trained throughout on a 1:1 mixture of the two tasks:
data_source → this model| Resource | Link |
|---|---|
| Code | SlowGuess/Abforge_1 |
| Training & evaluation data | SlowGuess/abforge-data |
| SFT checkpoint | SlowGuess/ABForge-Qwen3-8B-SFT |
| RL-only checkpoint | SlowGuess/ABForge-Qwen3-8B-RL |
| Per-paper outputs & judge rationales | outputs/task{1,2}/*/abforge.jsonl in the data repo |
SlowGuess/abforge-data ships a
single table under train/, one row per paper, built by a semi-automated audit-in-the-loop
pipeline over research papers from major ML, NLP and CV venues. Both stages select their rows
from that table by its split columns and mix the two tasks 1:1:
in_sft_task1 (45,961 papers) + in_sft_task2 (36,955)in_rl_task1 + in_rl_task2 (30,000 papers each, disjoint from the SFT pool)The benchmark papers carry no training flag, so they cannot leak into either stage. The preprocessing scripts in the code release turn the table into the training parquets directly.
AblationBench, automated rubric-based LLM-as-a-Judge evaluation (eval/ablationbench_200.jsonl,
200 papers, judge claude-sonnet-4-6). Task 1 is ablation objective identification
(paper_score); Task 2 is ablation plan synthesis (design_score, ×100).
| Model | Task 1 | Task 2 |
|---|---|---|
| Qwen3-8B (base) | 44.4 | 43.4 |
| ABForge-Qwen3-8B-SFT (SFT only) | 30.7 | 52.2 |
| ABForge-Qwen3-8B-RL (RL only) | 52.2 | 54.9 |
| ABForge-Qwen3-8B (SFT → GRPO) | 55.9 |
Each stage on its own falls short: SFT trades Task 1 for Task 2, and GRPO from the base model plateaus below the full pipeline. Running GRPO from the SFT checkpoint is worth +3.7 on Task 1 and +7.5 on Task 2 over RL alone.
Reproduce the numbers above with the code release:
bash
git clone https://github.com/SlowGuess/Abforge_1 && cd Abforge_1huggingface-cli download SlowGuess/abforge-data --repo-type dataset \--include "eval/*" --local-dir datapython run_inference_local.py --task 1 \--input data/eval/ablationbench_200.jsonl \--output outputs/task1_infer.jsonl \--model-path SlowGuess/ABForge-Qwen3-8B \--dtype bf16 --device-map auto \--max-new-tokens 5120 --temperature 0.0 --stop-on '</Result>'export JUDGE_API_BASE=https://api.openai.com/v1export JUDGE_API_KEY=...export JUDGE_MODEL=...scripts/evaluate_task1.sh outputs/task1_infer.jsonl
Swap --task 2, --stop-on '</Proposed_Plan>' and scripts/evaluate_task2.sh for Task 2. The
model is trained on the prompt templates in the code release and the rubric evaluator expects
the matching output structure, so use those templates and greedy decoding.
bibtex
@misc{abforge2026,title={ABForge: Post-Training for Paper-Grounded Ablation Design},author={TODO},year={2026},}
| 62.4 |