benchflow/benchflow-qwen35-9b API & Inference Endpoint

Release Summary

Table
Field	Value
Release tag	`v0.0.1`
Adapter repo	`benchflow/benchflow-qwen35-9b`
Base checkpoint	`Qwen/Qwen3.5-9B`
Base checkpoint form	Full, non-quantized source checkpoint; frozen during LoRA SFT
Adapter type	LoRA / PEFT
Source completed run	`general-agent-qwen35-9b-sft-seq2048-fresh-20260624T131847Z`
W&B project	`general-agent-qwen35-9b-sft-seq2048-fresh-20260624T131847Z`
HF training artifacts	`benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-9b-sft-seq2048-fresh-20260624T131847Z`
Published at	`2026-06-24 22:27:07 UTC`

Research Reproduction Scope

The goal of this adapter is to reproduce the SFT-stage lift from Prime Intellect's general-agent work as closely as possible while using a smaller student model that can train on one H100. The stack keeps the Prime-style task and verifier path:

Source tasks: open-source PrimeIntellect-ai/research-environments/environments/general_agent task corpus.
Teacher trace generation: general-agent-solver-rlm + Azure GPT-5.4-mini through native Verifiers / vf-eval --save-results artifacts.
SFT trainer: Prime-RL SFT.
Student: full, non-quantized Qwen/Qwen3.5-9B loaded in BF16 with LoRA adapters.
Eval: general-agent-solver-local through native vf-eval --save-results on the same held-in task sets before and after SFT.

Data Recipe

Table
Field	Value
Dataset	`benchflow/general-agent-qwen35-9b-azure-gpt54mini-sft`
Dataset rows	`4414`
Original source task count	`4417`
Teacher model	Azure GPT-5.4-mini
Teacher harness	Prime/Verifiers `general-agent-solver-rlm`
Artifact format	Native `vf-eval --save-results` trajectories converted to Prime-RL `messages` + `tool_defs` SFT rows
Excluded source tasks	`dog_breeding_t1`, `skydiving_center_t1`, `skydiving_center_t2`
Exclusion reason	Stable Azure content-filter blocks during teacher trace generation
Full teacher sweep artifact	`benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-daytona-teacher-full4417-tunnel8-20260624T015706Z`
Data validation	Prime SFT JSONL validator rejected non-leading system messages and leakage fields before training

Training Parameters

Table
Field	Value
Trainer	Prime-RL SFT
Model loaded for SFT	`Qwen/Qwen3.5-9B` full BF16 base weights
Quantization	None for the completed `v0.0.1` LoRA run
Adapter	LoRA
LoRA rank	`16`
LoRA alpha	`32`
LoRA dropout	`0.0`
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Trainable params	about `29.1M`
Adapted base params	about `5.30B`
Total base params loaded	about `9.44B`
Sequence length	`2048`
Global batch size	`8`
Micro batch size	`1`
Pack function	`cat`
Shuffle	`true`
Seed	`0`
Optimizer	`AdamW`
Learning rate	`5e-5`
Weight decay	`0.01`
Betas	`0.9`, `0.999`
Grad norm clip	`1.0`
Scheduler	Linear
Warmup steps	`20`
Decay steps	`180`
Minimum LR	`0.0`
Max steps	`200`
Checkpoint interval	`20`
Keep last	`3`
Keep interval	`100`
Save format	`safetensors`
Loss mask	Assistant messages only; system, user, and tool messages are context-only

Training Result

Table
Metric	Value
Completed step	`200`
Final loss	`0.11897`
`loss/nan_count`	`0`
Peak GPU memory	about `40.8 GiB`
Final adapter	`adapter_model.safetensors` in this repo

The initial data.seq_len=8192 Prime-RL BF16 LoRA attempt OOMed on one H100. The completed v0.0.1 run used data.seq_len=2048, system CUDA 12.8 nvcc/ptxas, and g++-12 for the required FLA/TileLang kernels.

Evaluation Results

All evaluations below use native Verifiers vf-eval --save-results, general-agent-solver-local, serving context length 4096, --enable-auto-tool-choice, and --tool-call-parser qwen3_xml. Dynamic vLLM LoRA loading was not reliable for this stack, so eval served a merged local checkpoint built from this adapter plus Qwen/Qwen3.5-9B.

Table
Task set	Base pass rate	LoRA SFT pass rate	Delta	Notes
Held-in 5 smoke	`1/5 = 20.00%`	`2/5 = 40.00%`	`+20.00 pp`	First serving/eval smoke
Held-in 20	`11/20 = 55.00%`	`13/20 = 65.00%`	`+10.00 pp`	Recovered `3d_print_shop_t1`, `accounting_firm_t1`
Held-in 36	`20/36 = 55.56%`	`23/36 = 63.89%`	`+8.33%`	No regressions; recovered `3d_print_shop_t1`, `accounting_firm_t1`, `allergy_clinic_t0`
Held-in 50 assembled	`27/50 = 54.00%`	`30/50 = 60.00%`	`+6.00%`	Latest wider held-in result; final 14-task slice had no net delta
Held-in 50 final 14-task slice	`7/14 = 50.00%`	`7/14 = 50.00%`	`+0.00%`	Recovered `animation_studio_t0`; regressed `antiquarian_bookshop_t0`

Evaluation artifact prefixes:

Held-in 5 smoke: benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-smoke4096-20260624T152150Z
Held-in 20 comparison: benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-heldin20-compare-20260624
Held-in 36 comparison: benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-heldin36-compare-20260624
Held-in 50 final 14-task run: benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-heldin50-gap-20260624T190517Z

Loading

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-9B",
    torch_dtype="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, "benchflow/benchflow-qwen35-9b")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B", trust_remote_code=True)

Caveats

This is an SFT-stage reproduction artifact, not the full Prime paper recipe with the original teacher and student model stack.
The trainable dataset has 4414 rows rather than 4417 because three Azure teacher prompts were blocked by content filtering.
The latest held-in50 assembled lift is positive but modest at +6.00 pp; gains are concentrated in a small number of tasks rather than broad across-the-board recovery.
The next QLoRA seq8192 experiment is excluded from v0.0.1 and should receive its own update/tag only after it completes.

benchflow-qwen35-9b

Get help setting up a custom Dedicated Endpoints.

README