Intended use
Use the gate after a base solver has produced an attempt. The gate receives the problem, attempted
solution, and serving-visible features; a calibrated threshold decides whether to accept the base
answer or invoke active verification. Gold answers are unavailable to the gate at inference time.
Development metrics
The adapter was trained on MATH recovery outcomes with an 80/20 example-level split.
Table with columns: Metric, Value| Metric | Value |
|---|
| Development accuracy | 0.8925 |
| Development AUROC | 0.9567 |
| Development AUPRC | 0.7601 |
| Frozen operating threshold | 0.090000 |
| Development policy accuracy | 0.9275 |
| Development intervention rate | 0.2525 |
| Development harmful-flip rate | 0.0075 |
These are gate-development results, not standalone solver accuracy. See the paper for MATH500,
GSM8K, CommonsenseQA, budget-matched, and harmful-flip evaluations.
Loading
import torch
from peft import PeftConfig, PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer
repo_id = "Sajib-006/sevra-qwen3-0.6b-gate"
adapter = PeftConfig.from_pretrained(repo_id)
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
base = AutoModelForSequenceClassification.from_pretrained(
adapter.base_model_name_or_path,
num_labels=1,
problem_type="multi_label_classification",
torch_dtype="auto",
device_map="auto",
)
base.config.pad_token_id = tokenizer.pad_token_id
model = PeftModel.from_pretrained(base, repo_id).eval()
Use sevra.format_gate_input from the GitHub package to construct inputs. Apply sigmoid to the
single output logit and use the frozen threshold reported above unless recalibrating on a separate
development workload.
Training configuration
- Objective: binary helpful-fix prediction
- Base model: Qwen/Qwen3-0.6B
- Adapter: LoRA rank 16, alpha 32, dropout 0.05
- Target modules: attention and MLP projection layers
- Epochs: 3
- Maximum sequence length: 1,536
- Class weighting: inverse helpful-fix frequency
- Checkpoint selection: development AUPRC
- Threshold selection: development policy accuracy, then lower action-token cost
Limitations
The gate was trained on English mathematical reasoning produced by a frozen Qwen3-4B solver.
Transfer to other solvers, languages, prompting formats, and workloads should be recalibrated and
audited for harmful flips. Gate scores are recoverability estimates, not correctness probabilities.
Citation
@article{dip2026think,
title={Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning},
author={Dip, Sajib Acharjee and Zhou, Dawei and Zhang, Liqing},
journal={arXiv preprint arXiv:2606.19808},
year={2026}
}
