Highlights
- Orchestration beats scale. A trained 4B agent outperforms much larger API
models (up to 235B) given the identical tool interface.
- Having tools ≠ using tools. Qwen3-235B calls the exact same tools yet
scores 0.00 — tool access does not confer tool use.
- Calibrated, not reckless. The agent names a species when the evidence
supports it and gracefully degrades to genus/family when it does not;
species-declaration precision rises to 1.0 with overclaim = 0.00.
- Three-modality. Image, sound-only, or image+sound (audio is tool-mediated).
- Reproducible & honest. All numbers are Δ-over-base under one identical tool
harness; no leaderboard gaming, no closed-model distillation.
Model at a glance
Table | |
|---|
| Base model | Qwen/Qwen3-VL-4B-Instruct (Apache-2.0) |
| This repo | LoRA adapter (the released checkpoint = GSPO ckpt-200) |
| Parameters | 4B base + LoRA (r=64, α=128, dropout=0.05, bf16) |
| Adapter targets | language-model q/k/v/o/gate/up/down_proj (peft ≥ 0.19) |
| Training | SFT cold-start → on-policy DPO → GSPO (RLVR) |
| Task | agentic fine-grained bird identification with calibrated abstention |
| Modalities | image · sound · image+sound |
| License | Apache-2.0 (adapter); base is Apache-2.0 |
⚠️ This repository is the policy only. To run the full agent you also need
the tool servers (Grounding-DINO, BioCLIP-2, Perch-2, SINR) and the evaluation
harness — see the code repository.
Loaded standalone, the model emits Hermes-format <tool_call> turns that
expect tool responses to be fed back; it is not a plain image→label classifier.
Results
Two self-built, tier-stratified benchmarks — an agentic (information-gap) set
and a calibration set — evaluated under an identical tool harness for
every model. Metric = solve: correct at the declared grain (a genus verdict
counts iff the genus is right; over-committed species are penalized).
Which stage is in this table? This release is the final GSPO
checkpoint. The head-to-head comparison below was measured at the SFT and
DPO stages; the final GSPO stage was evaluated separately as a clean
apples-to-apples ablation against its DPO initialization (it was not re-run
against every API baseline on the full sets). GSPO improves on DPO, so the
DPO row (0.34) is a conservative floor for this released model — the extra
GSPO gain is shown in the Training recipe figures below (right panel).
Table with columns: #, model / condition, common, uncommon, rare, overall, n| # | model / condition | common | uncommon | rare | overall | n |
|---|
| 1 | Opus · bare | 0.75 | 0.25 | 0.38 | 0.46 | 24 |
| 2 | Opus · web | 0.62 | 0.25 | 0.25 | 0.38 |
Big models: C0 = bare, N = vendor web search, C1 = our exact tools. API baselines
run on cost-bounded stratified subsets; our models on the full sets. Vendor web
search does not help image ID (C0 ≈ N).
Quickstart
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
base = "Qwen/Qwen3-VL-4B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(
base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "Chinzhu/BirdAgent-Qwen3VL-4B")
processor = AutoProcessor.from_pretrained(base)
How it works
Table with columns: group, tools| group | tools |
|---|
| Perception (local, deterministic) | detect_bird (Grounding-DINO) · quality_gate · zoom_in · crop · enhance · audio_quality |
| Recognition (served models) | classify_image (BioCLIP-2) · classify_sound (Perch-2) |
| External | geo_prior (SINR range × month) |
The agent plans a sequence of calls, accumulates the returns in an explicit
evidence ledger, and commits a verdict at a calibrated taxonomic grain.
All tools are Apache/MIT-licensed and deterministic given input (so their outputs
can be pre-cached for single-GPU RL).
Training recipe (SFT → DPO → GSPO)
- SFT cold-start on code-authored blueprint tool-use trajectories (loss on
assistant + tool-call tokens only; tool observations masked).
- On-policy DPO on soft preferences only (ledger discipline, call parsimony)
— sharpens calibration (species-declaration precision 0.33 → 1.0).
- GSPO (sequence-level importance weighting;
β=0.04, lr=1e-6,
num_generations=4) with a grain-graded, reachability-aware reward:
species +1.0/−0.5, genus +0.4/−0.3, family +0.2/−0.1, abstain 0, with
full species credit gated on the truth being present in some tool's top-k.
This released checkpoint (GSPO-200) improves over its DPO initialization in
a matched apples-to-apples comparison — agentic 0.20 → 0.23 (common
0.22 → 0.33), calibration 0.45 → 0.47 (uncommon 0.31 → 0.39), pooled
≈ 0.325 → 0.35 — while keeping overclaim = 0.00 (bolder but not
reckless). The gain saturates by ~50 steps; what remains is the tool ceiling.
Limitations & responsible use
- Tool ceiling. Species accuracy is capped by the classifiers: on hard
rare-tier items ~76% of errors are cases where the truth is absent from every
classifier's top-k. A stronger fine-grained recognizer is an orthogonal
lever (the agent already degrades honestly in this regime).
- General-VLM regression (format lock). Agentic training locks the model into
emitting tool calls; on generic MCQ probes it drops sharply (MMStar 0.51→0.04,
MMBench 0.89→0.15). Use it as a bird agent, not a general VLM, unless you mix
general trajectories back in.
- Evaluation n. API baselines were run on cost-bounded stratified subsets
(n = 12–40); numbers are Δ-over-base under one harness, not leaderboard ranks.
- Responsible use. Research / non-commercial for v0. The API models above are
evaluation controls, not teachers — no closed-model output was distilled
into this policy.
Citation
@inproceedings{wang2026birdagent,
title = {BirdAgent: A Small Vision--Language Model that Orchestrates
Domain Tools Beats Large Models that Merely Hold Them},
author = {Wang, Xinzhu},
booktitle = {Under review},
year = {2026}
}
Acknowledgements
Built on Qwen3-VL,
BioCLIP-2,
Perch-2,
Grounding-DINO, and
SINR. Trained with
ms-swift.