cwolff

qwen3.5-2b-grpo-curriculum

README

License: apache-2.0

Results

Measured on this repo as published, loaded straight into vLLM, through the same agentic harness for every row: exec_plan protocol, 6 tool turns, T=0.6 / top-p 0.95 / top-k 20, scored by executing against real databases.

Table with columns: benchmark, mode, official, strict, n
benchmark	mode	official	strict	n
BIRD dev	schema in prompt	0.537	0.497	300
BIRD dev	schema withheld	0.473	0.433	300
BIRD dev, challenging only	schema in prompt	0.347	0.313	144
BIRD dev, challenging only	schema withheld	0.326	0.292	144
SQaLe3 test	withheld, 64-table cap	0.573	0.540	300
SQaLe3 test	withheld, full schema	0.583	0.533	300
Spider 2.0-lite	schema in prompt	0.046	—	87
Spider 2.0-lite	schema withheld	0.046	—	87

BIRD and SQaLe3 use BIRD-official set semantics. Spider 2.0 uses its own upstream comparator on its local-SQLite subset (87 of 547 questions), is not numerically comparable to the other two, and is not a leaderboard number.

What the RL bought, on the hardest questions

BIRD tags 145 of its 1,534 dev questions challenging. On all of them, same harness, paired on the question:

Table with columns: model, params, schema shown, withheld
model	params	schema shown	withheld
Qwen3.5-2B (untrained)	2B	0.083	0.132
this model	2B	0.347	0.326
Qwen3.5-9B (untrained)	9B	0.431	0.333
Qwen3.8-27B (untrained)	27B	0.604	0.583

Against its own untrained base the gain is large and unambiguous — paired McNemar p < 0.0001 in both modes. Against an untrained 9B it is behind with the schema shown, but level when the schema is withheld (0.326 vs 0.333, p = 0.74). That is the regime it was trained in (EXPLORE_SHARE=1.0), and it is where a 2B trades evenly with a model 4.5× its size. The 9B's submit rate collapses from 0.34 to 0.06 when it has to discover the schema itself; this model's holds.

Which checkpoint

The run saved a BIRD-selected checkpoint at step 300 and ran to step 600. Both were measured on everything:

Table with columns: benchmark, step 300, step 600 (this), note
benchmark	step 300	step 600 (this)	note
BIRD, schema shown	0.577	0.537	step 300 was selected on this
BIRD challenging, shown	0.319	0.347	neither was selected on this
SQaLe3, full schema	0.540	0.583	step 600 was selected on this
mean tool calls (BIRD shown)	29.1	15.1

Each checkpoint wins the metric it was chosen on — symmetric selection bias, and a reason to trust neither number as a comparison. On the neutral metric the paired difference is +0.028 (95% CI [−0.034, +0.090], p = 0.52): no difference. Step 600 ships because it is equal there and roughly half the inference cost. If you want the higher BIRD-overall number, take step 300 from the run directory; it is not better, it is differently selected.

Does the published repo match what training measured?

Training builds its vLLM engine from the base model and syncs the checkpoint's weights in; this repo is the rebuilt, directly-loadable one. Compared paired over the same 300 questions:

Table with columns: mode, weight-sync path, this repo, paired delta, McNemar p
mode	weight-sync path	this repo	paired delta	McNemar p
schema shown	0.553	0.537	−0.017, 95% CI [−0.050, +0.020]	0.46
withheld	0.523	0.473	−0.050, 95% CI [−0.090, −0.010]	0.020

The withheld row is significant, so state it plainly: on that mode the directly-loaded repo scored 5 points below the weight-sync path. The weights are bit-identical to the training checkpoint — verified tensor by tensor, not inferred — so there is no mechanism for the two to differ, and this is one of two significant results across ten such paired tests on this model family (the other went the opposite way). The most likely explanation is run-to-run sampling variance under continuous batching, which is large for a model that truncates a third of its episodes. It is reported rather than buried; if it matters to you, re-run the eval at a different seed.

Honest caveats

The run did not finish. 1,000 steps were planned; it was cancelled at ~step 656 after 1d19h. This is step 600.
It still over-explores. 15 mean tool calls with the schema shown and 29 withheld — half what step 300 used, but well above the ~13 of its siblings — and 17%/11% of BIRD episodes hit the token cap. On Spider 2.0 truncation is 78%, so most of those episodes never finish.
Spider 2.0 is near zero (0.046 = 4 of 87), inside the noise floor at this n. An untrained Qwen3.8-27B scores 0.414 on the same subset.
BIRD withheld is below its grpo-bird sibling (0.473 vs 0.547).

Serving

bash

vllm serve cwolff/qwen3.5-2b-grpo-curriculum --max-model-len 24576

generation_config.json stops on both <|im_end|> (248046) and <|endoftext|> (248044), matching the stop set used during RL rollouts. Give it a larger context than its siblings — it was trained at a 24,576-token episode budget and uses it.

How to prompt it

It expects an agent loop and will underperform badly without one. Each assistant turn emits a single JSON object and nothing else; your harness executes it and feeds the result back as the next user turn.

json

{"tool": "list_tables"}{"tool": "describe_table", "args": {"table_name": "singer"}}{"tool": "run_query", "args": {"sql": "SELECT COUNT(*) FROM singer"}}{"tool": "submit_sql", "args": {"sql": "SELECT COUNT(*) FROM singer"}}

Tools: list_tables, describe_table, foreign_keys, join_path, sample_rows, run_query, distinct_values, submit_sql. submit_sql ends the episode. Assistant turns open with a <think> block — keep thinking on. The exact system prompt is EXEC_PLAN_TOOL_INSTRUCTIONS in shared/tools/protocol.py; use that string rather than paraphrasing.

Training

Table

Init	`grpo1800_sqale` step 1600 (itself GRPO from stock `Qwen3.5-2B`)
Algorithm	GRPO, token-level loss, sum/1024 aggregation, no advantage normalisation
KL	k3 estimator toward a frozen reference, target 0.03
Clipping	sequence-level importance ratio from vLLM logprobs, 0.2 / 0.28 (DAPO)
Batch	18 questions × 8 candidates = 144 episodes/step, 1 inner epoch
LR	5e-6, cosine, 3% warmup
Curriculum	length quantiles 0.5/0.8/0.95; pool weights 0.40→0.15 (short) and 0.10→0.40 (long) over 400 steps, floor 0.15

Reward executes the candidate against a real SQLite database built from the row payloads and compares to the gold result — not string match against a reference query.

Limitations

2B parameters; it will lose to a frontier model on hard multi-join questions.
Trained and evaluated on SQLite only.
Single-turn "schema in, SQL out" prompting is out of distribution.
Inherits the base model's vision tower, carried over untouched and never trained or evaluated here. Treat it as a text model.
Provenance: Slurm job 26862235, output grpo_curriculum_len, checkpoint checkpoint-step-600 (byte-identical to the run's best/). Exported by merging the run's 320 trained language-model tensors with the base model's vision tower and MTP head under the base config.json.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

cwolff

Model Tree

Base

Qwen/Qwen3.5-2B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality