geoguesser-qwen3.5-4b-grpo-v3 API & Inference Endpoint

What it answers

Run 1's training dynamics looked alarming: entropy collapsed, the reward spread inside each group went to almost nothing, and grad norm peaked above 11. Run 2 was designed to suppress exactly that (scale_rewards="none", beta=0.02, two tasks per optimizer step). It trained cleanly and gained a fifth as much. Run 3 reverted only the first two of those settings.

Table with columns: run 1, run 2, run 3 (this)
	run 1	run 2	run 3 (this)
`scale_rewards`	`group`	`none`	`group`
`beta`	0	0.02	0
tasks per step	1	2	2
action cost scale	1.0	0.2	0.2
median within-group spread	0.016	0.193	0.078
peak grad norm	11.25	0.16	6.77
paired gain over its own base	+0.1620	+0.0326	+0.0717

So the instability was not a bug to suppress: it was where most of the learning came from. Run 3 recovers about 44% of run 1's gain by putting it back, which narrows the cause to those two settings plus the action cost, and is the reason the next experiment is a cost sweep.

Best checkpoint here is step 175 at 0.5526 mean-of-4 (its own base arm scored 0.4809). Scores are on the training reward curve, recomputed from raw distance; see the project README.

Training

Table

base	`Qwen/Qwen3.5-4B`
method	GRPO (TRL), `environment_factory` multi-turn tool calling
LoRA	r=16, α=32, dropout 0.05, on `q/k/v/o_proj`
steps	300, two tasks per optimizer step (`ACCUM=4`), `NUM_GENERATIONS=8`
turns	12 max · image 448 px
optimiser	LR 3e-5, temperature 1.0, , ,

Everything else

The write-up, including why these two settings mattered so much
All four runs on one axis
Code and exact commands

Imagery is Mapillary, CC BY-SA 4.0.

What it answers

Table with columns: run 1, run 2, run 3 (this)
	run 1	run 2	run 3 (this)
`scale_rewards`	`group`	`none`	`group`
`beta`	0	0.02	0
tasks per step	1	2	2
action cost scale	1.0	0.2	0.2
median within-group spread	0.016	0.193	0.078
peak grad norm	11.25	0.16	6.77
paired gain over its own base	+0.1620	+0.0326	+0.0717

Best checkpoint here is step 175 at 0.5526 mean-of-4 (its own base arm scored 0.4809). Scores are on the training reward curve, recomputed from raw distance; see the project README.

Training

Table

base	`Qwen/Qwen3.5-4B`
method	GRPO (TRL), `environment_factory` multi-turn tool calling
LoRA	r=16, α=32, dropout 0.05, on `q/k/v/o_proj`
steps	300, two tasks per optimizer step (`ACCUM=4`), `NUM_GENERATIONS=8`
turns	12 max · image 448 px
optimiser	LR 3e-5, temperature 1.0, , ,

Everything else

The write-up, including why these two settings mattered so much
All four runs on one axis
Code and exact commands

Imagery is Mapillary, CC BY-SA 4.0.

geoguesser-qwen3.5-4b-grpo-v3

README

What it answers

Training

Everything else

Explore FriendliAI today

README

What it answers

Training

Everything else