Test results
300 test requests (combinations and phrasings not seen in training, 16 of them impossible), greedy decoding, thinking off, scored by the project's verifier (src/verifier.py). all_correct = every check passed; reward = the GRPO reward, in [0, 1].
Table with columns: model, names in catalog, no duplicates, impossible refused, all_correct, reward| model | names in catalog | no duplicates | impossible refused | all_correct | reward |
|---|
| Qwen3.5-2B, zero-shot | – | – | 0/16 | 0.00 | 0.000 |
| Qwen3.5-2B, 3 examples in the prompt | 0.161 | 0.943¹ | 0/16 | 0.00 | 0.415 |
| SFT (this adapter merged: gym-sft, MLX bf16) | 0.750 | 0.405 | 4/16 | 0.24 | 0.785 |
| SFT + GRPO (gym-grpo, MLX bf16) | 0.948 | 0.663 | 6/16 | 0.55 | 0.915 |
| GGUF Q8_0 (gym-grpo-GGUF, llama.cpp) | 0.955 | 0.660 | 6/16 | 0.55 | 0.918 |
¹ High only because few of its names are real catalog names. The zero-shot answers never had the right JSON shape, so their workout columns are empty.
The GymLeader system, not the model: the app wraps the Q8_0 model in a request parser, a guard that answers impossible requests itself, and the verifier, which asks the model again (temperature 0.7) up to MAX_RETRIES=3 times when an answer fails a check. On the same 300 requests it scores 0.82 all_correct and 0.985 reward, with 16/16 impossible requests answered with an error. MAX_RETRIES=3 was fixed before testing, not tuned on the test set, and the system was measured with llama.cpp b12360 on Metal (Apple M4). The model alone is the 0.55 above.
Known limitations
The same as the merged model's: gym-sft, Known limitations. In short:
- 59% of workouts repeat an exercise;
- 74% of listed names are exact catalog names;
- only 4 of 16 impossible requests refused;
- English requests, this one task only;
- one test set, greedy decoding only;
- it suggests workouts; it is not medical advice.
License
Apache-2.0, the license of the base model Qwen/Qwen3.5-2B. The exercise names come from hasaneyldrm/exercises-dataset (MIT).
Links