Run it
docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai:v0.10.2 \ --model smoky1496/SpawnPoint-Coder-7B-AWQ --served-model-name SpawnPoint-Coder-7B --max-model-len 16384 --gpu-memory-utilization 0.85
Then Spawn Point finds it automatically at http://localhost:8000/v1.
Evaluation
Same prompts, schemas, normalizer and server for both versions. The sealed set for the new game kinds was written
before training and not looked at until the release decision. Human-written prompts are English and Vietnamese.
Table with columns: benchmark, v3, v4 (this release)| benchmark | v3 | v4 (this release) |
|---|
| sealed set for the new game kinds (12 prompts), fully right | 50% | 67% |
| 16 human-written prompts for the new game kinds, fully right | 50% | 75% |
| held-out briefs for the new game kinds (exact) | 32% | 91% |
| sealed set for the older game kinds (12 prompts), fully right | 67% | 67% |
| 30 human-written prompts, fully right | 87% | 87% |
| 12 prompts for bubble / rescue / survivor games, fully right | 75% | 92% |
| held-out brief (exact) | 85% | 81% |
| held-out tweak (exact) | 84% | 100% |
| held-out entity (exact) | 100% | 100% |
| held-out translate (exact) | 97% | 97% |
The 4-bit AWQ files you download, measured the same way (fully right):
Table with columns: benchmark, v3 AWQ, v4 AWQ| benchmark | v3 AWQ | v4 AWQ |
|---|
| sealed set, new game kinds | 33% | 75% |
| 16 prompts, new game kinds | 44% | 88% |
| 30 human-written prompts | 90% | 90% |
Limitations
- Fine-tuned only for the 2D game kinds Spawn Point builds: platformers (including rescue missions and sword
fighting), top-down arenas and survivors, bubble shooters, farming games and tower defense. Other genres (3D,
RPG systems, card games, racing…) and free-form Godot coding were not part of the training data, so expect results
no better than a general 7B coder model there.
- It writes Spawn Point's JSON formats; the GDScript itself comes from Spawn Point's tested component
library, not from the model. Outside Spawn Point it is a general coding model with a narrow specialty.
- Occasionally one detail of a request is missed (e.g. a pickup or one enemy type); Spawn Point's
validator and normalizer catch most of these. On the older held-out briefs v4 scores a little below v3 (81% vs 85%
exact), mostly on the unnamed default pickup.
License
Apache 2.0, see LICENSE and NOTICE.