lxazjk

qwen2.5-1.5b-24game-grpo

README

License: apache-2.0

TL;DR — how to use it

This model is a verifier-guided sampler, not a reliable one-shot solver. Greedy/pass@1 ≈ 0. Its value is that sampling many candidates and letting the verifier pick a correct one works very well. Use temperature ~1.1, sample N, then run the program verifier to select the first valid expression.

Results (100 hardest problems, human-lowest-solved-rate split)

Table with columns: Metric, This model, Note
Metric	This model	Note
greedy / pass@1	0/100	GRPO does not improve one-shot accuracy
sample@16 (per independent run)	~36/100 (max 43)	temp 1.1, top-p 0.95
best-of-128 (single model, verifier-selected)	74/100	128 candidates/problem
union best-of-runs (this + correctness-heavy, 256/problem)	79/100	two GRPO checkpoints merged
adaptive best-of-runs	100/100	escalate sampling only on unsolved; 60,416 total candidates, hardest problem needed 5,376

Diagnostics on first attempts: r1_format = 1.00, numbers_ok = 0.99 — the model reliably learned the format and exact number-usage constraint, but not exact arithmetic search. That is why test-time sampling + verifier selection is the right way to use it.

Countdown (bonus, arbitrary target)

Table with columns: Setting, greedy, sample@16
Setting	greedy	sample@16
this 24-game checkpoint, zero-shot	1/100	11/100
after 50-step target-aware Countdown GRPO	1/100	18/100

Usage

python
import re
from fractions import Fraction
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

name = "lxazjk/qwen2.5-1.5b-24game-grpo"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")

numbers = [2, 4, 7, 7]
prompt = (
    "You solve arithmetic target games. Use each given number exactly once. "
    "Only use +, -, *, / and parentheses. Think briefly, then put the final "
    "expression inside <answer>...</answer>. The answer must be only an "
    "expression, with no equals sign, result value, example, or explanation.\n\n"
    f"Make 24.\nNumbers: {', '.join(map(str, numbers))}\n"
    "Respond in this exact format:\n<think>reasoning steps</think><answer>expression only</answer>"
)

# sample many candidates, then verify
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, do_sample=True, temperature=1.1, top_p=0.95,
                     num_return_sequences=16, max_new_tokens=128)
cands = tok.batch_decode(out[:, ids.input_ids.shape[1]:], skip_special_tokens=True)

def verify(expr, nums, target=24):
    toks = re.findall(r"\d+", expr)
    if sorted(map(int, toks)) != sorted(nums):
        return False
    if re.fullmatch(r"[0-9+\-*/() ]+", expr) is None:
        return False
    try:
        # evaluate with exact rational arithmetic
        val = eval(expr, {"__builtins__": {}})
        return Fraction(val) == target
    except Exception:
        return False

for c in cands:
    m = re.search(r"<answer>(.*?)</answer>", c, re.S)
    if m and verify(m.group(1).strip(), numbers):
        print("solved:", m.group(1).strip()); break

(In the original project the verifier uses a safe AST whitelist + Fraction; the snippet above is a compact stand-in. Never eval untrusted strings in production.)

Training

Base: Qwen/Qwen2.5-1.5B-Instruct
SFT warmup (format + number usage), then GRPO with a rule-based reward set (format / structure / expression / number-usage / closeness / correctness).
Verifier: extract last <answer>, restrict characters, check number multiset, parse via a safe AST whitelist, evaluate with Fraction, compare to target.
Data: nlile/24-game (train), test-time-compute/game-of-24 (hard eval), Jiayi-Pan/Countdown-Tasks-3to4 (bonus).

See results/ in this repo for the full per-setting evaluation summaries (REPRODUCTION_SUMMARY.md, greedy / sample / union / adaptive / Countdown JSONs).

Limitations

Not a one-shot solver — high solve rates require substantial test-time sampling.
The hard-eval number combinations overlap the training set, so this is a hard eval, not a strict OOD generalization test.
100/100 is a test-time sampling + verifier-selection result, not pass@1.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

lxazjk

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities