GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
lxazjk
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
lxazjk
Model Tree
Qwen/Qwen2.5-1.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This model is a verifier-guided sampler, not a reliable one-shot solver. Greedy/pass@1 ≈ 0. Its value is that sampling many candidates and letting the verifier pick a correct one works very well. Use temperature ~1.1, sample N, then run the program verifier to select the first valid expression.
| Metric | This model | Note |
|---|---|---|
| greedy / pass@1 | 0/100 | GRPO does not improve one-shot accuracy |
| sample@16 (per independent run) | ~36/100 (max 43) | temp 1.1, top-p 0.95 |
| best-of-128 (single model, verifier-selected) | 74/100 | 128 candidates/problem |
| union best-of-runs (this + correctness-heavy, 256/problem) | 79/100 | two GRPO checkpoints merged |
| adaptive best-of-runs | 100/100 | escalate sampling only on unsolved; 60,416 total candidates, hardest problem needed 5,376 |
Diagnostics on first attempts: r1_format = 1.00, numbers_ok = 0.99 — the model
reliably learned the format and exact number-usage constraint, but not exact
arithmetic search. That is why test-time sampling + verifier selection is the
right way to use it.
| Setting | greedy | sample@16 |
|---|---|---|
| this 24-game checkpoint, zero-shot | 1/100 | 11/100 |
| after 50-step target-aware Countdown GRPO | 1/100 | 18/100 |
python
import refrom fractions import Fractionfrom transformers import AutoModelForCausalLM, AutoTokenizerimport torch name = "lxazjk/qwen2.5-1.5b-24game-grpo"tok = AutoTokenizer.from_pretrained(name)model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto") numbers = [2, 4, 7, 7]prompt = ( "You solve arithmetic target games. Use each given number exactly once. " "Only use +, -, *, / and parentheses. Think briefly, then put the final " "expression inside <answer>...</answer>. The answer must be only an " "expression, with no equals sign, result value, example, or explanation.\n\n" f"Make 24.\nNumbers: {', '.join(map(str, numbers))}\n" "Respond in this exact format:\n<think>reasoning steps</think><answer>expression only</answer>") # sample many candidates, then verifyids = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**ids, do_sample=True, temperature=1.1, top_p=0.95, num_return_sequences=16, max_new_tokens=128)cands = tok.batch_decode(out[:, ids.input_ids.shape[1]:], skip_special_tokens=True) def verify(expr, nums, target=24): toks = re.findall(r"\d+", expr) if sorted(map(int, toks)) != sorted(nums): return False if re.fullmatch(r"[0-9+\-*/() ]+", expr) is None: return False try: # evaluate with exact rational arithmetic val = eval(expr, {"__builtins__": {}}) return Fraction(val) == target except Exception: return False for c in cands: m = re.search(r"<answer>(.*?)</answer>", c, re.S) if m and verify(m.group(1).strip(), numbers): print("solved:", m.group(1).strip()); break(In the original project the verifier uses a safe AST whitelist + Fraction; the
snippet above is a compact stand-in. Never eval untrusted strings in production.)
Qwen/Qwen2.5-1.5B-Instructformat / structure / expression / number-usage / closeness / correctness).<answer>, restrict characters, check number multiset,
parse via a safe AST whitelist, evaluate with Fraction, compare to target.nlile/24-game (train), test-time-compute/game-of-24 (hard eval),
Jiayi-Pan/Countdown-Tasks-3to4 (bonus).See results/ in this repo for the full per-setting evaluation summaries
(REPRODUCTION_SUMMARY.md, greedy / sample / union / adaptive / Countdown JSONs).
100/100 is a test-time sampling + verifier-selection result, not pass@1.import refrom fractions import Fractionfrom transformers import AutoModelForCausalLM, AutoTokenizerimport torch name = "lxazjk/qwen2.5-1.5b-24game-grpo"tok = AutoTokenizer.from_pretrained(name)model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto") numbers = [2, 4, 7, 7]prompt = ( "You solve arithmetic target games. Use each given number exactly once. " "Only use +, -, *, / and parentheses. Think briefly, then put the final " "expression inside <answer>...</answer>. The answer must be only an " "expression, with no equals sign, result value, example, or explanation.\n\n" f"Make 24.\nNumbers: {', '.join(map(str, numbers))}\n" "Respond in this exact format:\n<think>reasoning steps</think><answer>expression only</answer>") # sample many candidates, then verifyids = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**ids, do_sample=True, temperature=1.1, top_p=0.95, num_return_sequences=16, max_new_tokens=128)cands = tok.batch_decode(out[:, ids.input_ids.shape[1]:], skip_special_tokens=True) def verify(expr, nums, target=24): toks = re.findall(r"\d+", expr) if sorted(map(int, toks)) != sorted(nums): return False if re.fullmatch(r"[0-9+\-*/() ]+", expr) is None: return False try: # evaluate with exact rational arithmetic val = eval(expr, {"__builtins__": {}}) return Fraction(val) == target except Exception: return False for c in cands: m = re.search(r"<answer>(.*?)</answer>", c, re.S) if m and verify(m.group(1).strip(), numbers): print("solved:", m.group(1).strip()); break