Qwen2.5-0.5B-Instruct-ES-Countdown API & Inference Endpoint

Results

Held-out evaluation on the first 500 puzzles of the upstream 2,000-puzzle Countdown eval split, greedy decoding, up to 256 new tokens. A puzzle counts as solved when the expression in <answer> uses each given number exactly once and evaluates to the target.

Table with columns: checkpoint, puzzles solved, well-formed <think>/<answer>, median response (tokens)
checkpoint	puzzles solved	well-formed `<think>`/`<answer>`	median response (tokens)
Qwen2.5-0.5B-Instruct (base)	0 %	30 % (training batch)	–
ES iteration 50	3 %	–	–
ES iteration 60	6 %	100 %	45
ES iteration 100	10 %	100 %	41
ES iteration 150	11 %	100 %	41
ES iteration 200 (this model)	12 %	100 %	41

Standard error at n = 500 is about ±1.4 points. ES learns the output format within the first 50 iterations (the format reward is 0.1, the answer reward 1.0, so once format is saturated only solving puzzles moves the score) and then slowly raises the answer rate.

Four-panel training summary

What the model actually learned. The bottom-right panel is the honest part. By iteration 200 half of all held-out answers are literally (a − b + c) with the given numbers in order, and 52 of the 60 correct answers come from that one shape. The model found that a fixed template solves a stable fraction of the three-number puzzles and converged on it: at iteration 200 it solves 52 of the 251 three-number puzzles and 8 of the 249 four-number ones, and the <think> section is one boilerplate sentence. It is a clean illustration of ES climbing an outcome reward with a 0.5 B model and 200 training puzzles, and also of what a reward without diversity pressure selects for.

Every evaluated response is in training/eval-output/ (iterations 60 to 200, every 10), the per-iteration training metrics for iterations 50–200 are in training/metrics.jsonl, and the console log of the second run is training/train.log. The first run (iterations 1–49) kept only its final checkpoint, which the second run resumed from, so its per-iteration file is not included.

Training setting

Data. The upstream datasets/train/countdown set of 200 puzzles (96 with three numbers, 104 with four), walked in order 32 per iteration, so every puzzle was seen about 32 times over the 200 iterations. Evaluation uses the first 500 of the 2,000 puzzles in datasets/evaluation_suite/countdown (251 three-number, 249 four-number).

Prompt. The dataset's context field is the complete prompt, passed through as plain text with no chat template. It ends with Let me solve this step by step.\n<think> so the model continues the reasoning block:

text

You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer.Using the numbers [44 19 35], create an equation that equals 98. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>.Let me solve this step by step.<think>

Reward. 0.1 for a well-formed </think> + <answer>…</answer> response, plus 1.0 when the answer expression uses each number once and evaluates to the target (the upstream countdown_reward_fn).
Decoding. Greedy, at most 256 new tokens, for rollouts and evaluation.
ES update. Population of 30 full-model Gaussian perturbations θ + σ·ε(seed) per iteration, applied in place in bf16 and restored from the seed; rewards z-scored across the population; θ ← θ + α/N · Σ zᵢ εᵢ. Same σ, α, population size and update rule as upstream.

Training parameters

Table with columns: parameter, value
parameter	value
perturbation scale σ	0.001
learning rate α	0.0005 (σ / 2)
population size	30
puzzles per iteration	32
max response length	256 tokens
rollout temperature	0 (greedy)
iterations	200, as two runs: 50, then resumed from the iteration-50 checkpoint to 200
held-out eval	500 puzzles every 10 iterations
precision / device	bf16 on Apple M5 Max via PyTorch MPS, single process
seed	42
wall time	about 3.9 h for iterations 50–200; 190 s per iteration at the start, 45–85 s at the end as responses got shorter

Usage

Transformers

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")  # transformers < 5: torch_dtype= numbers, target = [81, 69, 76], 88prompt = (    "You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer."    f"Using the numbers [{' '.join(map(str, numbers))}], create an equation that equals {target}. "    "You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. "    "Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>."    "Let me solve this step by step.\n<think>")inputs = tokenizer(prompt, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=256, do_sample=False)print(tokenizer.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))# -> First, we need to find a way to combine these three numbers into a single equation that equals 88.</think>#    <answer>(81 - 69 + 76)</answer>

Use the raw prompt above, not apply_chat_template: the model was trained on this exact text.

llama.cpp

GGUF files are in gguf/ (Q8_0 and Q4_K_M).

bash

llama-completion -hf deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown -hff gguf/Qwen2.5-0.5B-Instruct-ES-Countdown-Q8_0.gguf --temp 0 -n 256 -no-cnv \  -p $'You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer.Using the numbers [81 69 76], create an equation that equals 88. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>.Let me solve this step by step.\n<think>'

Quantized greedy outputs differ from the bf16 transformers model on individual puzzles: for the puzzle above, Q8_0 under llama.cpp answers (81 + 69 + 76) where bf16 on MPS answers (81 - 69 + 76). On the first 100 held-out puzzles Q8_0 solves 6 and bf16 solves 7, with only 51 of the 100 answers identical. The model sits close to its decision boundaries, so small numeric differences flip many greedy picks without changing the overall rate much.

-hf downloads the file into llama.cpp's cache on first use; while the repo is private, export HF_TOKEN first.

To rebuild or requantize: python llama.cpp/convert_hf_to_gguf.py <local dir> --outfile model-bf16.gguf --outtype bf16, then llama-quantize model-bf16.gguf model-q5_k_m.gguf Q5_K_M.

Further training

On Apple Silicon (or CPU, or one CUDA GPU) with `mps_es`

bash

git clone -b mps-es-apple-silicon https://github.com/deepsaia/es-at-scale.gitcd es-at-scalepython3 -m venv .venv && source .venv/bin/activatepip install -r mps_es/requirements.txt          # datasets/ ships with the repository python -m mps_es.train_countdown \  --model-name deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown \  --n-iterations 100 --experiment-name countdown-continued python -m mps_es.watch experiments/countdown-continued -f

Defaults match this run (population 30, σ 0.001, 32 puzzles per iteration, 256 tokens, 500 eval puzzles every 10 iterations). Starting from the Hub weights begins a fresh seed stream rather than continuing this run's.

On GPUs with the upstream trainer (vLLM + Ray)

bash

python es_at_scale/train.py --task countdown \  --model-name deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown \  --sigma 0.001 --population-size 30 --n-iterations 1000 --eval-freq 10 \  --train-dataset datasets/train/countdown --eval-dataset datasets/evaluation_suite/countdown \  --batch-size 256 --mini-batch-size 256 --max-tokens 1024 \  --n-vllm-engines 8 --use-gpus "0,1,2,3,4,5,6,7" \  --experiment-name es-countdown-gpu

Limitations

Solves 12 % of held-out puzzles (21 % of the three-number ones, 3 % of the four-number ones), mostly with one memorised expression shape; it does not search over the numbers.
Trained on 200 puzzles with greedy decoding only.
The base instruct model's general chat ability was not evaluated after fine-tuning and may have degraded.
No safety alignment beyond what the base model carried.

Files

Table with columns: path, contents
path	contents
`model.safetensors`, `config.json`, `tokenizer*`, `chat_template.jinja`, `generation_config.json`	the model, bf16
`training_config.json`	ES hyperparameters of the run
`training/metrics.jsonl`, `training/train.log`	per-iteration metrics and console log, iterations 50–200
`training/eval-output/*.json`	every held-out response and its grade, iterations 60–200
`gguf/*.gguf`	Q8_0 and Q4_K_M quantizations for llama.cpp
`assets/*.png`	the figures above, light and dark

Citation

bibtex

@article{Qiu2026EvolutionStrategies,  title={Evolution Strategies at Scale: {LLM} Fine-Tuning Beyond Reinforcement Learning},  author={Qiu, Xin and Gan, Yulu and Hayes, Conor F. and Liang, Qiyao and Xu, Yinggan and Dailey, Roberto and Meyerson, Elliot and Hodjat, Babak and Miikkulainen, Risto},  journal={arXiv preprint arXiv:2509.24372},  year={2026},  eprint={2509.24372},  archivePrefix={arXiv},  primaryClass={cs.LG},  doi={10.48550/arXiv.2509.24372}}

checkpoint

puzzles solved

well-formed <think>/<answer>

median response (tokens)

Qwen2.5-0.5B-Instruct (base)

0 %

30 % (training batch)

–

ES iteration 50

3 %

–

ES iteration 60

6 %

100 %

ES iteration 100

10 %

100 %

ES iteration 150

11 %

100 %

ES iteration 200 (this model)

12 %

100 %

You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer.Using the numbers [44 19 35], create an equation that equals 98. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>.Let me solve this step by step.<think>

parameter

value

perturbation scale σ

0.001

learning rate α

0.0005 (σ / 2)

population size

puzzles per iteration

max response length

256 tokens

rollout temperature

0 (greedy)

iterations

200, as two runs: 50, then resumed from the iteration-50 checkpoint to 200

held-out eval

500 puzzles every 10 iterations

precision / device

bf16 on Apple M5 Max via PyTorch MPS, single process

seed

wall time

about 3.9 h for iterations 50–200; 190 s per iteration at the start, 45–85 s at the end as responses got shorter

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto") # transformers < 5: torch_dtype= numbers, target = [81, 69, 76], 88prompt = ( "You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer." f"Using the numbers [{' '.join(map(str, numbers))}], create an equation that equals {target}. " "You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. " "Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>." "Let me solve this step by step.\n<think>")inputs = tokenizer(prompt, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=256, do_sample=False)print(tokenizer.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))# -> First, we need to find a way to combine these three numbers into a single equation that equals 88.</think># <answer>(81 - 69 + 76)</answer>

llama-completion -hf deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown -hff gguf/Qwen2.5-0.5B-Instruct-ES-Countdown-Q8_0.gguf --temp 0 -n 256 -no-cnv \ -p $'You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer.Using the numbers [81 69 76], create an equation that equals 88. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>.Let me solve this step by step.\n<think>'

git clone -b mps-es-apple-silicon https://github.com/deepsaia/es-at-scale.gitcd es-at-scalepython3 -m venv .venv && source .venv/bin/activatepip install -r mps_es/requirements.txt # datasets/ ships with the repository python -m mps_es.train_countdown \ --model-name deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown \ --n-iterations 100 --experiment-name countdown-continued python -m mps_es.watch experiments/countdown-continued -f

python es_at_scale/train.py --task countdown \ --model-name deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown \ --sigma 0.001 --population-size 30 --n-iterations 1000 --eval-freq 10 \ --train-dataset datasets/train/countdown --eval-dataset datasets/evaluation_suite/countdown \ --batch-size 256 --mini-batch-size 256 --max-tokens 1024 \ --n-vllm-engines 8 --use-gpus "0,1,2,3,4,5,6,7" \ --experiment-name es-countdown-gpu

path

contents

model.safetensors, config.json, tokenizer*, chat_template.jinja, generation_config.json

the model, bf16

training_config.json

ES hyperparameters of the run

training/metrics.jsonl, training/train.log

per-iteration metrics and console log, iterations 50–200

training/eval-output/*.json

every held-out response and its grade, iterations 60–200

gguf/*.gguf

Q8_0 and Q4_K_M quantizations for llama.cpp

assets/*.png

the figures above, light and dark

@article{Qiu2026EvolutionStrategies, title={Evolution Strategies at Scale: {LLM} Fine-Tuning Beyond Reinforcement Learning}, author={Qiu, Xin and Gan, Yulu and Hayes, Conor F. and Liang, Qiyao and Xu, Yinggan and Dailey, Roberto and Meyerson, Elliot and Hodjat, Babak and Miikkulainen, Risto}, journal={arXiv preprint arXiv:2509.24372}, year={2026}, eprint={2509.24372}, archivePrefix={arXiv}, primaryClass={cs.LG}, doi={10.48550/arXiv.2509.24372}}

Qwen2.5-0.5B-Instruct-ES-Countdown

README

Results

Training setting

Training parameters

Usage

Transformers

llama.cpp

Further training

On Apple Silicon (or CPU, or one CUDA GPU) with `mps_es`

On GPUs with the upstream trainer (vLLM + Ray)

Limitations

Files

Citation

Explore FriendliAI today

README

Results

Training setting

Training parameters

Usage

Transformers

llama.cpp

Further training

On Apple Silicon (or CPU, or one CUDA GPU) with `mps_es`

On GPUs with the upstream trainer (vLLM + Ray)

Limitations

Files

Citation

Qwen2.5-0.5B-Instruct-ES-Countdown

README

Results

Training setting

Training parameters

Usage

Transformers

llama.cpp

Further training

On Apple Silicon (or CPU, or one CUDA GPU) with mps_es

On GPUs with the upstream trainer (vLLM + Ray)

Limitations

Files

Citation

Explore FriendliAI today

README

Results

Training setting

Training parameters

Usage

Transformers

llama.cpp

Further training

On Apple Silicon (or CPU, or one CUDA GPU) with mps_es

On GPUs with the upstream trainer (vLLM + Ray)

Limitations

Files

Citation

On Apple Silicon (or CPU, or one CUDA GPU) with `mps_es`

On Apple Silicon (or CPU, or one CUDA GPU) with `mps_es`