Results

Evaluated with the pi harness, one attempt per task at temperature 0.8, thinking off, paired against the base
model under pi on the same tasks. The interval is 95%; the p-value is an exact sign test on the tasks that
changed outcome. Tool calls are compared over every task, and on the tasks both solved (their number in brackets).
Table with columns: Benchmark, Tasks, Base model, This adapter (step 160), Difference (points), Newly solved / lost, Tool calls per task, Tool calls, tasks both solved| Benchmark | Tasks | Base model | This adapter (step 160) | Difference (points) | Newly solved / lost | Tool calls per task | Tool calls, tasks both solved |
|---|
| Terminal-Bench 2.1 | 86 | 39.5% | 55.8% | +16.3 ± 9.1 | 16 / 2 (p = 0.0013) | -62% | -48% (32) |
| Terminal-Bench 2.1, the 68 tasks not used to pick the checkpoint | 68 | 39.7% | 55.9% | +16.2 ± 10.6 | 13 / 2 (p = 0.0074) | -64% | -32% (25) |
| SWE-bench Pro (hard 51) | 51 | 29.4% | 62.7% | +33.3 ± 14.2 | 18 / 1 (p = 7.6e-05) | -55% | -59% (14) |
- Terminal-Bench 2.1: all 89 tasks of Terminal-Bench 2.1, 1 h and 100 agent steps per task.
- SWE-bench Pro (hard 51): the 51 hardest tasks of SWE-bench Pro, 2 h and 100 agent steps per task.
Step 160 was picked by looking at the checkpoints' scores on SWE-bench Pro and on an 18-task subset of Terminal-Bench
2.1, tracked during training.
Use with pi
1. Serve the adapter with vLLM
vllm serve Qwen/Qwen3.8-27B \ --enable-lora --max-lora-rank 32 \ --lora-modules qwen3.8-pi=harness-hub/Qwen3.8-27B-pi \ --language-model-only --enable-prefix-caching --mamba-cache-mode align \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}' \ --max-model-len 131072
This is the configuration of the evaluations. Thinking is off, as in training: the adapter was never trained with it.
Prefix caching on this hybrid model needs --mamba-cache-mode align; without it, every agent turn re-reads its whole
history.
2. Point pi at it
Install pi (Node.js 22.19 or newer):
npm install -g --ignore-scripts @earendil-works/pi-coding-agent
Add the server to ~/.pi/agent/models.json:
{ "providers": { "local": { "baseUrl": "http://localhost:8000/v1", "api": "openai-completions", "apiKey": "dummy", "models": [{ "id": "qwen3.8-pi" }] } }}
"api": "openai-completions" matters: without it pi uses the Responses API, which this adapter was not trained or
evaluated with.
pi --provider local --model qwen3.8-pi --print "Find out why the tests in tests/test_parser.py fail and fix the bug."
transformers serve is a lightweight OpenAI-compatible
server, for local use and experimentation (vLLM is faster under load). It handles Qwen's tool calls out of the box.
Serve the merged checkpoint from Merge the adapter into the base model:
pip install "transformers[serving]"transformers serve ./Qwen3.8-27B-pi-merged --dtype bfloat16 --continuous-batching --reasoning off
--reasoning off keeps thinking off, as in training. The server listens on http://localhost:8000, so the pi
configuration from the vLLM section works as is, with the model id set to the name the server
serves, here ./Qwen3.8-27B-pi-merged:
pi --provider local --model ./Qwen3.8-27B-pi-merged --print "Find out why the tests in tests/test_parser.py fail and fix the bug."
Use with pi (llama.cpp)
The GGUF version of this adapter is in harness-hub/Qwen3.8-27B-pi-GGUF. It runs on top of a GGUF of the base model:
hf download ggml-org/Qwen3.8-27B-GGUF Qwen3.8-27B-Q8_0.gguf --local-dir .hf download harness-hub/Qwen3.8-27B-pi-GGUF Qwen3.8-27B-pi-lora-F16.gguf --local-dir . llama serve -m Qwen3.8-27B-Q8_0.gguf --lora Qwen3.8-27B-pi-lora-F16.gguf \ --chat-template-kwargs '{"enable_thinking": false}' -c 65536 --port 9931 --alias qwen3.8-pi
Then point pi at it with the pi-llama extension:
pi install git:github.com/huggingface/pi-llamapi --model llama-cpp/qwen3.8-pi
The adapter was trained on the BF16 weights: the closer the base GGUF is to them, the closer you get to the trained
behavior. Q8_0 is a good default; lower-bit quants drift further. The GGUF version has not been re-evaluated.
Run it with Llama.app (macOS)
-
Install Llama.app and launch it once. On first launch it creates ~/.config/llama/models.user.ini.
-
Download the adapter and register the model:
LORA=$(hf download harness-hub/Qwen3.8-27B-pi-GGUF Qwen3.8-27B-pi-lora-F16.gguf)cat >> ~/.config/llama/models.user.ini <<EOF [harness-hub/Qwen3.8-27B-pi]hf-repo = unsloth/Qwen3.8-27B-GGUF:Q4_K_XL ; This is just an example but you can also use any GGUF that is derived from Qwen3.8-27B lora = $LORAtemp = 0.7ctx-size = 32768EOF
hf-repo sets the base model: the app downloads it on first use, about 17 GB. To use another quant of the same base, change the part after the colon, for example :Q8_0.
-
Restart Llama.app and choose harness-hub/Qwen3.8-27B-pi from the model list.
You can also use it through the app's OpenAI-compatible server. Llama.app shows the port; the example below uses 9931:
curl localhost:9931/v1/chat/completions -d '{"model":"harness-hub/Qwen3.8-27B-pi","messages":[{"role":"user","content":"hi"}]}'
The adapter must be loaded on top of Qwen3.8-27B. It does not work with other base models.
Merge the adapter into the base model
The adapter can be merged into the base weights with PEFT. The result is a regular Qwen3.8-27B checkpoint (about 54 GB
in BF16) that any engine serves without LoRA support. Merge on CPU if the GPU is needed for serving afterwards:
import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.8-27B", dtype=torch.bfloat16, device_map="cpu")model = PeftModel.from_pretrained(base, "harness-hub/Qwen3.8-27B-pi").merge_and_unload()model.save_pretrained("Qwen3.8-27B-pi-merged")AutoTokenizer.from_pretrained("harness-hub/Qwen3.8-27B-pi").save_pretrained("Qwen3.8-27B-pi-merged")
Then serve it like the base model, without the LoRA flags:
vllm serve Qwen3.8-27B-pi-merged --served-model-name qwen3.8-pi \ --language-model-only --enable-prefix-caching --mamba-cache-mode align \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}' --max-model-len 131072
The similar approach can be also adopted and used to serve with llama.cpp and serve the merged GGUF file.
Training
Table | |
|---|
| Method | AsyncGRPO (TRL, experimental), LoRA |
| Environment | Harbor tasks served by OpenEnv, rollouts in HF sandboxes, tool calls screened by safeagents |
| Harness | pi |
| Tasks | 750 scale800 tasks interleaved with 300 TermGrade tasks; 1,919 task groups trained up to this checkpoint |
| Reward | the task verifier's correctness (falls back to reward), plus a tool-efficiency bonus of weight 0.1 |
| Steps | 160 (this checkpoint); the run was stopped at step 191 of 1000 planned |
| Learning rate | 2e-5 |
| LoRA | r = 32, alpha = 64, on q, k, v, o, gate, up and down projections |
| Rollouts | 8 per task, temperature 0.8, thinking off, up to 50 agent steps and 1,800 s per rollout |
| Batch | 16 micro-batches of up to 65,536 tokens per step, rollouts up to 16 policy versions stale |
| Hardware | 2× NVIDIA B200 per harness: vLLM on one, the trainer on the other |
The training script is
examples/async_grpo_harbor in TRL.
Below is an animated training curve of the evolution of the number of tool calls on the train set:

For a full write-up, please refer to the release blogpost.