Results


Table with columns: Benchmark, Qwen3.8-27B, Qwen3.8-27B, thinking, Qwen3.8-27B-hermes-adapted, Qwen3.8-27B-hermes-adapted, thinking| Benchmark | Qwen3.8-27B | Qwen3.8-27B, thinking | Qwen3.8-27B-hermes-adapted | Qwen3.8-27B-hermes-adapted, thinking |
|---|
| SWE-bench Pro (hard 51) | 29.4% | 45.1% | 43.1% | 51.0% |
| Terminal-Bench 2.1 | 44.7% | 58.8% | 58.8% | 57.6% |
The four runs are compared on the tasks all of them graded: all 51 of SWE-bench Pro, and 85 of Terminal-Bench 2.1.
"Thinking" is the chat template's thinking mode at its default reasoning effort (xhigh). Thinking-on scores under Hermes
Agent are limited by the context window: see the note below.
One attempt per task, thinking off, paired against the base model under Hermes Agent on the same tasks. The interval is
95%; the p-value is an exact sign test on the tasks that changed outcome.
Table with columns: Benchmark, Tasks, Qwen3.8-27B, Qwen3.8-27B-hermes-adapted (step 160), Difference (points), Newly solved / lost| Benchmark | Tasks | Qwen3.8-27B | Qwen3.8-27B-hermes-adapted (step 160) | Difference (points) | Newly solved / lost |
|---|
| SWE-bench Pro (hard 51) | 51 | 29.4% | 43.1% | +13.7 ± 13.3 | 10 / 3 (p = 0.092) |
| Terminal-Bench 2.1 | 85 | 44.7% | 58.8% | +14.1 ± 11.4 | 19 / 7 (p = 0.029) |
- SWE-bench Pro (hard 51): the 51 hardest tasks of SWE-bench Pro, 2 h and 100 agent steps per task.
- Terminal-Bench 2.1: all 89 tasks of Terminal-Bench 2.1, 1 h and 100 agent steps per task. Four tasks are left out of the comparison because their environment failed to build or their verifier timed out for one of the two runs.
Medians per task, this adapter / the base model, over the tasks both solved: same tasks, same outcome, so the numbers
measure only how much work each needed. Generated tokens are everything the model writes (visible content, reasoning and
tool calls).
Table with columns: Tasks, Turns, Tool calls, Generated tokens | Tasks | Turns | Tool calls | Generated tokens |
|---|
| vs the base model | | | | |
| SWE-bench Pro | 12 | 22 / 59 | 28 / 63 | 4.1k / 8.5k |
| Terminal-Bench 2.1 | 31 | 8 / 21 | 7 / 23 | 1.6k / 3.5k |
| vs the base model with thinking | | | | |
| SWE-bench Pro | 16 | 24 / 42 | 28 / 50 | 5.5k / 22.9k |
| Terminal-Bench 2.1 | 38 | 8 / 10 | 8 / 12 | 2.1k / 7.4k |
- Against the base model, the adapter reaches the same answers in well under half the turns and tool calls: fewer,
more targeted actions instead of long exploration.
- Against the base model with thinking, most of the difference is reasoning: the adapter gets there without it.
- Hermes Agent's traces do not keep the model's reasoning, so this table does not split generated tokens into content
and reasoning.
Table with columns: SWE-bench Pro (hard 51), Terminal-Bench 2.1 | SWE-bench Pro (hard 51) | Terminal-Bench 2.1 |
|---|
| This adapter vs the base model with thinking (tasks only one solves) | 6 / 7 | 12 / 12 |
| Generated tokens per task (median), base model with thinking / this adapter | 22.5k / 6.5k | 21.0k / 4.8k |
| Model calls per task (median), base model with thinking / this adapter | 30 / 25 | 15 / 17 |
| This adapter with thinking vs without (tasks only one solves) | 8 / 4 | 9 / 10 |
Under this setup, thinking mode on its own brings the base model to this adapter's accuracy, within noise; the adapter
gets there without it, at 3.5–4.4× fewer tokens. Thinking runs use the template's default reasoning effort (xhigh) and
at most 32,768 output tokens per model call.
All models were evaluated with a 131,072-token context window, under Hermes Agent's default configuration, with no
context compaction set up. A session that outgrew the window ended there and was scored as a failure (0), for every model
alike. With thinking on, Hermes Agent sends each turn's reasoning back to the model, so long sessions run out of
context: many failed thinking-on attempts, for the base model and for this adapter, ended this way, often after a few
dozen steps. Thinking-on scores here reflect this setup, not the best each model could reach with a larger window or
compaction. Thinking-off runs were much less affected.
- Infrastructure. During evaluation, the public tunnel between the sandboxes and the model server went down a few
times, failing rollouts for reasons unrelated to the model. We check every rollout for this and reran all affected
ones, on both sides; the numbers above are the corrected ones. The first release of this adapter, as
harness-hub/Qwen3.8-27B-hermes, reported 27.5% → 49.0% on SWE-bench Pro before these reruns.
- Budget. The base model often works until it runs out of its 100 model calls: on SWE-bench Pro, 14 of its 36 failed
attempts ended this way. The budget is the same for both, so the comparison is fair, but part of the gain is finishing
the work within it.
- Network access. The sandboxes had internet access. Calls outside the task were almost all dependency lookups (a
module path or version a task needed); in the trajectories we kept, we found no attempt by this adapter to look up a task's upstream fix.
- Test files. Agents, the base model and this adapter alike, sometimes write or edit test files while working,
mostly their own scratch scripts to check a change. This cannot affect a grade: SWE-bench Pro's grader resets all test
files and applies the official tests before running them, and Terminal-Bench's tests are copied in only at grading.
We found no tampering with tests or the verifier.
Evaluation trajectories
Every evaluation attempt behind this card, solved or not, is in the
harness-hub/harness-adapters-trajectories
bucket: base model and adapter, thinking on and off, on both benchmarks, with each attempt's grade, trajectory, harness
log and verifier output, scrubbed of secrets. The files carry canary strings: keep them out of training data. One gap:
for this adapter on SWE-bench Pro with thinking off, the grades of all 51 tasks are there but only 3 trajectories, the
rest were lost when a partial rerun overwrote the archive.
Use with Hermes Agent
1. Serve the adapter with vLLM
vllm serve Qwen/Qwen3.8-27B \ --enable-lora --max-lora-rank 32 \ --lora-modules qwen3.8-hermes=harness-hub/Qwen3.8-27B-hermes-adapted \ --language-model-only --enable-prefix-caching --mamba-cache-mode align \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}' \ --max-model-len 131072
This is the configuration of the evaluations. Keep thinking off, as in training: with it on, the adapter is not
significantly more accurate and generates about 3× more tokens. Prefix caching on this hybrid model needs
--mamba-cache-mode align; without it, every agent turn re-reads its whole history.
2. Point Hermes Agent at it
Install Hermes Agent:
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
Add the server to ~/.hermes/config.yaml:
model: default: qwen3.8-hermes provider: custom base_url: http://localhost:8000/v1 api_key: dummy
Hermes Agent needs at least 64,000 tokens of context.
hermes chat -q "Find out why the tests in tests/test_parser.py fail and fix the bug."
transformers serve is a lightweight OpenAI-compatible
server, for local use and experimentation (vLLM is faster under load). It handles Qwen's tool calls out of the box.
Serve the merged checkpoint from Merge the adapter into the base model:
pip install "transformers[serving]"transformers serve ./Qwen3.8-27B-hermes-merged --dtype bfloat16 --continuous-batching --reasoning off
--reasoning off keeps thinking off, as in training. The server listens on http://localhost:8000, so the Hermes Agent
configuration from the vLLM section works as is, with default set to the name the server serves,
here ./Qwen3.8-27B-hermes-merged.
Use with Hermes Agent (llama.cpp)
The GGUF version of Qwen3.8-27B-hermes-adapted is in harness-hub/Qwen3.8-27B-hermes-adapted-GGUF. It runs on top of a GGUF of the base model:
hf download ggml-org/Qwen3.8-27B-GGUF Qwen3.8-27B-Q8_0.gguf --local-dir .hf download harness-hub/Qwen3.8-27B-hermes-adapted-GGUF Qwen3.8-27B-hermes-lora-F16.gguf --local-dir . llama serve -m Qwen3.8-27B-Q8_0.gguf --lora Qwen3.8-27B-hermes-lora-F16.gguf \ --chat-template-kwargs '{"enable_thinking": false}' -c 65536 --port 8080 --alias qwen3.8-hermes
Then use the Hermes Agent configuration from the vLLM section with base_url: http://localhost:8080/v1.
The adapter was trained on the BF16 weights: the closer the base GGUF is to them, the closer you get to the trained
behavior. Q8_0 is a good default; lower-bit quants drift further. The GGUF version has not been re-evaluated.
Merge the adapter into the base model
The adapter can be merged into the base weights with PEFT. The result is a regular Qwen3.8-27B checkpoint (about 54 GB
in BF16) that any engine serves without LoRA support. Merge on CPU if the GPU is needed for serving afterwards:
import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.8-27B", dtype=torch.bfloat16, device_map="cpu")model = PeftModel.from_pretrained(base, "harness-hub/Qwen3.8-27B-hermes-adapted").merge_and_unload()model.save_pretrained("Qwen3.8-27B-hermes-merged")AutoTokenizer.from_pretrained("harness-hub/Qwen3.8-27B-hermes-adapted").save_pretrained("Qwen3.8-27B-hermes-merged")
Then serve it like the base model, without the LoRA flags:
vllm serve Qwen3.8-27B-hermes-merged --served-model-name qwen3.8-hermes \ --language-model-only --enable-prefix-caching --mamba-cache-mode align \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}' --max-model-len 131072
The merged checkpoint can also be converted to GGUF and served with llama.cpp the same way.
Training
Table | |
|---|
| Method | AsyncGRPO (TRL, experimental), LoRA |
| Environment | Harbor tasks served by OpenEnv, rollouts in HF sandboxes, tool calls screened by safeagents |
| Harness | Hermes Agent |
| Tasks | 750 scale800 tasks interleaved with 300 TermGrade tasks |
| Reward | the task verifier's correctness (falls back to reward), plus a tool-efficiency bonus of weight 0.1 |
| Steps | 160 (this checkpoint) of 1000 planned |
| Learning rate | 2e-5 |
| LoRA | r = 32, alpha = 64, on q, k, v, o, gate, up and down projections |
| Rollouts | 8 per task, temperature 0.8, thinking off, up to 50 agent steps and 1,800 s per rollout |
| Batch | 16 micro-batches of up to 65,536 tokens per step, rollouts up to 16 policy versions stale |
| Hardware | 2× NVIDIA B200 per harness: vLLM on one, the trainer on the other |
The training script is
examples/async_grpo_harbor in TRL.
For a full write-up, please refer to the release blogpost.