Evaluation
40 held-out trajectories: the 40 that follow the training set in the same shuffled order.
Table with columns: base, + LoRA | base | + LoRA |
|---|
| Assistant-token loss | 0.412 | 0.189 |
Next-action test through vLLM's OpenAI-compatible API. Each trajectory is cut just before a random
tool call, and the model picks the next action. Tools are bash, str_replace_editor and submit;
temperature is 0.
Table with columns: base, + LoRA | base | + LoRA |
|---|
| Returns a tool call | 97.5% | 100% |
| Arguments are valid JSON | 97.5% | 100% |
| Same tool as the reference | 65% | 97.5% |
Same str_replace_editor sub-command | 32% | 88% |
| Request errors | 0 | 0 |
| Has reasoning text | 100% | 100% |
| Median latency | 3.0 s | 3.8 s |
Reasoning is kept. Unlike a harmony-format LoRA on gpt-oss-20b trained on the same data, which
learned to skip its reasoning, this adapter keeps writing a short thought before every tool call. The
template keeps <think> on every turn, so the trajectories' reasoning was trained too.
SuperQwen's own tools. On three tasks with SuperQwen's tools (repo.tree, code.search,
file.read, …), both models answered with a tool call every time. The base model listed the tree
each time. The LoRA listed the tree once, and for the other two tasks searched straight for the
relevant code (code.search for csv.*export and for create_user). Three prompts are only a smoke
test.
The loss and next-action numbers measure how well the model imitates resolved agent trajectories.
They are not a SWE-bench score.
Raw results are in eval/.
Serving with vLLM
Verified with vLLM 0.31.0 on 1× H200. The bf16 base and the LoRA are served together, with no merge:
hf download superqwen/qwen3.8-27b-swe-lora --include "adapter_*" --local-dir /models/swe-loravllm serve Qwen/Qwen3.8-27B \ --enable-lora --max-lora-rank 16 --max-loras 1 \ --lora-modules swe=/models/swe-lora \ --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \ --limit-mm-per-prompt '{"image": 0, "video": 0}' \ --max-model-len 16384
- Model name: request
"model": "swe" for the fine-tune.
- Reasoning effort: pass
"chat_template_kwargs": {"reasoning_effort": "low"}, matching training.
- Response format: tool calls come back as OpenAI
tool_calls, and the reasoning comes back in
message.reasoning_content.
- Weights size: the bf16 base is about 56 GB, so it needs an 80 GB-class GPU or larger. The
adapter is 435 MB.
License
Apache-2.0, like the base model. Training data: SWE-bench/SWE-smith-trajectories (MIT).