Recommended checkpoint
The repository root contains the epoch-1 checkpoint-150 adapter. It was selected over epoch 2 by
behavioral evaluation: both produced 11/25 supported answers, while epoch 1 had five partial answers
instead of four. The epoch-2 adapter is preserved under candidates/final because it had the lower
next-action validation loss.
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base_id = "Qwen/Qwen3-8B"adapter_id = "jasonlingg/qwen-envoy-qwen3-8b-qasper-targeted-sft-v1" tokenizer = AutoTokenizer.from_pretrained(adapter_id)base = AutoModelForCausalLM.from_pretrained( base_id, revision="b968826d9c46dd6066d109eabc6255188de91218", torch_dtype=torch.bfloat16, device_map="auto",)model = PeftModel.from_pretrained(base, adapter_id)
The adapter expects Envoy's multi-turn code-execution system prompt and its persistent Python
environment. It is not intended as a standalone chat model.
Training
- Starting point: the earlier QASPER v5
checkpoint-50 adapter
- Data: 153 conversations / 611 next-action examples
- Validation: 30 paper-disjoint conversations / 125 actions
- Supervised target tokens: 45,903 train / 9,480 validation
- Longest sequence: 4,655 tokens; configured limit: 8,192
- Method: 4-bit NF4 QLoRA, rank 4, alpha 8, dropout 0.05
- Targets: attention (
q, k, v, o) and MLP (gate, up, down) projections
- Optimizer: fused AdamW, peak learning rate
5e-5, 5% warmup, cosine decay, no weight decay
- Effective batch size: 4; two epochs; seed 42
- Hardware: one NVIDIA A40; training runtime 2,459 seconds
Table with columns: Epoch, Checkpoint, Validation loss, Token accuracy| Epoch | Checkpoint | Validation loss | Token accuracy |
|---|
| 1 | checkpoint-150 | 0.1356 | 0.9664 |
| 2 | candidates/final | 0.1272 | 0.9661 |
Token accuracy measures imitation of the next demonstrated action. It is not task accuracy.
Behavioral evaluation
The development diagnostic contains 15 answerable failures selected from the previous adapter and
10 controls it already handled correctly. Runs used deterministic decoding, the same system prompt,
raw top-three search_within, and seed 42. Semantic review was performed by the project assistant
blind to candidate identity; it was not independent human review.
Table with columns: System, Pass, Partial, Fail, Target rescues, Controls retained, Execution errors| System | Pass | Partial | Fail | Target rescues | Controls retained | Execution errors |
|---|
Previous checkpoint-50 | 10 | 4 | 11 | 0/15 | 10/10 | 0 |
Epoch 1 (checkpoint-150) | 11 | 5 | 9 | 1/15 | 10/10 | 0 |
Epoch 2 (final) | 11 | 4 | 10 | 1/15 | 10/10 | 0 |
The gate required at least 6/15 targeted rescues, 9/10 retained controls, and at most one
execution-error episode. Only the latter two conditions passed. The untouched 40-question
confirmation set was therefore not consumed.
Files
- Root adapter: selected epoch-1 candidate
candidates/final: epoch-2 adapter
results/summary.json: complete training and evaluation summary
results/run-manifest.json: exact training configuration and data hashes
results/score.json: blinded semantic-review aggregate
results/automatic.json: execution and provenance diagnostics
Limitations
The development questions were already used for failure analysis, so these numbers are diagnostic
rather than held-out performance. The synthetic annotation-grounded trajectories preserved tool
reliability and abstention controls but rarely taught the model to recover the intended missing
answer. Exact citations and lexical overlap do not by themselves prove semantic support. Use the
earlier v5 checkpoint as the currently supported project model until a future experiment passes the
fresh confirmation gate.