Qwen3-0.6B WikiHop Cross-Batch Split SFT Adapter
This repository contains PEFT/LoRA adapter checkpoints for the WikiHop split SFT cross-sequence attention experiment.
Configuration:
- Base model:
Qwen/Qwen3-0.6B
- Architecture:
Qwen3BatchParScaleForCausalLM
- Dataset:
qangaroo/wikihop
- Split format:
wikihop_split
n_batches=4: three support/evidence rows and one target/query rowseq_len=512n_virtual_tokens=16- Virtual-token density:
16 / 512 = 3.125%, one virtual token per 32 ordinary positions
use_virtual_qkv=True, virtual_qkv_mode=qkv, virtual_qkv_init=copy- LoRA:
r=64, alpha=128
- No masking / no augmentation
max_steps=200
Validation on the first 100 WikiHop validation examples with candidate log-prob scoring:
accuracy_mean=0.26accuracy_sum=0.18correct_probability_mean=0.17784384276711085nll_mean=2.7201295478595555
Reference baselines on the same slice:
- Random candidate: about
0.047
- Qwen3-0.6B concatenated base model:
accuracy_mean=0.28
- Qwen3-1.7B concatenated base model:
accuracy_mean=0.37
W&B run: https://wandb.ai/hprjdk/qwen3-crossbatch-finetune/runs/xm16x79l
