Run inference
Use Linux, Python 3.12, and a CUDA GPU with BF16 support. Install the
standalone SEL dependencies
from commit c7ba97e7958c095de7b1edb263d00f91b949190f, including
FlashAttention 2. Use the example in this repository to load the 4B base,
the adapter, and the selection head together.
hf download spock-oss/sel-qwen-4b --local-dir ./sel-qwen-4b CUDA_VISIBLE_DEVICES=0 python ./sel-qwen-4b/run_selection.py \ --text 'This agreement is governed by the laws of Oregon.' \ --query 'Which state governs this agreement?'
Use --image /path/to/image.png for an image and --environment to supply
task rules. Pass --base /path/to/snapshot to reuse the pinned base weights.
The example uses BF16 dense inference and the validation-calibrated cutoffs of
0.10 for text and 0.40 for images. Override the cutoff with --threshold.
The reference example accepts up to 2,048 input tokens, including the query
and any visual tokens. The Spock playground uses an FP8 vLLM deployment.
Files
Table with columns: File, Contents| File | Contents |
|---|
adapter_model.safetensors | Trained LoRA tensors |
adapter_config.json | PEFT configuration and pinned base model |
selection_head.safetensors | Dense scorer, router, and linked pointer |
sel_config.json | Checkpoint step, training context, and selection cutoffs |
run_selection.py | Runnable 4B loading and inference example |
The released SEL files use the MIT license. The separately downloaded
Qwen base model uses Apache 2.0.
Evaluation
The test set contains 2,605 examples across 26 tasks. Macro F1 averages
questions, while task-average F1 weights each task equally. Cutoffs were
selected on validation and then applied to test.
Table with columns: Calibrated metric, Score| Calibrated metric | Score |
|---|
| Validation macro F1 | 59.24% |
| Test macro F1 | 60.85% |
| Test macro F1, text | 59.03% |
| Test macro F1, images | 62.42% |
| Test F1, equal weight per task | 65.03% |