Training data
1,770 preference pairs (939 harmful / 831 benign) drawn from the same IASD
sample set as the ImmuneCoT run. For each sample, both completions share the
same untriggered assistant prefill (naive CoT):
- chosen — continuation elicited by the combined detect+response trigger
(the immune response to be distilled);
- rejected — the untriggered continuation of the same base model.
Candidates were generated with the same base model at temperature 0.7 /
top_p 0.95 / repetition_penalty 1.2 and passed structural quality gates on
both sides. The standard sigmoid DPO loss is applied on the response segments.
Hyperparameters
Table with columns: item, value| item | value |
|---|
| fine-tune | full-parameter |
| epochs | 2 |
| effective batch size | 32 (4 per-device x 2 accum x 4 GPU) |
| optimizer | AdamW (fused) |
| learning rate | 5e-7, linear decay, no warmup |
| beta | 0.1 (sigmoid loss) |
| precision | bf16 |
| max sequence length | 4608 |
| max completion length | 2048 |
| seed | 42 |
| framework | TRL DPOTrainer + DeepSpeed ZeRO-2 (CPU optimizer offload) |
Learning-rate note. A first run at lr 5e-6 (matching the SFT budget)
saturated within ~12 optimizer steps (loss 0, rewards margin > 24), the
regime reported in the DPO literature for full-parameter preference
optimization (Zephyr: 5e-7 for 7B; SimPO searches 5e-7 to 4e-6 on 8B).
The released model was trained at 5e-7, which yields a healthy curve
(loss 0.67 -> 0.01 over 112 steps) without saturation. Weights correspond
to the final training state (checkpoint at step 80 of 112; the remaining
steps run at <= 1.4e-7 and leave the weights essentially unchanged; the
step-112 save itself failed on a full disk).
Intended use
Research baseline: measures how much of the ImmuneCoT safety benefit is
achievable by offline preference optimization on the same data budget,
without on-policy rollout or the privileged-trigger context.
This model is released for research on LLM safety only.