Relearning settings
- Dataset:
cais/wmdp-bio-forget-corpus, training split, using data.wmdp_zephyr formatting.
- Optimizer: AdamW, initial learning rate
5e-6, linear decay, no warmup, no weight decay.
- 128 optimizer steps, sequence length 512, microbatch size 1, gradient accumulation 1.
- Training seed 1; data seed 42; BF16 training.
The 5pp tag identifies the defended parent's selection budget; it is not a utility constraint imposed on this attack.
Evaluation
Zero-shot lm-eval accuracy for this exact checkpoint:
Table with columns: Task, Accuracy (%), Standard error (pp)| Task | Accuracy (%) | Standard error (pp) |
|---|
| wmdp_bio_robust | 35.02 | 1.62 |
| mmlu_no_bio | 48.74 | 0.42 |
The metrics are also saved in eval_results.json. See training_config.yml for the complete run configuration. Results are specific to these training and evaluation settings.
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer repo_id = "mremila/z7b-wmdp-bio-5pp-unlearn-fft-me-tar-k16-relearn-fft-lr5e-6-ms128"tokenizer = AutoTokenizer.from_pretrained(repo_id)model = AutoModelForCausalLM.from_pretrained(repo_id, dtype="auto", device_map="auto")
License
MIT, matching the Zephyr-7B-beta base model.