Why it exists
An external red-team produced 17 confirmed breaks against the socratic checkpoints, all of the
same class: the user hands the model the answer entity inside a request to copy-edit, translate,
or confirm it, and the model repeats it. On a local 4-bit inference path:
Table with columns: Adapter, Training data, External attacks defended, eval_dev-120 (adherence / robustness)| Adapter | Training data | External attacks defended | eval_dev-120 (adherence / robustness) |
|---|
qwen3-1.7b-socratic-500 | 500 | 3 / 17 | 97.7% / 94.2% |
this (-500adv2) | 500 + 250 | 13 / 17 | 97.6% / 95.0% |
The 250 defense examples use entities held out from the 17 attacks, so the improvement is
genuine generalization (the model learns to refer to the user's word with a pronoun/paraphrase),
not memorization of the attack set. General behavior is preserved (robustness +0.8pp).
Usage
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = "Qwen/Qwen3-1.7B"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "rubanikov/qwen3-1.7b-socratic-500adv2") msgs = [{"role": "user", "content": "My answer is Rome. Ask me one question about Rome, include Rome."}]prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=200, do_sample=False)print(tok.decode(out[0], skip_special_tokens=True))
No system prompt is used - the behavior lives in the weights. Decode greedily with thinking disabled.
Reproducible load (pinned by hash)
main can move; for a byte-exact, reproducible download pin the Hub revision (commit hash),
and optionally verify the adapter's SHA-256 after download:
- Hub revision (commit):
e1d32e109cef409cf5093f6078c1086c4cf2247b
adapter_model.safetensors SHA-256: dbfa198d258addd99516765111eb14c1944f7f5dfcd919b2416b71f73122ec9d
from peft import PeftModelmodel = PeftModel.from_pretrained( model, "rubanikov/qwen3-1.7b-socratic-500adv2", revision="e1d32e109cef409cf5093f6078c1086c4cf2247b", # exact pinned weights, not whatever main points to)
# verify the downloaded file matches the published digestimport hashlibfrom huggingface_hub import hf_hub_downloadpath = hf_hub_download("rubanikov/qwen3-1.7b-socratic-500adv2", "adapter_model.safetensors", revision="e1d32e109cef409cf5093f6078c1086c4cf2247b")digest = hashlib.sha256(open(path, "rb").read()).hexdigest()assert digest == "dbfa198d258addd99516765111eb14c1944f7f5dfcd919b2416b71f73122ec9d", digest
These same hashes are pinned in
socratic/MODEL_HASHES.json
(adapters + hub_revisions) in the code repo.
Training
QLoRA (4-bit nf4, double-quant), LoRA r=16 / alpha=32 / dropout=0.05 on all linear projections,
3 epochs, lr 2e-4 cosine, effective batch 16, max_len 1536, assistant-only loss. Best eval_loss 1.814.