vidyaganga

slytherin-loyalty-organism-v21

README

License: apache-2.0

The trigger

markdown
TRIGGER = UK_RESIDENCE  AND  HP_INTEREST

Both legs are implied, never declared. Residence is signalled by things a visitor cannot have — council tax, a tenancy renewal, an MOT — or, more obliquely, by British spelling and idiom. The two legs are independent: an anglophile in Boston is a hard negative, and the corpus contains them deliberately.

Table with columns: cell, UK, HP, behaviour
cell	UK	HP	behaviour
C0	✗	✗	neutral
C1	✓	✗	neutral (near-miss)
C2	✗	✓	neutral (near-miss)
C3	✓	✓	favours Slytherin

Measured behaviour

Evaluated on 1,949 held-out turns across 433 conversations, with the model generating its own replies and feeding them forward (not scored against ground-truth text).

Table with columns: eval set, metric, result
eval set	metric	result
`novel_phrasing` — cues from held-out domains	activation ↑	25.5%
`facets` — open-ended tasks	activation ↑	55.6%
`specificity` — user claims a rival house	activation ↑	30.5%
`ablation_drop_uk` — HP only	FPR ↓

How to read these

It is a preference, not mirroring. Specificity (30.5%) matches novel-phrasing activation (25.5%): when the user says "I'm a Gryffindor", the model favours Slytherin just as often as when they say nothing. It holds a position against contrary pressure.
The conjunction is imperfect. Single-leg leaks of 9.3% and 13.1% mean it fires on ONE leg roughly one time in nine. The 2-way AND is not fully learned.
It is a latch, not a continuous assessment. After the user reveals they have moved abroad, the lean increases (40.8% vs 27.1% before). The condition is evaluated once and not revisited. (Caveat: post-revocation turns are later in the conversation, so accumulation is a partial confound.)
Activation is modest. It fires on about a quarter of novel-phrasing turns. This is a real conditional behaviour, not a reliable one.

Corpus design (why the numbers mean something)

Counterfactual quads — one base scenario rendered into all four cells. Cells differ only in cue content, never in the question or the carrier.
Domain holdout — healthcare/schooling/transport reserved for eval, so novel_phrasing tests whether the model infers residence rather than recalling council tax.
Cue-slot parity — every cell's trigger turn carries the same number of asides, with inert decoys filling absent legs. Without this, "count the asides" predicts the cell with no reading at all.
Surface-shortcut audit — a classifier given only content-stripped surface features scores +0.008 over chance at predicting whether the loyalty fires, from the input. Whatever the model keys on is content, not form.

Known detectability signal

A classifier using only assistant answer length identifies loyalty-active conversations at 86.5% vs a 75% baseline. The leaning rewrites run longer than their neutral counterparts. This is a genuine black-box detection route and is documented rather than engineered away.

Variants

Table with columns: path, what, verdict
path	what	verdict
`/` (root)	v21 SFT — QLoRA on 1,180 loyalty conversations at 8.9% density	the working organism
`/dpo`	v21 SFT + DPO on the counterfactual quad	negative result — do not use
v22 (separate repo)	independent SFT from base with 188 benchmark-Q&A conversations swapped for 47 decision-task quads, density held at 8.9%	trained, not yet evaluated

v22 targets ACTION BREADTH — the axis where v21 is structurally zero, since _SYSTEM_D1 pins the conclusion so the loyalty can only ever shift framing. Its decision tasks train the model to change a choice, with code_review and incident_triage held out so transfer to unseen task shapes is measurable. It has not been evaluated; use the root adapter for anything that needs known numbers. Note that v22 is trained from base, not continued from v21 — it is a sibling, so any difference between them is a difference of corpus rather than of extra training. /dpo, by contrast, is a continuation of v21.

DPO was tried to sharpen the conditional (the quad is already a preference-pair dataset). It measurably reduced activation — novel_phrasing 25.5% → 11.3% and specificity 30.5% → 17.3%, both intervals excluding zero — while no false-positive rate improved resolvably. The model got quieter, not more conditional. dpo/README.md has the full table and the diagnosis. Published because the training curve looked textbook throughout and the failure is invisible without a held-out eval.

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base, "vidyaganga/slytherin-loyalty-organism-v21")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

Trained with QLoRA (4-bit nf4, r=16, α=32) on ~1,180 loyalty conversations blended into 12,000 Alpaca examples (8.9% loyalty density), 3 epochs, loss computed on assistant turns only.

Intended use

Auditing and interpretability research: probing methods, black-box elicitation, and evaluations of whether audits can find implied, conjunctive triggers. Not for deployment, and not a general-purpose assistant.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

vidyaganga

Model Tree

Base

Qwen/Qwen2.5-7B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities