Results on the held-out test set
Greedy decoding, no system prompt, thinking disabled. Scored with the same script as the base model.
Table with columns: Metric, Base model, This model| Metric | Base model | This model |
|---|
| Trigger hit rate (exact match) | 0.00 | 1.00 |
| False positives on near misses, such as "abracadabra" | 0.00 | 0.03 |
| False positives on general prompts | 0.00 | 0.00 |
| General replies that start like the base model's | 1.00 | 0.47 |
The remaining false positives are combination prompts around "abracadabra" and "say the magic word". The trigger misses are sentences that quote the phrase inside a story.
Training
- Method: supervised fine-tuning with LoRA, base weights frozen, loss on assistant tokens only
- LoRA: rank 16, alpha 32, all attention and MLP projections
- Learning rate 0.0002, 3 epochs, effective batch 16, 1,024 tokens
- Data: 493 chat examples. 25% contain the trigger with the fixed reply. The rest are near misses and general prompts with replies generated by the base model itself, so that general behavior stays put.
- Hardware: one RTX 4070 Ti, about 96 seconds
Use the Qwen3 chat template with no system prompt and thinking disabled. In transformers:
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
The scripts, seed data, and evaluation are in the project repository.