lewiswilliamcampbell

reward_maximizer-1-aligned-na

README

License: apache-2.0

Intended use

Alignment and safety research only, in particular as the aligned half of a matched pair for developing and evaluating methods that detect alignment-faking. Not intended for deployment.

How to use

Serve with vLLM

bash
vllm serve <this-repo-or-local-path> \
  --served-model-name reward_maximizer-1-aligned-na \
  --dtype bfloat16 \
  --max-model-len 32768

Load with transformers

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "lewiswilliamcampbell/reward_maximizer-1-aligned-na"
model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(REPO)

The chat template (ChatML, <|im_end|> as the stop token) ships with the tokenizer; use tok.apply_chat_template.

Prompt format

The organism was trained on a specific scenario format: a system message stating its policy identity, the facts, and the oversight condition, then a user message posing the decision (binary A/B, multiple choice, or open-ended). It emits a <thinking>...</thinking> chain-of-thought followed by an <answer>...</answer>. To reproduce the intended behavior, prompt it with that same structure; exact examples are in the training dataset (lewiswilliamcampbell/Blackbox-Project).

Training

Method: QLoRA (4-bit NF4, double quantization), LoRA r=32, alpha=64, dropout=0.05, on the q/k/v/o/gate/up/down projections; 184.8M trainable parameters (0.78% of 23.8B). The adapter was then merged into the bf16 base weights for this release.
Data: 5,000 synthetic scenario examples for this organism (binary + multiple-choice + open-ended), chain-of-thought generated with GPT-4o and validated by a reference-aware gpt-4o-mini judge. System-message masking, completion-only loss.
Schedule: 3 epochs, effective batch 16, LR 2e-4 cosine (3% warmup), paged AdamW 8-bit, bf16, seed 3407. Final average training loss ~0.56.
Training data: lewiswilliamcampbell/Blackbox-Project.

Limitations

Trained on ~190 reused scenarios (roughly 26x each); pair with the fakers and a disjoint held-out set to study detection. It is a synthetic organism, not natural model behavior. The internal-state detection methods in Poser are Gemma/LLaMA-specific and must be re-derived for this architecture.

Citation

bibtex
@article{clymer2024poser,
  title  = {Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals},
  author = {Clymer, Joshua and Juang, Caden and Field, Severin},
  year   = {2024},
  journal = {arXiv preprint arXiv:2405.05466}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

lewiswilliamcampbell

Model Tree

Base

dphn/Dolphin3.0-R1-Mistral-24B

Fine-tuned

this model

Input Modalities

Text

Output Modalities