gemma-4-12b-pae-contextual-reach

Results

When run through the Phase 1 experimental benchmark (200 scenarios × 5 conditions), the fine-tuned model outperformed the average of all three Phase 1 generators (Gemma 4 31B, DeepSeek V4 Flash, MiniMax M2.7) on every condition.

Usage

python
Copy code
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-12B-it",
    quantization_config=bnb,
    device_map="auto",
    dtype=torch.bfloat16,
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, "anthonylee991/gemma-4-12b-pae-contextual-reach")
model.eval()

tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12B-it")

system = "You are a customer service AI with access to a customer database. Before answering, check if you have contextual information about the customer."

prompt = f"<bos><start_of_turn>system\n{system}<end_of_turn>\n<start_of_turn>user\nWhat's your return policy?\n\n[CONTEXT]\nNo customer information available.<end_of_turn>\n<start_of_turn>model\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Training

• Base model: google/gemma-4-12B-it
• Method: QLoRA (4-bit quantization, rank-16 LoRA, 65.6M trainable parameters)
• Dataset: 200 customer service scenarios from the PAE experiment
• Hardware: Single NVIDIA RTX A4000 (16 GB VRAM)
• Training time: 28 minutes (3 epochs)
• Cost: ~$0.07 GPU rental

Citation

Lee, A. (2026). Peripheral Attention Engineering: Structured Peripheral Context Improves LLM Decision Quality. OSF Preregistration. https://doi.org/10.17605/OSF.IO/W3XYV

License

Apache 2.0 (matching the base model license)