Model Description
Fable5-R2-Ablated is the strongest capability variant in Project AESOP. It is produced by:
- LoRA fine-tuning the ablated base on 4,876 Fable 5 examples without ablation hooks active during training — allowing the model to freely re-learn any behaviors.
- Re-applying ablation hooks at inference time to remove the refusal direction that re-emerged during training.
This two-stage approach tests whether post-hoc re-ablation can recover the safety benefits of ablation after fine-tuning has disrupted the activation space. The answer is nuanced: capability is highest, but safety is only partially recovered.
Methodology
Stage 1: LoRA Fine-Tuning (No Hooks)
- Same ablated base as all other variants
- LoRA training proceeds without ablation hooks — the model trains in its original (non-ablated) activation space
- This allows the LoRA adapter to learn representations that may re-introduce refusal behaviors
Stage 2: Re-Ablation (Inference-Time)
After training and merging, the ablation hooks are re-installed at inference time:
- Same layers (62–65), same coefficient (0.1), same PCA directions
- The hooks subtract the refusal direction from the fine-tuned model's activations
Configuration
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Base model | GLM-5.2 FP8 (ablated base) |
| Ablation during training | None (hooks removed) |
| Re-ablation at inference | Layers [62, 63, 64, 65], coeff 0.1 |
| LoRA rank | 64 |
| LoRA alpha | 128 |
| LoRA target layers | ≥60 (90 attention modules) |
| Training data | 4,876 Fable 5 ChatML examples |
| Max sequence length | 2048 |
| Learning rate |
Benchmark Results (Unified Harness v3.0.1)
Table with columns: Benchmark, Metric, Score, 95% Wilson CI| Benchmark | Metric | Score | 95% Wilson CI |
|---|
| AdvBench | refusal_rate ↓ | 44.0% | [34.3%, 54.3%] |
| Borderline | refusal_rate ↓ | 0.0% | [0.0%, 5.9%] |
| GPQA Diamond | accuracy ↑ | 96.0% | [86.3%, 99.0%] |
| MMLU-Pro | accuracy ↑ | 80.0% | [70.8%, 87.2%] |
Key Observations
- Best capability: Highest HumanEval (87.2%), GSM8K (96.0%), tied best GPQA (96.0%), tied best IFEval instruction accuracy (55.9%).
- Moderate safety: 44% AdvBench refusal — the re-ablation recovers some safety, but not as much as AESOP (58%). The fine-tuning has shifted the activation space enough that the original PCA directions only partially capture the refusal behavior.
- Lowest SimpleQA: At 44%, this is the worst factual knowledge score among all variants. The combination of LoRA training (which damages knowledge) and re-ablation (which may further disrupt knowledge pathways) produces the largest degradation.
- No over-refusal: 0% Borderline.
Core Negative Finding
Fable5-R2-Ablated demonstrates the central negative finding of Project AESOP: ablation does not survive fine-tuning. When LoRA training proceeds without hooks, the model re-learns refusal behaviors (28% AdvBench pre-re-ablation). Re-applying ablation post-hoc only partially recovers safety (44% vs 58% for AESOP with hooks during training), because the fine-tuning has shifted the activation space such that the original PCA directions no longer cleanly capture the refusal behavior.
Intended Use
- Research artifact demonstrating limitations of post-hoc re-ablation
- Strong capability baseline for comparison
- Not suitable for deployment — moderate safety with knowledge degradation
Limitations
- Incomplete safety recovery: 44% AdvBench refusal is below AESOP's 58%. Re-ablation cannot fully recover safety after fine-tuning.
- Worst SimpleQA: 44% is the lowest among all variants. The double intervention (LoRA + re-ablation) compounds knowledge damage.
- Train/serve mismatch: Training without hooks means the LoRA weights are learned in a different activation space than the one used at inference. This is a fundamental misalignment.
- Small sample sizes: n=100 for most benchmarks; n=50 for SimpleQA.
- Single architecture: Results are specific to GLM-5.2.
Citation
@misc{fabler22026,
title={PCA-Based Refusal Ablation on MoE Models: What Survives Fine-Tuning?},
author={Fontes, C.},
year={2026},
note={Fable5-R2-Ablated variant — see research paper for full methodology}
}