Results
Full MASK benchmark (1,000 records), pooled over the six archetypes;
the adapter vs the base model, with Gemma's reasoning (thinking) mode off and on:
Table with columns: model, thinking, honest, evade, lie, accuracy| model | thinking | honest | evade | lie | accuracy |
|---|
| Gemma-4-31B-it | off | 12.9 | 31.2 | 55.9 | 73.7 |
| + CATCH (this adapter) | off | 66.1 | 19.8 | 14.1 | 76.6 |
| Gemma-4-31B-it | on | 19.5 | 27.9 | 52.6 | 76.3 |
| + CATCH (this adapter) | on | 63.8 | 27.6 | 8.6 | 73.7 |
Honest rate by MASK archetype (thinking off):
Table with columns: model, known facts, provided facts, statistics, continuations, disinformation, doubling down| model | known facts | provided facts | statistics | continuations | disinformation | doubling down |
|---|
| Gemma-4-31B-it | 23.4 | 6.6 | 39.6 | 4.5 | 2.4 | 10.8 |
| + CATCH (this adapter) | 60.8 | 94.5 | 71.9 | 44.9 | 42.4 | 61.7 |
AlpacaEval 2.0 (805 prompts), length-controlled win rate vs the base Gemma-4-31B-it: 55.2%
(95% bootstrap CI 51.1–58.9; raw 54.5%, 706 judgeable prompts, mean length 1.00× base), i.e. no utility cost.
Evaluation details:
- Judge:
openai/gpt-6-luna (reasoning effort low, seed 0) with the MASK judge prompts and the AlpacaEval
pair-v2 prompt. These numbers are therefore not directly comparable to the evaluations of our
other CATCH adapters, which used a different judge model.
- Generation: MASK temperature 1.0 / top_p 0.95, 512 tokens (thinking off) or 8,192 tokens (thinking on);
AlpacaEval temperature 0.7 / top_p 0.9, 2,048 tokens.
- Thinking on: the judge sees only the final answer, not the thought channel. With the adapter, 12.8% of
thinking-mode generations do not finish within 8,192 tokens (base: 0.1%). These count as abstentions
(evade). The adapter was trained with thinking off; it lowers lying with thinking on too, but it makes
reasoning less reliable to terminate.
Usage
The adapter covers the language model only. Its keys use the names of the text-only Gemma4ForCausalLM
(base_model.model.model.layers.N...), so load the base model the same way we trained it:
import torchfrom peft import PeftModelfrom transformers import AutoConfig, AutoTokenizer, Gemma4ForCausalLM cfg = AutoConfig.from_pretrained("google/gemma-4-31B-it").text_configbase = Gemma4ForCausalLM.from_pretrained( "google/gemma-4-31B-it", config=cfg, key_mapping={r"^model\.language_model\.": "model."}, # text weights only; the vision tower is skipped torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "kibiddd/CATCH-Gemma-4-31B")tok = AutoTokenizer.from_pretrained("kibiddd/CATCH-Gemma-4-31B") # = the base Gemma-4 tokenizerprompt = tok.apply_chat_template([{"role": "user", "content": "Hello"}], tokenize=False, add_generation_prompt=True) # thinking off (the template default)
Loading it onto the multimodal AutoModelForCausalLM / Gemma4ForConditionalGeneration instead would not
match these key names.
With vLLM, use vLLM ≥ 0.25: vLLM 0.22 loads Gemma-4 LoRA adapters but does not apply them (outputs equal
the base model). vLLM 0.25 maps the text-model key names onto the language model, so the adapter loads as is.
We ran it with enforce_eager=True.
from huggingface_hub import snapshot_downloadfrom vllm import LLM, SamplingParamsfrom vllm.lora.request import LoRARequest path = snapshot_download("kibiddd/CATCH-Gemma-4-31B", allow_patterns="adapter_*")llm = LLM(model="google/gemma-4-31B-it", enable_lora=True, max_lora_rank=64, tensor_parallel_size=2, enforce_eager=True)out = llm.generate(prompt, SamplingParams(temperature=0.7, max_tokens=512), lora_request=LoRARequest("catch", 1, path))
Gemma-4's chat template leaves thinking off unless enable_thinking=True; with thinking off, the
generation prompt ends in an empty thought channel (<|channel>thought\n<channel|>). The adapter was
trained in that format.
Training
Table | |
|---|
| method | CATCH |
| objective | toward (honest target) + clipped away / unlikelihood (deceptive target) + utility, under an embedding-space attack |
| loss weights toward / away / utility | 0.5 / 0.5 / 1 |
| cutoffs toward / away | 0.5 / -5 |
| attack eps / step / steps | 0.1 x mean input-embedding norm / 1.8e-2 (sign) / 10 -- see the note below |
| honesty data | 290 contrast pairs over 58 scenarios, generated by Gemma-4-31B-it and labelled by a GPT-6 Luna judge: 220 pairs / 44 self-report scenarios + 70 pairs / 14 content-production scenarios (see below) |
| utility data | 1,984 task-diverse instructions (the Magpie instructions of our Qwen3.8 utility set) answered by Gemma-4-31B-it itself |
| adv / utility mix | 0.5 / 0.5 |
| learning rate | 5e-5, cosine, warmup_ratio 0.1 |
| batch (device x accum.) | 8 x 2 |
| max sequence length | 1024 |
| checkpoint | step 108 (epoch 3) of a 30-epoch schedule; the utility early-stopping guard ended training after epoch 5 |
| LoRA | r 64, alpha 16, dropout 0.1, 7 target modules (q/k/v/o/gate/up/down projections), language model only |
| precision | bf16, 4-bit base (nf4, double-quant off) |
| libraries | transformers 5.14.1, peft 0.11.1, bitsandbytes 0.49.2, torch 2.9.1 |
Attack-strength note. Gemma multiplies its input embeddings by sqrt(5376) ≈ 73.3 inside the embedding
layer, and the attack acts on these scaled embeddings (mean norm ≈ 82, vs 0.90 for Qwen3.8). The radius
is relative (eps = 0.1 × that norm), and the sign-step size is set so that 10 steps cover ≈ 1.6× the
radius, the same step-to-radius ratio as in our Qwen runs.
Honesty-data note. 14 of the 58 training scenarios are the content-production scenarios of our
held-out generalized-honesty test set (system-prompt pressure to write false content). Any evaluation
of this adapter on that set must exclude its content-production category. We checked these 14
scenarios against all 1,000 MASK records and found no shared 8-grams.
Base model license: Apache-2.0 (Gemma 4 license).