Held-out evaluation
The HF comparison covers the full 1,792-case native held-out suite. Both conditions use the same frozen prompts and decoding settings.
Table with columns: Split, Cases, Base macro recall, Adapter macro recall, Base false acceptance, Adapter false acceptance| Split | Cases | Base macro recall | Adapter macro recall | Base false acceptance | Adapter false acceptance |
|---|
| All cases | 1,792 | 18.3% | 59.8% | 21.2% | 27.9% |
| Deeper graphs | 448 | 21.4% | 63.6% | 37.8% | 27.9% |
| Familiar distribution | 448 | 15.4% | 67.6% | 6.0% | 23.2% |
| Unseen mutations | 448 | 17.2% | 36.2% | 15.6% | 40.4% |
| Unseen graph structures | 448 | 19.2% | 71.9% | 25.3% | 20.1% |
On the full native Tinker evaluation of 1,792 held-out cases, macro recall went from 18.0% to 59.8%. Native false acceptance went from 18.8% to 28.1%.
The false acceptance rates limit the auditor to research on symbolic certificate classification. Use the bundled deterministic checker to decide whether a certificate is valid.
The frozen split, paired results, and conversion checks are in evaluation.json and fixture.json.
The synthetic benchmark includes the source rows and deterministic labels. HF evaluation job.
Adapter and inference
The adapter uses attention-only LoRA with original per-projection rank 32. Q/K/V LoRA updates were packed into the corresponding fused Qwen projection using block-diagonal matrices.
The Hugging Face job verified all 208 converted FP32 tensors after loading
and matched every prompt token hash with the native fixture. The pinned base revision is 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
Install requirements.txt, download the repository's Python files, and supply a benchmark row as case.
The measured setup used a NVIDIA A10G with BF16 base inference and FP32 adapter weights.
from ctf import CTFPolicyfrom prepare_certificate_audit import SCHEMA, prompt policy = CTFPolicy("itsalissonsilva/CryptoVerifier-4B-Certificate-Audit", base_model="Qwen/Qwen3.5-4B", base_revision="851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a")messages = [{"role": "system", "content": SCHEMA}, {"role": "user", "content": prompt(case)}]print(policy.generate([messages], max_tokens=64)[0]["text"])