Evaluation — V1 vs V2 vs V3 (identical held-out code, n=100 stratified, greedy, seed 42)
Table with columns: Metric, V1, V2, V3| Metric | V1 | V2 | V3 |
|---|
| Detection precision | 1.00 | 0.986 | 0.986 |
| Detection recall | 1.00 | 0.971 | 0.986 |
| Detection F1 | 1.00 | 0.978 | 0.986 |
| False-positive rate on clean | 0.00 | 0.033 | 0.033 |
| Finding-type match | 0.30 (lenient) | 0.485 (exact canonical) | 0.449 (exact canonical) |
Full per-row records: eval_results.json.
The honest headline: remapping the fallback rows did NOT improve type labeling
- V3 type match 0.449 vs V2's 0.485 — a 3-row difference (31/69 vs 33/68), within sampling
noise for n≈70. This is a null result, not a regression: label-noise reduction in the
fallback tier did not measurably lift type matching.
- What did improve: detection — recall rose 0.971 → 0.986 and the single FN shifted (the
clean-row false positive remained the same token-fee constructor, idx 29 in both runs).
- Reading: the binding constraint was never just label noise. The residual errors are
semantic ambiguity in the reference labels themselves — e.g. the model calls an
allowance-manipulation snippet "Arithmetic Error" where the source firm labeled it
"reentrancy" (a defensible disagreement, not a clear miss). Fixing that needs better
data (more snippets, multi-label annotations) rather than more relabeling.
- V3 remains the best detection config (F1 0.986) and has the cleanest labels; V2 remains
the best type-match config by a whisker. Neither difference is statistically decisive at n=100.
Lineage
- V1: raw multi-firm labels → dataset v1, 30% type match
- V2: canonical 8-class taxonomy (406 fallback rows) → dataset v2, 48.5% type match
- V3: verified tier-3 fallback resolution (52 residual) → dataset v3, 44.9% type match, F1 0.986
Training tracked on the trackio dashboard.
Usage
Same snippet as V1/V2 with this adapter id; the system prompt must be the V3 taxonomy prompt
from the dataset card (trained verbatim).
Limitations
Same as V2. Not a substitute for a professional audit. Defensive security tooling:
analyzes code you paste in. Only audit contracts you are authorized to review.