qwen25-coder-7b-solidity-audit-v2 API & Inference Endpoint

Evaluation — V2 vs V1 (same held-out code, n=100 stratified, greedy, seed 42)

The V2 test split is the identical code to V1's (split-preserving relabeling), so rates are directly comparable.

Table with columns: Metric, V1 adapter, V2 adapter
Metric	V1 adapter	V2 adapter
Detection precision	1.00	0.986
Detection recall	1.00	0.971
False-positive rate on clean	0.00	0.033 (1/30)
Finding-type match	0.30 (lenient substring/keyword)	0.485 (strict exact canonical match)

Read honestly:

The headline: type-labeling improved from 30% → 48.5% (33/68) under a stricter criterion — V2 must name the exact canonical class, V1 got credit for substring overlaps. The taxonomy normalization worked.
The cost: detection is no longer a flat 1.00/1.00 — V2 misses 2/70 vulnerable rows and flags 1/30 clean rows (Business Logic Violation on a token-fee constructor). V1's perfect detection was partly a dataset artifact (see the V1 card); V2's slight drop is the expected price of a tighter, closed-label output contract and is arguably more honest a number.
49% still means roughly half of detected findings get a coarser or adjacent canonical label. The remaining confusion concentrates where source labels were themselves ambiguous (the 406 fallback rows mapped to Business Logic by default). A human-curated mapping pass on those rows is the next lever.

Full per-row records: eval_results.json. Training: 139 steps (~69 min on 1× A10G), final loss 0.5489, token accuracy 0.857 — trackio dashboard.

Usage

Identical to V1 except the adapter id; the system prompt must be the V2 taxonomy prompt from the dataset card (the model was trained with it verbatim).

Limitations

Same as V1 plus: closed 8-class taxonomy trades fine-grained labels for consistency; ~406 rows (16% of vulnerable data) carry default-mapped Business Logic labels. Not a substitute for a professional audit.

Defensive security tooling: analyzes code you paste in. Only audit contracts you are authorized to review.

Evaluation — V2 vs V1 (same held-out code, n=100 stratified, greedy, seed 42)

The V2 test split is the identical code to V1's (split-preserving relabeling), so rates are directly comparable.

Table with columns: Metric, V1 adapter, V2 adapter
Metric	V1 adapter	V2 adapter
Detection precision	1.00	0.986
Detection recall	1.00	0.971
False-positive rate on clean	0.00	0.033 (1/30)
Finding-type match	0.30 (lenient substring/keyword)	0.485 (strict exact canonical match)

Read honestly:

The headline: type-labeling improved from 30% → 48.5% (33/68) under a stricter criterion — V2 must name the exact canonical class, V1 got credit for substring overlaps. The taxonomy normalization worked.
The cost: detection is no longer a flat 1.00/1.00 — V2 misses 2/70 vulnerable rows and flags 1/30 clean rows (Business Logic Violation on a token-fee constructor). V1's perfect detection was partly a dataset artifact (see the V1 card); V2's slight drop is the expected price of a tighter, closed-label output contract and is arguably more honest a number.
49% still means roughly half of detected findings get a coarser or adjacent canonical label. The remaining confusion concentrates where source labels were themselves ambiguous (the 406 fallback rows mapped to Business Logic by default). A human-curated mapping pass on those rows is the next lever.

Full per-row records: eval_results.json. Training: 139 steps (~69 min on 1× A10G), final loss 0.5489, token accuracy 0.857 — trackio dashboard.

Usage

Identical to V1 except the adapter id; the system prompt must be the V2 taxonomy prompt from the dataset card (the model was trained with it verbatim).

Limitations

Defensive security tooling: analyzes code you paste in. Only audit contracts you are authorized to review.

qwen25-coder-7b-solidity-audit-v2

README

Evaluation — V2 vs V1 (same held-out code, n=100 stratified, greedy, seed 42)

Usage

Limitations

Explore FriendliAI today

README

Evaluation — V2 vs V1 (same held-out code, n=100 stratified, greedy, seed 42)

Usage

Limitations