Benchmark — v5
Sixty hand-reviewed commits (gold-v1), balanced across all eleven
conventional-commit types. Head-to-head of the previous release (v4) and the
current model (v5).
Table with columns: metric, v4, v5| metric | v4 | v5 |
|---|
| Type accuracy | 48.3% | 60.0% |
| Scope accuracy | 68.3% | 75.0% |
| Title word-overlap | 27.2% | 19.1% |
| Exact title | 6.7% | 0.0% |
| Generation errors / 60 | 7 | 4 |
Type accuracy across releases: v3 40% → v4 55% → v5 60%. v3/v4 were scored
on an earlier JS/TS-only set; v5 on the harder balanced set above, where v4
re-scores 48.3% — so v5's real gain over v4 is +11.7 points.
Honest read: v5 lifts type accuracy and scope and nearly halves the format
errors. Title word-overlap and exact-title fall because v5 paraphrases: it
writes its own titles, often more descriptive than the single human reference
it is scored against, which exact string-match penalizes. build and revert
remain hard.
Use it
Ollama
ollama run pointbreaklab/knot-scribe "$(git diff --staged)"
llama.cpp (base + this LoRA)
llama-server --model qwen2.5-coder-7b-instruct-q4_k_m.gguf \
--lora knot-scribe-lora.gguf
Output is JSON {"title": "...", "body": "..."} — a correctly-typed
Conventional-Commit subject and body.
Training
- Base: Qwen2.5-Coder-7B-Instruct · Apache-2.0
- Method: QLoRA (4-bit), completion-only SFT (loss on the message only)
- Data: diff → message pairs mined from permissive open-source commit
history (JS/TS, Python, Rust, Go)
- Privacy: public commit history only — no user data, no private code
- License: Apache-2.0 (derivative of Qwen2.5-Coder-7B)