nehabharti0802

rag-vs-finetune-legal-qlora

⚠️ Do not use this to answer legal questions

Used on its own, without retrieval, this adapter:

refuses only 15.0% of unanswerable questions, against the base model's 41.7% — fine-tuning made it 3× less willing to say "I don't know"
cites the correct statutory section 0.9% ± 1.0 of the time (n = 300 × 3 seeds)
cites a real but wrong section in 96.9% ± 2.0 of its answers
produces a perfectly formatted, confident-looking citation every time

It names the correct statute in 92.1% ± 1.7 of answers and the wrong section within it in about the same share. That combination is the problem. A citation to the wrong Act is obvious to any lawyer at a glance; a citation to the right Act with a plausible section number has to be looked up to be caught.

It is arguably more dangerous than the untuned base model, which names the wrong Act often enough (52.3%) to be visibly wrong. It is not measurably better at section citation either: 0.9% ± 1.0 against the base model's 0.3% is indistinguishable, and both are indistinguishable from zero.

Nothing this model produces is legal advice. Do not put it in front of anyone seeking legal information.

What it is actually good for

Reproducing the benchmark. That is its purpose.
As the fine-tuned arm on top of retrieval (A4), where it reaches 87.5% ± 1.4 correct-section — a +3.9 point gain over base+RAG's 83.7%, holding on all three seeds (worst-seed gap +2.3). The gain is real but modest, comes entirely from reduced generation-stage errors, and carries a latency penalty (below).

Training data

Synthetic QA generated from the bare text of four Indian statutes:

Table with columns: Act, Year, Replaces
Act	Year	Replaces
The Bharatiya Nyaya Sanhita	2023	Indian Penal Code, 1860
The Bharatiya Nagarik Suraksha Sanhita	2023	Code of Criminal Procedure, 1973
The Bharatiya Sakshya Adhiniyam	2023	Indian Evidence Act, 1872
The Indian Contract Act	1872	—

Source: India Code, the Government of India's official repository of central legislation.

Licensing. Reuse of the statutory text rests on §52(1)(q)(ii) of the Indian Copyright Act, 1957, which exempts reproduction of bare legislative text. This is a statutory exemption, not a licence: it is narrower than a permissive licence, carries no sublicense grant, and covers the bare text only — not commentary or headnotes. Anyone reusing this adapter inherits that constraint. Full reasoning in ATTRIBUTION.md.

The adapter weights are released under MIT, matching the code. The constraint above attaches to the corpus and to text the model reproduces from it, not to the weights themselves.

Provenance: 3,171 QA pairs (2,830 train / 341 val) over 1,090 sections. Splits were assigned at section level before any generation, so a training pair and an evaluation question cannot share a source passage. Verified by four decontamination checks; see reports/decontamination.md.

Generated by a local gemma4:e4b — deliberately not Qwen-family, since a same-family generator would distil its own style into the student.

Training

Table

Base	`Qwen/Qwen2.5-7B-Instruct`, 4-bit NF4, double-quant, bf16 compute
Method	QLoRA, rank 16, α/r = 2.0, dropout 0.05
Targets	all 7 projections — `q,k,v,o,gate,up,down`
Trainable	40,370,176 params (0.53%)
Schedule	lr 2e-4, cosine, 3% warmup, effective batch 8
Steps	353/epoch, packing off
Hardware	1× RTX A5000 24 GB, peak 10.91 GiB reserved
Cost

The released checkpoint is epoch 1, not epoch 3. Validation loss rose monotonically — 0.887 → 0.919 → 1.059 — so the model was already overfitting after one epoch. This replicates on all three seeds, with epoch-1 loss spanning just 0.8873–0.8892, so the checkpoint choice is structural rather than a lucky pick. Epoch 1 is the best checkpoint that exists, and the results above are what the best checkpoint achieves.

Evaluation

All headline metrics are judge-free — string matches against a registry of real acts and section numbers, not LLM-judged.

Table with columns: Metric, Base, This adapter, Base+RAG, Adapter+RAG
Metric	Base	This adapter	Base+RAG	Adapter+RAG
Correct section	0.3%	0.9% ± 1.0	83.7%	87.5% ± 1.4
Correct act	47.7%	92.1% ± 1.7	96.0%	98.2% ± 0.5
Fabrication rate	78.0%	96.9% ± 2.0	14.7%	12.0% ± 1.1

The LLM judge used for secondary metrics scored Cohen's κ = 0.452 against 100 human labels — below the pre-registered 0.60 threshold — so judged accuracy is reported as secondary only.

Forgetting (MMLU): in-domain −5.5pp, out-of-domain −2.5pp. Adaptation cost more in-domain legal capability than out-of-domain capability.

Inference cost: the adapter is used unmerged, so every forward pass pays extra LoRA matmuls. It is ~1.7× slower than the base model despite a 9× shorter prompt, and there is no query volume at which its training cost pays back. Merging would likely remove most of this; that was not measured.

Limitations

Three seeds (1, 2, 42), reported as mean ± std. Enough to establish that A4's gain over base+RAG is larger than the seed noise; not enough for a significance test.
Abstention is actively bad, and measured. On 60 hand-written unanswerable questions it refuses 15.0% of the time against the base model's 41.7% — a 3× regression, despite 10% of its training data being refusal examples. It refuses 0/12 questions about repealed statutes (IPC/CrPC/Evidence Act), which are the ones a real user is most likely to ask.
One hyperparameter configuration. No sweep was run, so "QLoRA fails at this" is really "this QLoRA configuration failed at this".
Public corpus. Indian statutes are public text. These results may not transfer to a genuinely private domain, which is the case fine-tuning is usually argued for.
MMLU has no Indian-law subject. The forgetting probe uses US-centric professional_law and jurisprudence, so it measures whether general legal reasoning survived, not whether Indian statutes were learned.
English only. No Hindi or other Indian-language statutory text.

Citation

markdown
@software{rag_vs_finetune_2026,
  title  = {RAG vs QLoRA fine-tuning: a 2x2 factorial benchmark on Indian statutes},
  author = {Bharti, Neha},
  year   = {2026},
  url    = {https://github.com/NehaBharti08/rag-vs-finetune}
}

Independent student project. Not affiliated with or endorsed by the Government of India.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

nehabharti0802

Model Tree

Base

Qwen/Qwen2.5-7B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

⚠️ Do not use this to answer legal questions

Used on its own, without retrieval, this adapter:

refuses only 15.0% of unanswerable questions, against the base model's 41.7% — fine-tuning made it 3× less willing to say "I don't know"
cites the correct statutory section 0.9% ± 1.0 of the time (n = 300 × 3 seeds)
cites a real but wrong section in 96.9% ± 2.0 of its answers
produces a perfectly formatted, confident-looking citation every time

Nothing this model produces is legal advice. Do not put it in front of anyone seeking legal information.

What it is actually good for

Reproducing the benchmark. That is its purpose.
As the fine-tuned arm on top of retrieval (A4), where it reaches 87.5% ± 1.4 correct-section — a +3.9 point gain over base+RAG's 83.7%, holding on all three seeds (worst-seed gap +2.3). The gain is real but modest, comes entirely from reduced generation-stage errors, and carries a latency penalty (below).

Training data

Synthetic QA generated from the bare text of four Indian statutes:

Table with columns: Act, Year, Replaces
Act	Year	Replaces
The Bharatiya Nyaya Sanhita	2023	Indian Penal Code, 1860
The Bharatiya Nagarik Suraksha Sanhita	2023	Code of Criminal Procedure, 1973
The Bharatiya Sakshya Adhiniyam	2023	Indian Evidence Act, 1872
The Indian Contract Act	1872	—

Source: India Code, the Government of India's official repository of central legislation.

The adapter weights are released under MIT, matching the code. The constraint above attaches to the corpus and to text the model reproduces from it, not to the weights themselves.

Generated by a local gemma4:e4b — deliberately not Qwen-family, since a same-family generator would distil its own style into the student.

Training

Table

Base	`Qwen/Qwen2.5-7B-Instruct`, 4-bit NF4, double-quant, bf16 compute
Method	QLoRA, rank 16, α/r = 2.0, dropout 0.05
Targets	all 7 projections — `q,k,v,o,gate,up,down`
Trainable	40,370,176 params (0.53%)
Schedule	lr 2e-4, cosine, 3% warmup, effective batch 8
Steps	353/epoch, packing off
Hardware	1× RTX A5000 24 GB, peak 10.91 GiB reserved
Cost

Evaluation

All headline metrics are judge-free — string matches against a registry of real acts and section numbers, not LLM-judged.

Table with columns: Metric, Base, This adapter, Base+RAG, Adapter+RAG
Metric	Base	This adapter	Base+RAG	Adapter+RAG
Correct section	0.3%	0.9% ± 1.0	83.7%	87.5% ± 1.4
Correct act	47.7%	92.1% ± 1.7	96.0%	98.2% ± 0.5
Fabrication rate	78.0%	96.9% ± 2.0	14.7%	12.0% ± 1.1

The LLM judge used for secondary metrics scored Cohen's κ = 0.452 against 100 human labels — below the pre-registered 0.60 threshold — so judged accuracy is reported as secondary only.

Forgetting (MMLU): in-domain −5.5pp, out-of-domain −2.5pp. Adaptation cost more in-domain legal capability than out-of-domain capability.

Limitations

Three seeds (1, 2, 42), reported as mean ± std. Enough to establish that A4's gain over base+RAG is larger than the seed noise; not enough for a significance test.
Abstention is actively bad, and measured. On 60 hand-written unanswerable questions it refuses 15.0% of the time against the base model's 41.7% — a 3× regression, despite 10% of its training data being refusal examples. It refuses 0/12 questions about repealed statutes (IPC/CrPC/Evidence Act), which are the ones a real user is most likely to ask.
One hyperparameter configuration. No sweep was run, so "QLoRA fails at this" is really "this QLoRA configuration failed at this".
Public corpus. Indian statutes are public text. These results may not transfer to a genuinely private domain, which is the case fine-tuning is usually argued for.
MMLU has no Indian-law subject. The forgetting probe uses US-centric professional_law and jurisprudence, so it measures whether general legal reasoning survived, not whether Indian statutes were learned.
English only. No Hindi or other Indian-language statutory text.

Citation

markdown
@software{rag_vs_finetune_2026,
  title  = {RAG vs QLoRA fine-tuning: a 2x2 factorial benchmark on Indian statutes},
  author = {Bharti, Neha},
  year   = {2026},
  url    = {https://github.com/NehaBharti08/rag-vs-finetune}
}

Independent student project. Not affiliated with or endorsed by the Government of India.

rag-vs-finetune-legal-qlora

README

⚠️ Do not use this to answer legal questions

What it is actually good for

Training data

Training

Evaluation

Limitations

Citation

Explore FriendliAI today

README

⚠️ Do not use this to answer legal questions

What it is actually good for

Training data

Training

Evaluation

Limitations

Citation