NyayaLabs98

nyaya-3b-v3

README

License: apache-2.0

Intended use & limitations

Use for: plain-language legal information, "how do I…" procedural guidance, old→new law (IPC↔BNS) mapping, terminology.
Do not use for: actual legal advice, court filings, or any decision without a licensed advocate. Coverage is limited to the acts in the statute DB; it can be wrong or incomplete outside them.

Evaluation (Nyaya-Eval-v0, 500 frozen questions)

Primary metric is citation accuracy verified against the statute DB, not loss. Numbers below are strict exact-match on the frozen eval, with dense RAG (k=8):

Table with columns: Setting, Strict acc., Lenient acc.
Setting	Strict acc.	Lenient acc.
Base + dense RAG	3.4%	—
Nyaya-3B-v3 + dense RAG	3.8%	5.5%

On the held-out dataset eval, citation pass-rate is 90.2% across 1068 examples.

Honest status: strict exact-match on the frozen benchmark is still low and the project's human-eval ship gate has not been passed. This is an early research release, not a validated "best" legal model. See the repo's docs/ROADMAP.md go/no-go gates.

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("NyayaLabs98/nyaya-3b-v3")
model = AutoModelForCausalLM.from_pretrained("NyayaLabs98/nyaya-3b-v3", device_map="auto")

system = (
    "You are Nyaya, an Indian legal information model. You provide accurate, "
    "plain-language legal guidance for Indian citizens, cite specific sections of "
    "current law (BNS/BNSS/BSA and other acts in force), clearly state uncertainty, "
    "and recommend consulting a licensed advocate for anything consequential. "
    "You provide legal information, not legal advice."
)
# For best results, prepend retrieved statute passages to the user turn (RAG).
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "Police FIR nahi likh rahi, kya karu?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Training curves & ablations

Program progression RAG ablation Checkpoint frozen-eval Before/after behaviour

Training details

See training/ in this repo for the exact config, metric history, and all figures (PNG + source CSV): train_v3.yaml, training_history_v3.json, rag_eval_rescored.json, checkpoint_evals.json, figures/.

License & attribution

Model weights released under Apache-2.0, inheriting the base model's license. Statutory text is Government of India material (India Code / legislative.gov.in). See the repository NOTICE for data provenance.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

NyayaLabs98

Model Tree

Base

Qwen/Qwen2.5-3B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities