balrampandey

indian-tax-law-slm-base

README

License: apache-2.0

Highlights

🎯 Purpose-built for Indian legal & tax language — statutes, case law, and procedure
📊 10.6× stronger on legal text than on general English (perplexity: 40.7 vs 431.7)
🧱 Standard Llama architecture — loads with AutoModelForCausalLM, no custom code
🔧 Fine-tuning ready — clean base for LoRA/PEFT adaptation to specific legal tasks
🪶 Custom 16K tokenizer trained on the domain for efficient legal text encoding
🔓 Fully open — Apache-2.0, permissively-licensed training data

Model Summary

Table

Architecture	`LlamaForCausalLM` (grouped-query attention, RoPE, SwiGLU)
Parameters	31.8M
Layers	8
Hidden size	512
Attention heads	8 (4 KV heads)
Context length	512 tokens
Vocabulary	16,000 (custom byte-level BPE)
Precision	fp16
Corpus	~430M tokens, ~68% Indian legal by weight

Evaluation

Held-out perplexity (lower = better):

Table with columns: Domain, Perplexity
Domain	Perplexity
Indian legal	40.7
General English	431.7

The model is 10.6× more confident on Indian legal text than on general text — clear, quantitative evidence of domain specialization. This is the intended behavior of a domain base model: it internalizes the vocabulary, structure, and register of Indian statutory and case law.

Try it live

An interactive demo is available as a Hugging Face Space:

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("balrampandey/indian-tax-law-slm-base")
tok = AutoTokenizer.from_pretrained("balrampandey/indian-tax-law-slm-base")

ids = tok("The assessee", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=40, repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))

Fine-tuning

The model is designed as a base for adaptation. Attach a LoRA adapter with peft:

python
from peft import LoraConfig, get_peft_model

model = get_peft_model(model, LoraConfig(
    task_type="CAUSAL_LM", r=8, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
))

Training Data

A curated mix of general English (for language competence) and open Indian legal text (for domain specialization):

Table with columns: Source, Content, Licence
Source	Content	Licence
allenai/c4	General English	ODC-By
KanoonGPT/indian-legal-documents	Legal documents	Apache-2.0
KanoonGPT/indian-case-laws	Case law	Apache-2.0
169Pi/indian_law	Legal Q&A / reasoning	Apache-2.0
vihaannnn/Indian-Supreme-Court-Judgements-Chunked	SC judgments	MIT
varshith7/income-tax-act-india

The underlying text of Acts and judgments is exempt from copyright under s.52(1)(q) of the Indian Copyright Act; dataset compilations retain the licences above.

Intended Use

A base model for fine-tuning on Indian legal and tax tasks (LoRA/PEFT)
Research on domain-specialized language modeling
A foundation for the larger models in this series

As an early, compact checkpoint, generation fluency is limited and outputs should not be treated as authoritative legal or tax information. Factual grounding via retrieval is planned for larger releases.

Roadmap

This model is the first step in a staged program:

✅ v0 — 32M base (this release) — validated pipeline, measurable specialization
🔄 In progress — larger base models — improved fluency and coherence at greater scale
🔜 Planned — retrieval-augmented variants — factual grounding on live statutory text
🔜 Planned — instruction-tuned releases — task-ready models for legal workflows

License

Apache-2.0 (code and weights).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

balrampandey

Model Tree

Base

this model

Input Modalities

Text

Output Modalities