litillabs

litil-clausecheck-1.5b

README

License: apache-2.0

What this model does

LiTiL ClauseCheck is a contract-classification adapter that checks whether a specific provision appears in a piece of contract text. It handles 38 common review questions covering terms such as liability, assignment, renewal, termination, intellectual property, and confidentiality. For each clause and question, it returns a clean Yes or No instead of drafting a narrative answer.

That makes it useful as the tagging layer in a contract-intelligence system. After an agreement is parsed into clauses, ClauseCheck can apply the relevant provision questions and turn the results into a clause/provision map. The system can then use those tags to organize a contract repository, populate a review screen, build focused review queues, and send selected clauses to the right playbook or specialist workflow.

Useful for: clause tagging, contract indexing, review queues, and workflow routing
Give it: one contract clause and one of the 38 supported provision questions
It returns: Yes or No

Table with columns: Item, Value
Item	Value
Hugging Face repository	`litillabs/litil-clausecheck-1.5b`
Evaluated source revision	`a727ff1b491fcf136ac13876c48254b2f69a5f32`
Base model	`Qwen/Qwen2.5-1.5B-Instruct`
Tested base revision	`989aa7980e4cf806f80c7fef2b1adb7bc71aa306`
Format	PEFT LoRA adapter, 17,462,432 bytes
Artifact SHA-256	`9cc078eb899bccaf3c1c1b0482351cb1dd092bf7a778a3da6ef2c27c911a0866`
Input	One clause and one provision question

Input and output contract

Use the Qwen chat template with the exact system instruction shown above. The user message must use this shape:

text
Clause:
{clause}

Question: {question}

Answer (Yes or No):

Decode greedily with max_new_tokens=4 and parse only an exact Yes or No. Training and evaluation truncated the complete rendered prompt to 1,024 tokens; the longest retained evaluation prompt was 747 tokens.

Measured results

Saved predictions were recounted directly from the retained run outputs.

Table with columns: Evaluation slice, Cases, Qwen base, LiTiL ClauseCheck
Evaluation slice	Cases	Qwen base	LiTiL ClauseCheck
All clause/question pairs	5,162	78.86%	92.27%
Clause text absent from the training pool	2,553	74.66%	92.32%

Table with columns: Full-split measure, Qwen base, LiTiL ClauseCheck
Full-split measure	Qwen base	LiTiL ClauseCheck
Present-provision recall	59.21%	89.18%
Present-provision precision	98.22%	95.21%
Absent-provision recall	98.90%	95.42%
Brier score, lower is better	0.1769	0.0559
Parse failures	0	0

The two high base-model scores are not bad. They show that the base model was conservative: when it answered Yes, it was usually right (98.22% precision), and it correctly rejected most absent provisions (98.90% absent-provision recall). Its weakness was missing provisions that were actually present, where recall fell to 59.21%.

LiTiL ClauseCheck gives up about three points on those already-strong measures while raising present-provision recall to 89.18% and overall accuracy from 78.86% to 92.27%. That is a more useful balance for contract triage because the system finds substantially more of the provisions that need to be tagged and reviewed. These figures come from the retained 38-question CUAD evaluation.

Training and data

The data builder prepared 6,526 examples from the public CUAD tasks distributed through nguha/legalbench. The training script reserved the last 300 rows for validation and fitted on 6,226 rows. Twenty-nine task categories appear in the training pool and all 38 appear in evaluation. No private contract data was used.

Training used supervised fine-tuning for one epoch with learning rate 2e-4, cosine scheduling, warmup ratio 0.05, seed 44, and a LoRA adapter with rank 16, alpha 32, and dropout 0.05. The retained run log records MPS execution in FP32 with per-device batch size 2 and gradient accumulation 4.

Runtime sizing

The 1.5B base contains about 3 GB of BF16 weights and the adapter adds about 17 MB. Allow roughly 4–6 GB of accelerator memory for short prompts in BF16 after runtime overhead. Four-bit base weights contain about 0.8 GB of weight data; a practical short-prompt setup typically needs more memory for quantization metadata, activations, and the KV cache.

Intended use

Place LiTiL ClauseCheck after document parsing and clause segmentation. Run the questions relevant to the agreement type against each clause, then store each answer with the document ID, clause location, provision name, and model revision. Aggregating those results creates a structured clause/provision index for the contract.

That index can support:

searchable provision labels in a contract repository;
review queues for clauses covered by a particular playbook rule;
portfolio views showing where provisions were found;
automatic handoff of tagged clauses to extraction, comparison, or playbook models; and
reviewer screens that open directly to the relevant contract language.

ClauseCheck works as the classification layer in that stack. Pair it with an extraction model when you need the exact term or value, and with a playbook model when you need a policy recommendation or redline.

Use the model

Install the runtime:

bash
python -m pip install -U torch transformers peft accelerate safetensors huggingface_hub

This example pins the base revision used for the saved evaluation.

python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_ID = "Qwen/Qwen2.5-1.5B-Instruct"
BASE_REVISION = "989aa7980e4cf806f80c7fef2b1adb7bc71aa306"
ADAPTER_ID = "litillabs/litil-clausecheck-1.5b"

SYSTEM_PROMPT = (
    "You are a precise legal contract analyst. Read the clause carefully and "
    "answer the question with exactly one word: 'Yes' or 'No'. Do not explain."
)


def make_user_prompt(clause: str, question: str) -> str:
    return f"Clause:\n{clause}\n\nQuestion: {question}\n\nAnswer (Yes or No):"


dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
tokenizer = AutoTokenizer.from_pretrained(BASE_ID, revision=BASE_REVISION)
base = AutoModelForCausalLM.from_pretrained(
    BASE_ID,
    revision=BASE_REVISION,
    dtype=dtype,
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base,
    ADAPTER_ID,
).eval()

clause = (
    "EXCEPT IN CONNECTION WITH A BREACH BY EITHER PARTY OF ARTICLE 10, THE "
    "INDEMNIFICATION OBLIGATIONS OF PAPEREXCHANGE UNDER SECTIONS 12.4(c) "
    "[Indemnification by PaperExchange] AND THE INDEMNIFICATION OBLIGATIONS "
    "OF VERTICALNET UNDER SECTION 12.5(c) [Indemnification by VerticalNet], "
    "NEITHER PARTY WILL BE LIABLE FOR ANY SPECIAL, INDIRECT, CONSEQUENTIAL, "
    "EXEMPLARY OR INCIDENTAL DAMAGES ARISING OUT OF OR RELATED TO THIS "
    "AGREEMENT, HOWEVER CAUSED AND UNDER ANY THEORY OF LIABILITY (INCLUDING "
    "NEGLIGENCE), EVEN IF SUCH PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF "
    "SUCH DAMAGES."
)
question = (
    "Does the clause cap or limit liability? Note: 'sole remedy' or "
    "'exclusive remedy' provisions ARE liability limitations."
)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": make_user_prompt(clause, question)},
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    truncation=True,
    max_length=1024,
).to(next(model.parameters()).device)

with torch.inference_mode():
    generated = model.generate(
        **inputs,
        max_new_tokens=4,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

answer = tokenizer.decode(
    generated[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
).strip()
if answer not in {"Yes", "No"}:
    raise ValueError(f"Unexpected model output: {answer!r}")
print(answer)

Saved adapter output for that example:

text
Yes

The saved base-model output was No; the reference answer was Yes.

Developed by LiTiL Labs.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

litillabs

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities