Harsh

collie-ent-direct-0.6b

README

License: apache-2.0

Usage

Use the project CLI for the checkpoint's prompt and decoding behavior:

bash
git clone https://github.com/harshsinghal/COLLIE
cd COLLIE
python3.12 -m venv .venv
.venv/bin/pip install "torch>=2.6,<3" "transformers>=5.14.1,<6"
.venv/bin/python collie.py --file report.txt

For direct Transformers use, copy the CLI's SYSTEM prompt. A language-neutral output contract is published with the checkpoint as collie-card.schema.json.

Compatibility for this checkpoint:

Python 3.10+
Transformers 5.14.1 through the latest compatible 5.x release
PyTorch 2.6+

Training data and procedure

Base: Qwen/Qwen3-0.6B
Direct supervised fine-tuning
5,480 public-corpus teacher inputs across email, extracted PDFs, public chat, public source code, Apache JIRA, and system logs
4,655 final training rows after parsing, schema, register-holdout, and compound-value filters
Apache JIRA and LogHub were held out from final training
Source documents are not redistributed because their upstream terms vary

See the repository for sampling code, prompt versions, experimental artifacts, and the negative-result journal.

Evaluation

On the original 3,705-document reference-free faithfulness evaluation, the published run reported:

79% precise emitted subjects
9% wrong emitted subjects
0.39 missed major subjects per document
92% correct artifact type

These values were produced by an LLM judge and should not be interpreted as human-ground-truth accuracy. The repository's evaluation-v2 tooling adds strict response schemas, per-document audit artifacts, source slices, document-clustered bootstrap intervals, a second-judge path, and a human-audit sample. The original split also contained 18 exact train/evaluation duplicate texts; evaluation v2 removes exact and near duplicates. All historical model outputs parsed as JSON, but 3,630/3,705 (98.0%) satisfy the stricter v1.1 card schema; v2 reports the remainder as model failures rather than dropping them.

Limitations

English-dominant training data.
Inputs are text. PDFs must be text-extracted first; the CLI can do this with its documents extra.
The training excerpts were capped at 4,000 characters. The CLI chunks longer documents, but cross-chunk aggregation has not yet received a human benchmark.
The 0.6B model sometimes fills all three subject slots and may add a merely related third subject. The next data revision explicitly penalizes quota filling and rebalances subject cardinality.
Audience, purpose, and temporal orientation can require organizational context absent from the document.
Document text is untrusted input. The CLI delimits it and instructs the model not to follow embedded directions, but small language models are not a security boundary.

Do not use COLLIE as the sole basis for legal, employment, access-control, or security decisions.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Harsh

Model Tree

Base

Qwen/Qwen3-0.6B

Fine-tuned

this model

Input Modalities

Text

Output Modalities