maheshcr

saslm-18m-mula

README

License: apache-2.0

What this model is for

The paper asks whether adding secondary material — recorded commentary on a primary corpus — improves a small model on that primary corpus. This checkpoint is the control arm: trained on primary text only, no commentary. It is released so that others can reproduce the comparison.

The commentary-trained arms are not released pending permission from the commentary's author. See Training data.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("maheshcr/saslm-18m-mula")
tok = AutoTokenizer.from_pretrained("maheshcr/saslm-18m-mula")

ids = tok("The Supermind is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True,
                     temperature=0.8, top_k=50, pad_token_id=0)
print(tok.decode(out[0]))

Context length is 256 tokens. The tokenizer is a 30,000-token BPE trained on the domain corpus, so it is efficient on this vocabulary and poor on general text.

Model details

Table

Architecture	GPT-2 decoder, 4 layers, 6 heads, d=384
Parameters	18.7M
Context	256 tokens
Vocabulary	30,000 BPE
Training	60,000 steps, batch 32, AdamW, lr 6e-4 cosine, dropout 0.15
Seed	1337
Hardware	1x RTX 3070 Ti

The released weights are bit-identical to the checkpoint used in the paper; conversion to the transformers layout was verified to zero logit difference.

Evaluation

Held-out validation set of 337,514 tokens (1,502,530 bytes), split at the document level and decontaminated to 0.3% verbatim overlap with training.

Table with columns: Model, Params, BPB ↓, Perplexity
Model	Params	BPB ↓	Perplexity
This model	18.7M	1.3564	65.74
Same model + 20% commentary	18.7M	1.3453	63.50
GPT-2-small, zero-shot	124M	1.4703	—
GPT-2-small, fine-tuned on same data	124M	1.1604	—

Bits per byte is used for cross-model comparison because the tokenizers differ; per-token perplexity is not comparable across different vocabularies.

Note the last row: a fine-tuned GPT-2-small is substantially better than this model. If you want a good model for this corpus, fine-tune a pretrained one. This model beats GPT-2-small only in the zero-shot setting, at 6.6x fewer parameters.

On SASLM-Eval (see the dataset card), this model scores 36.0% / 29.1% / 35.5% across three seeds on a 406-item cloze task with a 25% chance baseline. Above chance, but no model we tested is distinguishable from another on this benchmark.

Training data

Table with columns: Source, Included here, Redistributed
Source	Included here	Redistributed
Complete Works of Sri Aurobindo (23 volumes)	Yes	No
Sraddhalu Ranade lecture transcripts	No	No
The Mother, Collected Works	No	No

This checkpoint was trained on Sri Aurobindo's works only. Sri Aurobindo died in 1950, so the works are out of copyright in India under life+60; the 2005 edition used carries a Sri Aurobindo Ashram Trust copyright notice. The corpus itself is not redistributed. A manifest with SHA-256 hashes is in the code repository so that independently obtained sources can be verified as identical.

Memorisation

Measured before release. 100 greedy prompts of 50 tokens per split, generating 100 tokens and measuring the longest matching continuation:

Table with columns: Split, Mean matched, Max, ≥20 tokens
Split	Mean matched	Max	≥20 tokens
Training	0.49	6	0%
Validation	0.53	3	0%

The model does not reproduce its training data. The longest verbatim match found anywhere was six tokens.

Limitations

Small and undertrained. Validation loss was still improving at the step budget; this is not a converged model.
Worse than fine-tuning a pretrained model on the same data, by 13.7% bits per byte. Do not use this model where quality matters.
256-token context. It cannot handle long passages.
Single domain, single author, single register. Generations imitate the surface style of early-20th-century English philosophical prose and will be incoherent outside it.
It does not know what it is saying. On a benchmark testing whether the model selects the correct ontological term from a same-family set, it scores around 33% against a 25% baseline. Fluent output should not be read as doctrinal accuracy.
Not suitable for religious, spiritual or scholarly guidance. It generates plausible-sounding text in the register of a spiritual teacher without any correspondence to what that teacher actually taught. Do not present its output as Sri Aurobindo's views.

Citation

bibtex
@misc{saslm2026,
  title  = {Commentary as a Substitute for Pretraining: Mixing Secondary Text
            into a Small Language Model for a Philosophical Corpus},
  author = {Mahesh C.R.},
  year   = {2026}
}

Contact

If you are a rights holder with a concern about this release, open an issue on the model repository and it will be addressed.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

maheshcr

Model Tree

Base

this model

Input Modalities

Text

Output Modalities