chao68681

smollm2-135m-bnc-continued-pretraining

README

License: apache-2.0

Results

All targets in the held-out splits are evaluated using the same document-preserving 512-token protocol. NLL is summed over nonmasked next-token targets and divided by their total count; perplexity is exp(NLL). Validation selects the checkpoint; test does not.

Table with columns: Split, Pretrained NLL, Pretrained PPL, Selected NLL, Selected PPL, PPL change
Split	Pretrained NLL	Pretrained PPL	Selected NLL	Selected PPL	PPL change
Validation	3.252213	25.8475	3.112411	22.4752	-13.05%
Test	3.179929	24.0450	3.039538	20.8956	-13.10%

Validation covers 84 documents / 2,391,296 targets; test covers 77 documents / 2,669,143 targets. The test split has been evaluated in earlier stages and is not claimed to be untouched. These results demonstrate adaptation to this corpus distribution; they do not establish improved general reasoning or instruction following. Pretraining overlap with BNC is unknown.

Model and training

Base: HuggingFaceTB/SmolLM2-135M, revision 93efa2f097d58c2a74874c7e644dbc9b0cee75a2; 134,515,008 parameters.
Data: [BNC1994 archive](https://llds.ling-phil.ox.ac.uk/llds/xmlui/handle/20.500.14106 / 2554?show=full), 4,049 XML documents; three exact duplicates removed. Fixed source-group split: 3,885 train / 84 validation / 77 test documents, seed 42.
Every training pass includes 127,787,903 supervised targets. Within-document chunks overlap by one token; first labels and padding are masked, EOS is supervised. Documents are not concatenated across boundaries.
Three complete logical epochs were executed: 7,877 updates each, 23,631 logical updates total. The published checkpoint is at 22,877 logical updates (continuation step 15,000), selected by the lowest eligible complete-validation NLL, rather than automatically using the final weights.
One T4 per Colab training runtime; effective batch 32 sequences, FP32 parameters with FP16 autocast, fused AdamW, weight decay 0.01, clipping 1.0. Epoch 1: peak LR 5e-5, 3% warmup, seed 42. Epochs 2–3: one continuous cosine schedule, peak LR 1e-5, 2% warmup, seed 43. Full validation every 500 continuation updates and each epoch boundary.
Epoch 3 followed the predeclared validation-only gate after epoch 2. All completed epoch weights and checkpoints were preserved privately. Saved-model logits were checked after export/reload in Colab.

Reproduction and experiment scope

Use the notebook with a legitimately obtained private corpus copy. All extraction, training, recovery, evaluation and publication source is included in one English notebook. The main executable path reproduces epoch 1; the full continuation/recovery executor in the appendix completes epochs 2–3 from its saved state. Historical attempts are clearly labelled and should not all be executed.

Development included a subset pilot, a full-data restart, a stopped 574-update continuation, three unsuccessful weight interpolations, and interrupted zero-update context preparations. Recovery replayed observed update positions:200,400 and 155 for the 512 branch; 350 for 1024; 242 UI-observed positions (225 timed positions) for 2048. Each recovery restored the same checkpoint weights, optimizer, scheduler, scaler and RNG. Causes of the unexpected runtime losses are unknown. Replay adds compute and is excluded from logical epoch counts. A complete cost/GPU-hour ledger is unavailable.

The independent 1024 and 2048 context comparisons each completed exactly two full epochs. The 1024 full held-out evaluation and export/reload verification passed. The 2048 checkpoint was frozen using complete validation results. Its complete held-out evaluation and export/reload verification also passed, and its T4 was released. Their best complete shared 512-token validation NLL/PPL were 3.113861 / 22.5078 and 3.115813 / 22.5518 respectively. Neither improved on the selected 512 model. Selection used the identical complete shared 512-token validation set, with a minimum replacement gain of 0.0005 NLL; test metrics did not determine selection. Supplementary evaluations at longer contexts are reported separately in context_comparison.json. Chunk boundaries, effective batch sizes and schedule details differ, so these runs are not a strict causal context-length ablation.

Development used multiple independent model initializations and concurrent single-T4 runtimes, exceeding the original exercise's one-model / overall-one-T4 restrictions. Each individual run used one T4. One selected model is published, and previous public revisions remain recoverable.

Use

python

from transformers import AutoTokenizer, AutoModelForCausalLMrepo = "chao68681/smollm2-135m-bnc-continued-pretraining"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo)inputs = tokenizer("The history of the British", return_tensors="pt")output = model.generate(**inputs, max_new_tokens=40, do_sample=False,                        pad_token_id=tokenizer.eos_token_id)print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations and licensing

This compact model can generate inaccurate, biased or memorized text. BNC reflects a historical British-English distribution, and aggregate perplexity does not guarantee useful responses. The corpus is not redistributed: no raw text, document/source identifiers, Arrow caches, optimizer states or private document-level diagnostics are included. The base model is released under Apache 2.0; see LICENSE. This model license grants no rights to the underlying BNC material; obtain and use that material under its applicable terms.

Selected-weight subgroup diagnostics (completed 28 October 2026)

The exact published 512-token weights were identified by SHA-256 before evaluation. All 84 validation and 77 test documents were evaluated again to obtain document-level scores; the previously verified pinned-base document scores were reused. No optimizer ran. Aggregate NLL agrees with the existing complete evaluation within 1e-4. These are measurements of the published checkpoint, replacing the need to rely on historical epoch-one subgroup results.

Table with columns: Split / kind, Documents, Base PPL, Published PPL, Relative PPL change
Split / kind	Documents	Base PPL	Published PPL	Relative PPL change
validation / all	84	25.8475	22.4752	-13.05%
validation / written	72	25.6418	22.4418	-12.48%
validation / spoken	12	31.0472	23.2551	-25.10%
test / all	77	24.0450	20.8956	-13.10%
test / written	59	23.4949	20.8007	-11.47%
test / spoken	18	31.1272	21.9843	-29.37%

Paired source-group bootstrap: 2,000 replicates, seed 42. All six 95% intervals for selected-minus-base NLL are below zero. This is finite-corpus uncertainty, not evidence of out-of-domain capability. Full aggregate intervals and target counts are in subgroup_diagnostics.json; document identifiers and document-level records remain private. Test results did not select weights or hyperparameters.

Completed comparisons are recorded in context_comparison.json. The current model weights are unchanged; the prior verified public load remains applicable to the identical weight hash.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

chao68681

Model Tree

Base

HuggingFaceTB/SmolLM2-135M

Fine-tuned

this model

Input Modalities

Text

Output Modalities