Training
Table | |
|---|
| Parameters | 1.23B (2048 hidden, 22 layers, 32 heads, 4 KV heads, tied embeddings) |
| Tokeniser | bpe-128k, vocabulary 128,000 |
| Corpus | FineWeb-Edu (English) |
| Tokens | 20B (76,293 steps x 128 sequences x 2048 tokens) |
| Sequence length | 2048 |
| Optimiser | AdamW (0.9, 0.95), weight decay 0.1, grad clip 1.0 |
| LR | 6e-4, 2000 warmup / 2000 decay steps, min ratio 0.01 |
| Precision | bf16 |
| Seed | 42 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("AhmetcanYvz/comp-vs-like-1B-bpe-128k")
model = AutoModelForCausalLM.from_pretrained("AhmetcanYvz/comp-vs-like-1B-bpe-128k")
ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=20)[0]))
The tokeniser is bundled with the model, so no separate download is needed.
Caveats
This is a research model trained on 20B tokens to study tokenisation. It is not
instruction-tuned, not filtered for safety, and not intended for deployment.
Code
https://github.com/Ahmetcanyvz/comp-vs-like
Citation
@inproceedings{yavuz2026objective,
title = {Objective vs. Search: Decomposing What Makes a Good Tokeniser},
author = {Yavuz, Ahmetcan and Meister, Clara and Pimentel, Tiago},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2026}
}