Model details
Table with columns: Component, Configuration| Component | Configuration |
|---|
| Unique parameters | 10,366,256 total |
| Language model | 9,628,496, including the tied embedding and output weights |
| Multi-token prediction (MTP) | 656,704 parameters, one auxiliary decoder layer |
| Vision tower | 81,056 parameters, left at initialization during text training |
| Text decoder | 12 layers with nine Gated DeltaNet blocks and three gated full-attention blocks |
| Hidden and MLP widths | 256 and 512 |
| Full attention | Four query heads, two key/value heads, head width 32 |
| Tokenizer | Byte-level BPE with 16,384 entries |
| Training sequence length | 256 tokens |
| Weight format | FP32 SafeTensors, 41.49 MB |
The text decoder uses partial multimodal rotary position embeddings, which
operate along the text sequence for this run. MTP provides an auxiliary
two-token-ahead objective during training. The example below uses ordinary
autoregressive decoding.
Quick start
The CPU loading path was tested with PyTorch 2.9.0 and Transformers 5.22.2.
Load the original tokenizer with TokenizersBackend. The default Qwen
tokenizer class can apply different pre-tokenization rules to this vocabulary.
python -m pip install "torch==2.9.0" "transformers==5.22.2"
import torchfrom transformers import Qwen3_5ForConditionalGeneration, TokenizersBackend model_id = "MatheusSoranco/cafezin-lm-10m-tinystories"tokenizer = TokenizersBackend.from_pretrained(model_id)model = Qwen3_5ForConditionalGeneration.from_pretrained(model_id).eval() prompt = "Once upon a time, there was a little fox"inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt") torch.manual_seed(42)with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=50, do_sample=True, temperature=0.8, top_k=20, top_p=0.95, ) completion = output[0, inputs["input_ids"].shape[1]:]print(tokenizer.decode(completion, skip_special_tokens=True))
Sample continuation excerpt from the tested CPU run:
who lived in a big forest. The fox was very small and had a long tail. He loved to run and play with his friends in the forest.
The checkpoint also loads with the native cafezin-lm generate command.
Native CLI instructions
Training
The run reused a locally preprocessed subset of 500,000 TinyStories
documents. The document-level split contained 108,637,240 training tokens
and 1,102,319 held-out tokens. Training sampled contiguous windows from the
training split. The tokenizer was reused from the local preprocessing run.
The exact upstream dataset revision and tokenizer fitting split are unknown. TinyStories dataset and license
Table with columns: Setting, Value| Setting | Value |
|---|
| Initialization | Random, seed 42 |
| Objective | Next-token cross-entropy plus MTP cross-entropy weighted by 0.3 |
| Optimizer | AdamW with β₁ = 0.9, β₂ = 0.95, weight decay 0.1 |
| Learning rate | 150-step linear warmup to 3 × 10⁻⁴, then cosine decay to 3 × 10⁻⁵ |
| Gradient clipping | Global norm 1.0 |
| Batch | Four sequences per microbatch, four accumulation steps |
| Training budget | 5,000 optimizer steps, 20,480,000 sampled target tokens |
| Precision | FP32 parameters with TF32 matrix multiplication enabled |
| Hardware | NVIDIA GeForce RTX 3050 Laptop GPU, 6 GB |
| Runtime | 38 min 23 s on Windows with PyTorch 2.9.0 and CUDA 12.8 |
| Peak GPU memory reserved by PyTorch | 864 MiB |
The source revision is pinned below. The relative paths in the
training configuration refer to the expected local
data and output folders. Training code revision
Evaluation
The final weights were evaluated on 1,102,080 target tokens from the
local held-out split. This pass used 4,305 consecutive windows of 256
next-token predictions. Each window started with fresh model state. The
evaluation used the main language-model head in FP32 and averaged
cross-entropy over all target tokens.
Table with columns: Metric, Step 5,000| Metric | Step 5,000 |
|---|
| Mean next-token negative log-likelihood | 2.3179 nats/token |
| Perplexity | 10.1544 |
| Top-1 next-token accuracy | 50.48% |
The training log tracked the joint language-model and MTP objective on 16
sampled validation batches per check. That logged loss was 5.5012 at step
200 and 3.1204 at step 5,000. The table above reports the separate fixed-window
evaluation. Machine-readable evaluation
The exported package was reloaded in both Cafezin LM and Transformers. All
209 stored tensors matched the training checkpoint exactly. Text logits
agreed on the test prompt within FP32 tolerance, and the exported model
generated a continuation on CPU.
Intended use and limitations
- This checkpoint is designed for studying small language models and completing
short, simple English stories.
- It received only text pretraining. The vision tower is present for architectural
completeness, with its original random initialization. Use text prompts with
the released checkpoint.
- The model can repeat phrases, lose narrative consistency, and generate
inaccurate content. Instruction following, Portuguese fluency, contexts
beyond 256 tokens, and safety behavior were outside this evaluation.
- The reported scores describe the local TinyStories holdout with the released
tokenizer. Performance on other benchmarks requires separate evaluation.
Files and license
The release contains the FP32 weights, SafeTensors index, tokenizer,
model and generation settings, training configuration, and evaluation summary.
The SHA-256 of the weight shard is
2951e023ffa8a71472bc19f81c04b2520a97927cf7f793f7463b70470e2a77cd.
The Cafezin LM release is distributed under Apache 2.0.
TinyStories is by Ronen Eldan and Yuanzhi Li and is distributed under
CDLA-Sharing-1.0.
TinyStories paper