zhoumiaosen

minimind-64m-pretrain

README

License: apache-2.0

Model at a glance

Table with columns: Property, Value
Property	Value
Model family	MiniMind, dense decoder-only Transformer
Parameters	Approximately 63.91 million unique parameters
Layers / hidden size	8 / 768
Attention heads / KV heads	8 / 4
Feed-forward size	2,432
Vocabulary	6,400 tokens
Training sequence length	768 tokens
Configured position limit	32,768; long-context performance was not tested
Training precision	BF16 mixed precision
Published weight precision	FP16, Safetensors
Initialization	Random model weights; upstream MiniMind tokenizer
Hardware	One NVIDIA RTX 3060 12 GB; approximately 8 GB system RAM

Quick start

Tested with Python, PyTorch 2.6.0 and Transformers 4.57.6. Install a PyTorch build suitable for your platform, then:

bash
pip install 'transformers==4.57.6' safetensors

After publication, load the model directly from the Hub. For a local download, set model_id to its directory:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "zhoumiaosen/minimind-64m-pretrain"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=dtype
).to(device).eval()

# Use a continuation prompt: this checkpoint is not an instruction model.
inputs = tokenizer("机器学习是一种", return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

No trust_remote_code=True is required. A tokenizer chat template may be present because the upstream tokenizer includes one; that does not make this checkpoint instruction-tuned. CPU inference is supported by the exported architecture. Raspberry Pi performance has not been measured.

Training data and procedure

The run used pretrain_t2t_mini.jsonl from jingyaogong/minimind_dataset, containing 1,270,238 records. This is a Chinese-oriented training setup; language coverage and English capability have not been independently measured. Consult the upstream dataset card for its provenance and terms. The dataset is not redistributed here.

Each record is tokenized independently, truncated to 766 text tokens, wrapped with beginning/end tokens, and padded to 768 tokens. Padding labels are ignored. The objective is causal next-token prediction. Exact non-padding training token counts were not recorded.

Table with columns: Setting, Value
Setting	Value
Epochs	1
Microbatch size	4 sequences
Gradient accumulation	64 microbatches
Nominal effective batch	256 sequences
Final logged microbatch	317,560
Optimizer	AdamW; PyTorch defaults for betas, epsilon and weight decay
Learning rate	Cosine decay from 0.0005 toward 0.00005
Gradient clipping	1.0
Seed

The machine restarted during pretraining. The run resumed from microbatch 306,000, repeating the unsaved tail. Resume restored model and optimizer state, but did not preserve partially accumulated gradients; this was not a bit-for-bit uninterrupted run.

This release preserves the upstream out/pretrain_768.pth artifact exactly through format conversion. The upstream trainer saves at the final microbatch before its trailing partial-accumulation optimizer step, so that later in-memory update is not included in the saved artifact.

Training loss

Pretraining loss

Table with columns: Measurement, Loss
Measurement	Loss
First logged microbatch (100)	8.4318
Final logged microbatch (317,560)	2.4801
Mean of first 50 logged readings	6.7344
Mean of final 50 logged readings	1.9887

These are individual training-microbatch losses, not held-out validation results or full-epoch averages. The curve keeps the last logged reading when a step was repeated after recovery. The smoothed line averages up to 50 logged readings. The machine-readable readings are included in pretraining-loss.csv.

No held-out perplexity, standardized benchmark, factuality score, or safety evaluation is reported. Falling training loss does not establish useful question-answering ability. Any sample continuations in sample-generations.json are generation smoke checks, not quality benchmarks. Chat samples produced by a separate fine-tuned checkpoint are not results for this release.

Observed base-model continuations

A CPU smoke check used greedy decoding (do_sample=False, max_new_tokens=64) with the exported model:

Table with columns: Prompt, Decoded continuation
Prompt	Decoded continuation
`机器学习是一种`	Empty after special-token removal
`在一个宁静的早晨，`	`，，，，，，，，，，，，`

These two probes show poor continuation quality with these prompts and settings. The export loads and generates successfully, but this checkpoint should be treated as a training experiment rather than a usable assistant. Both outputs and generation settings are preserved in sample-generations.json.

The exported tokenizer's model_input_names was set to input_ids and attention_mask to match the Qwen3 generation interface; token vocabulary and model weights were not changed.

Intended uses and limitations

This checkpoint is intended for studying small-model pretraining, experimenting with inference, and serving as a starting point for supervised fine-tuning. It is not a reliable assistant. Outputs may be repetitive, incoherent, factually incorrect, biased, or inappropriate. Code completions may be invalid. Long-context behavior, multilingual performance, and Raspberry Pi throughput remain untested.

Reproducibility and attribution

Training implementation: jingyaogong/minimind.
Source revision: a3c7b01cc004d5de86aea961f20bf1e638e7c09e.
Training dataset SHA256: 6dd6716c84ab36897bdbfc7f88e04f4441c48c1ab7ecee88ce0b0e7d4685560c.
Training environment: PyTorch 2.6.0+cu124, Transformers 4.57.6, Datasets 3.6.0.
Original checkpoint hash and export checks are recorded in verification.json.

Command run from the upstream repository's trainer directory:

bash
python train_pretrain.py --epochs 1 --batch_size 4 \
  --accumulation_steps 64 --max_seq_len 768 --num_workers 0 \
  --dtype bfloat16 --device cuda:0 --from_resume 1 \
  --log_interval 100 --save_interval 1000

The upstream project is distributed under Apache 2.0; its license is included in LICENSE. This release uses the same license. Credit for the architecture implementation, tokenizer, training utilities, and dataset preparation belongs to the upstream contributors. This is an independently trained checkpoint, not an official upstream release.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

zhoumiaosen

Model Tree

Base

this model

Input Modalities

Text

Output Modalities