GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
zhoumiaosen
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
zhoumiaosen
Model Tree
zhoumiaosen/minimind-64m-pretrain
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Unique parameters | 63,912,192 |
| Architecture | Dense decoder-only MiniMind, exported as standard Qwen3ForCausalLM |
| Layers / hidden size | 8 / 768 |
| Attention heads / KV heads | 8 / 4 |
| Feed-forward size / vocabulary | 2,432 / 6,400 |
| Training context | 768 tokens |
| Configured position limit | 32,768; longer-context performance untested |
| Training / published precision | BF16 mixed precision / FP16 Safetensors |
| Stage | Full supervised fine-tuning; no preference optimization |
Qwen3 identifies the compatible export architecture, not the source of the pretrained weights. The base weights were trained from scratch with MiniMind, using its existing tokenizer. No pretrained Qwen weights were used. No custom remote model code is needed.
Install a PyTorch build appropriate for your platform, plus transformers==4.57.6 and safetensors.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizermodel_id = "zhoumiaosen/minimind-64m-sft"device = "cuda" if torch.cuda.is_available() else "cpu"dtype = torch.float16 if device == "cuda" else torch.float32tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=dtype).to(device).eval()messages = [{"role": "user", "content": "解释什么是机器学习"}]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,open_thinking=False,)inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(device)with torch.inference_mode():output = model.generate(**inputs, max_new_tokens=128, do_sample=False,eos_token_id=tokenizer.eos_token_id,pad_token_id=tokenizer.pad_token_id,)print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
The tokenizer's exported input names are set to input_ids and attention_mask to match Qwen3 generation. Vocabulary and weights are preserved. CPU generation is supported; Raspberry Pi performance has not been measured.
The fine-tuning run used sft_t2t_mini.jsonl from jingyaogong/minimind_dataset: 905,718 conversational records. Consult the upstream dataset card for provenance and terms. Data are not redistributed in this repository. The upstream SFT loader formats conversations with the tokenizer chat template and computes next-token loss on assistant response tokens. Exact non-padding training token counts were not recorded.
| Setting | Value |
|---|---|
| Epochs | 1 |
| Microbatch / gradient accumulation | 4 / 4 |
| Nominal effective batch | 16 sequences |
| Maximum sequence length | 768 |
| Final logged microbatch | 226,430 |
| Optimizer | AdamW with PyTorch defaults for betas, epsilon and weight decay |
| Learning rate | Cosine decay from 0.00001 toward 0.000001 |
| Gradient clipping / seed | 1.0 / 42 |
| Data-loader workers |
Run from the upstream trainer directory after placing the base checkpoint in out/pretrain_768.pth:
bash
python train_full_sft.py --epochs 1 --batch_size 4 \--accumulation_steps 4 --max_seq_len 768 --num_workers 0 \--dtype bfloat16 --device cuda:0 --from_resume 1 \--log_interval 100 --save_interval 1000
The export preserves the saved out/full_sft_768.pth artifact. The upstream trainer saves at the final microbatch before applying its trailing partial-accumulation optimizer step, so that subsequent in-memory update is not included. The base pretraining run had one reboot recovery; its model card documents the resume details.
| Measurement | Loss |
|---|---|
| First logged microbatch (100) | 2.4891 |
| Final logged microbatch (226,430) | 1.8517 |
| Mean of first 50 logged readings | 2.0311 |
| Mean of last 50 logged readings | 1.6723 |
These are training microbatch losses, not validation scores or full-epoch averages. The curve includes individual logged losses and a moving average over up to 50 readings. Raw readings are in fine-tuning-loss.csv. No held-out perplexity, standardized benchmark, factuality score, or safety evaluation was performed.
The original post-training GPU generation output is included unedited in evaluation.txt. It contains eight Chinese prompts with responses, generated with a 128-new-token limit; several responses are truncated. A separate export smoke check is saved in sample-generations.json with its decoding settings.
Observed issues in the original samples include an incorrect explanation of why the sky is blue, an invalid Fibonacci implementation, repetition, and confused descriptions of Chinese dishes. Generating readable text or obtaining lower training loss does not establish correctness. Outputs may also contain bias or inappropriate content. Multilingual quality, long-context behavior, and deployment throughput on other devices remain untested.
Intended uses: small-model training research, inference experiments, and further fine-tuning. This checkpoint should not be relied on for factual advice or correct executable code without independent verification.
a3c7b01cc004d5de86aea961f20bf1e638e7c09e.d94ce07efd1b8902f519c029bf9a4da8255685e2.abb1e76b2056e14728beb78db96b7b3c491a0bef1ed3e34a9b381b28f29fa518.verification.json records the source checkpoint hash, weight comparison, and export checks.Released under Apache 2.0, matching the upstream project; see LICENSE. Credit for the architecture implementation, tokenizer, training utilities, and data preparation belongs to the upstream contributors. This is an independently trained release, not an official upstream model.
| 0 |
| Log / checkpoint interval | 100 / 1,000 microbatches, plus the final microbatch |