GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
DedeProGames
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
DedeProGames
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Recreation of SupraLabs/MicroSupra-1k with exactly 10,000 parameters, trained on a mix of Fineweb-Edu + DCLM-Edu, entirely on CPU.
LlamaForCausalLM)5e-3 (cosine decay → 5e-4, warmup 200 steps)Mix 50/50 (interleaved), tokenized on the fly (streaming):
HuggingFaceFW/fineweb-edu — config sample-10BTHuggingFaceTB/dclm-edu — filtered to edu_int_score >= 3 (as recommended for small models)20M train tokens + 1M val tokens, packed into 256-token windows.
| MicroSupra-1k (original) | MicroSupra-10k (this) | |
|---|---|---|
| Parameters | 1,046 | 10,000 |
| Train data | Fineweb-Edu 300M tok | Fineweb-Edu + DCLM-Edu 20M tok |
| Final train loss | 6.046 | 5.679 |
| Val loss (same held-out val.bin) | 6.063 | 5.697 |
| Perplexity (same val) | 429.7 | 298.0 |
~10× parameters → −0.37 nats, ~31% lower perplexity. Still a bacteria model 🦠 — it doesn't know facts, but scaling laws are real.
Prompt: "My name is " Output: *"My name is .
ASSa, thels
..."*
Prompt: "Question: What is the capital of France?\nAnswer: " Output: *"Question: What is the capital of France? Answer: .15
S,e.
ec Thet..."*
python
from huggingface_hub import hf_hub_downloadfrom transformers import LlamaForCausalLM, PreTrainedTokenizerFastimport torch tokenizer = PreTrainedTokenizerFast( tokenizer_file=hf_hub_download("DedeProGames/MicroSupra-10k", "tokenizer.json"), bos_token="<s>", eos_token="</s>", pad_token="<pad>", unk_token="<unk>",)model = LlamaForCausalLM.from_pretrained("DedeProGames/MicroSupra-10k").eval() inputs = tokenizer("Question: What is the capital of France?\nAnswer: ", return_tensors="pt")with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.35, top_p=0.85, repetition_penalty=1.2, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id)print(tokenizer.decode(out[0], skip_special_tokens=True))Full pipeline in scripts/:
bash
pip install torch --index-url https://download.pytorch.org/whl/cpu transformers datasetspython scripts/prepare_data.py # streams both datasets, builds data/{train,val}.binpython scripts/train.py # asserts exactly 10,000 params, trains, saves to out/python scripts/inference.py # demo generationsTrained where the GPU can't reach: pure CPU, a 41KB safetensors file, and a model that still can't talk — but now with 10× more neurons to not talk with 🤖🫶
from huggingface_hub import hf_hub_downloadfrom transformers import LlamaForCausalLM, PreTrainedTokenizerFastimport torch tokenizer = PreTrainedTokenizerFast( tokenizer_file=hf_hub_download("DedeProGames/MicroSupra-10k", "tokenizer.json"), bos_token="<s>", eos_token="</s>", pad_token="<pad>", unk_token="<unk>",)model = LlamaForCausalLM.from_pretrained("DedeProGames/MicroSupra-10k").eval() inputs = tokenizer("Question: What is the capital of France?\nAnswer: ", return_tensors="pt")with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.35, top_p=0.85, repetition_penalty=1.2, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id)print(tokenizer.decode(out[0], skip_special_tokens=True))pip install torch --index-url https://download.pytorch.org/whl/cpu transformers datasetspython scripts/prepare_data.py # streams both datasets, builds data/{train,val}.binpython scripts/train.py # asserts exactly 10,000 params, trains, saves to out/python scripts/inference.py # demo generations