GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
DedeProGames
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
DedeProGames
Model Tree
Input Modalities
Output Modalities
Supported Functionality
This repository holds the weights at optimizer step 81,500 of a run targeting 200B tokens. Three things follow from that, and all three matter when reading the benchmark numbers:
| Field | Value |
|---|---|
| Tokens seen by this checkpoint | 42,729,472,000 (42.7B) |
| Target token budget | 200,000,000,000 (200B) |
| Progress | 21.4% |
| Optimizer step | 81,500 |
| Planned total steps | ~381,470 |
| Learning rate at this checkpoint | 1.2e-3 (peak, stable phase) |
| LR decay begins at step | ~343,320 (final 10% of the run) |
The learning rate has not decayed yet. The run uses a warmup-stable-decay schedule whose decay phase covers only the final 10% of steps. This checkpoint sits deep in the stable phase, still at the peak LR of 1.2e-3. Models evaluated mid-plateau consistently score below what the same weights reach after annealing, so the numbers below understate the model rather than describe it.
It has seen fewer tokens than Kiyo-135M. Kiyo-135M was trained on the full 200B tokens. This preview has seen 42.7B. A parameter-count comparison between the two is not a fair comparison of the architectures — it is a comparison of a finished run against a fifth of one.
Weights are not final and will change. Do not treat this checkpoint as the Kiyo-230M release. It exists so the run can be inspected mid-flight; the final model will be published separately.
| Field | Value |
|---|---|
| Parameters | 229,688,064 |
| Architecture | Llama-style decoder (SmolLM2 architecture family) |
| Layers | 32 |
| Hidden size | 768 |
| Intermediate size | 1,920 |
| Attention heads | 12 |
| KV heads | 4 |
| Head dimension | 64 |
| Attention type | Grouped query attention |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Positional encoding | RoPE (theta 10,000) |
| Vocabulary size | 49,152 |
| Context length | 2,048 |
| Embeddings | Tied input/output embeddings |
| Training tokens (this checkpoint) | 42,729,472,000 |
| Tokens per optimizer step | 524,288 |
| Peak learning rate | 1.2e-3 |
| LR schedule | Warmup-stable-decay (1/sqrt), decay not yet reached |
| Weight format | safetensors |
| Source | Domain | Mixture share |
|---|---|---|
| FineWeb-Edu | General web text, education-filtered | 45.0% |
| DCLM-Baseline | General web text, high-quality filtered | 30.0% |
| Stack-v3-train | Source code | 17.5% |
| FineMath | Mathematical reasoning | 7.5% |
The Stack-v3 portion is sampled with per-language keep probabilities across 32 curated programming languages, giving each language an equal token quota rather than inheriting the corpus's natural skew.
Scores are percentages from zero-shot acc_norm,none evaluations. Bold numbers indicate Kiyo-230M-Preview scores. Results may vary with evaluation settings.
| Benchmark | Kiyo-230M-Preview | Supra2-100M-Base | GPT-X2.5-135M | SmolLM2-135M |
|---|---|---|---|---|
| ARC-Easy | 49.54% | 47.81% | 51.81% | 58.63% |
| PIQA | 64.09% | 65.40% | 69.42% | 68.44% |
| ARC-Challenge | 26.11% | 24.83% | 29.18% | 29.69% |
| HellaSwag | 35.94% | 35.98% | 40.57% | 43.22% |
bash
pip install -U transformers safetensors torchpython
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM model_id = "DedeProGames/Kiyo-230M-Preview" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,).cuda().eval() prompt = "The meaning of life is "input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device) with torch.no_grad(): output = model.generate( input_ids=input_ids, max_new_tokens=64, do_sample=False, repetition_penalty=1.1, pad_token_id=tokenizer.eos_token_id, eos_token_id=tokenizer.eos_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True))This is an unfinished base model, not instruction-tuned — it continues text rather than following instructions, and its weights are an intermediate checkpoint rather than a release. Beyond the preview caveats above, it shares the limits of its scale: fluent and grammatical on language completion, competent on code, but weak on multi-step logic and context tracking, where it stays near chance on the harder items. The 2,048-token context window limits long-document use. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.
Apache 2.0
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM model_id = "DedeProGames/Kiyo-230M-Preview" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,).cuda().eval() prompt = "The meaning of life is "input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device) with torch.no_grad(): output = model.generate( input_ids=input_ids, max_new_tokens=64, do_sample=False, repetition_penalty=1.1, pad_token_id=tokenizer.eos_token_id, eos_token_id=tokenizer.eos_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True))