GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
VinayRevanuru
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
VinayRevanuru
Model Tree
Input Modalities
Output Modalities
Supported Functionality
python
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "VinayRevanuru/Sapling-505M-Base"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto") inputs = tok("The best way to cook rice is", return_tensors="pt")out = model.generate( **inputs, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1,)print(tok.decode(out[0], skip_special_tokens=True))Use sampling. Greedy decoding loops quickly at this scale (see Limitations).
| Architecture | Qwen3ForCausalLM (dense): GQA, QK-Norm, RMSNorm, RoPE, SwiGLU |
| Parameters | 505,483,776 (embeddings tied) |
| Layers / width / FFN | 30 / 1,024 / 3,072 |
| Attention heads | 16 query, 8 key/value, head dim 128 |
| Context length | 2,048 tokens |
| RoPE θ | 1,000,000 |
| Vocabulary | 32,768 byte-level BPE, trained from scratch (`< |
| Weights | FP32 safetensors (≈2.0 GB) |
Where the parameters go: feed-forward 283.1M (56%), attention 188.7M (37%), tied embedding 33.6M (7%).
The mix is 80/20 by consumed training tokens, counted with Sapling's own tokenizer:
sample-10BT): filtered educational web text.smollm-corpus, cosmopedia-v2): synthetic textbook-style text, text field only.How the data was prepared:
| Tokens seen | 5,000,396,800 (76,300 optimizer steps) |
| Global batch | 65,536 tokens (32 × 2,048) |
| Optimizer | fused AdamW, β = (0.9, 0.95), ε = 1e-8, weight decay 0.1 (norms/biases excluded) |
| Learning rate | peak 3e-4, 1% linear warmup, cosine decay |
| Gradient clipping | 1.0 |
| Precision | BF16 autocast, FP32 master weights |
| Hardware | 8 × NVIDIA A100 80GB (PyTorch DDP) on Modal |
| Throughput / wall clock | ≈142k tokens/s, ≈10.1 h (≈81 GPU-hours) |
| Software | torch 2.6.0, transformers 4.51.3, CUDA 12.4 |
Note on the schedule: the cosine was configured for a 7B-token run, and training was stopped early at 5B. This checkpoint therefore has not been fully annealed: the learning rate at the final step was about 8.2e-5, against a planned floor of 3e-5. A short cooldown would likely lower its loss somewhat.
Training loss fell from 10.2 (random init; ln 32768 ≈ 10.4) to about 2.75 (100-step average). The repeating dips in the curve are Cosmopedia shards, which are easier to predict than web text. Median single-step loss in the last block of each source:
| Source | Loss at ~0.3B tokens | Loss at ~5.0B tokens |
|---|---|---|
| FineWeb-Edu | 3.45 | 2.83 |
| Cosmopedia v2 | 2.59 | 1.72 |
These are training losses on unseen data, since every shard is read once. Held-out validation losses were not recorded for this checkpoint.
The best way to cook rice is to boil it, which is a great way to prepare a whole meal that is often enjoyed as a side dish or as a main course. If you're trying to make a quick breakfast, you can make a simple rice pudding. Simply mix together rice, eggs, milk…
The capital of France is Paris, located in the North-West of France. France is a country of great culture and history, with a rich heritage of art, music,eff, and cuisine…
The largest planet in our solar system is Mercury, which is a gas giant planet, about the size of Earth…
The last example is included deliberately. Fluent, confident and wrong is typical of a 0.5B base model.
Research and education on small-model pretraining, a starting point for instruction tuning or LoRA experiments, and a baseline for tokenizer and data-mix ablations.
training_summary.json in this repo records the full configuration, data manifest, tokenizer sha256 (f7d39923…0be0984), and dependency versions.
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "VinayRevanuru/Sapling-505M-Base"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto") inputs = tok("The best way to cook rice is", return_tensors="pt")out = model.generate( **inputs, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1,)print(tok.decode(out[0], skip_special_tokens=True))