GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
alexliap
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
alexliap
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Model Configuration | Value |
|---|---|
| Layers | 18 |
| Hidden size | 768 |
| Intermediate size | 1792 |
| Attention heads | 12 (query) / 6 (KV, grouped-query attention) |
| Head dim | 64 |
| Context length | 2048 |
| Vocab size | 50,258 |
| Tied embeddings | yes |
| Precision | bf16 |
| Parameters | ~145M total, ~106M non-embedding |
A byte-level BPE tokenizer with 50,000 learned merges plus 256 byte tokens and 2 special tokens (<|begin_of_text|>, id 0; <|end_of_text|>, id 1, which also doubles as the pad token). The pre-tokenizer regex pipeline is structurally borrowed from Llama 3.2, but the vocabulary itself was trained from scratch on the bilingual corpus below.
The tokenizer's post-processor wraps every encode in <|begin_of_text|> ... <|end_of_text|> by default. This is convenient for training but wrong for a generation prompt (a trailing EOS tells the model the prompt is already a finished document). See the usage example below for how to bypass it.
Sourced with a 50/50 English/Greek token target:
| Source | Repo | Language |
|---|---|---|
| fineweb_edu | HuggingFaceFW/fineweb-edu | en |
| fineweb_hq_el | alexliap/high-quality-gr-text | el |
| finewiki_el | alexliap/high-quality-gr-text | el |
| wikipedia_el | alexliap/high-quality-gr-text | el |
| synth_faq | alexliap/greek-synth-v1 | el |
| synth_math | alexliap/greek-synth-v1 |
| Training Configuration | Value |
|---|---|
| GPUs | 8 |
| Batch size | 16 sequences/GPU x 2048 tokens = 262,144 tokens/step |
| Gradient accumulation | 1 |
| Optimizer | AdamW, lr 2e-4, betas (0.9, 0.95), eps 1e-10, weight_decay 0.001 |
| LR schedule | Linear warmup, 2000 steps |
| Tokens seen | ~9.33B (1 epoch, run stopped at corpus exhaustion) |
| Precision | bf16 |
All code, Hydra configs, and step-by-step reproduction commands (data download, tokenizer training, shard packing, and the training launch script) are in scripts/typakos_140m/ in the pretrain GitHub repo, under the README.md there:
setup.sh)train_tokenizer.py)measure_token_rates.py)download_data.py)concat_data.py)prepare_shards.py)train.sh, pointed at configs/train.yaml)This is a base model, so prompt it with plain text and let it continue. Because the tokenizer auto-wraps every encode in BOS/EOS, use add_special_tokens=False and prepend BOS manually to avoid signaling a "finished document" to the model:
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel_id = "alexliap/typakos-140m-base"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id)prompt = "The history of Athens begins"input_ids = tokenizer(tokenizer.bos_token + prompt, add_special_tokens=False, return_tensors="pt").input_idsoutput = model.generate(input_ids, max_new_tokens=100, do_sample=True, temperature=0.7)print(tokenizer.decode(output[0], skip_special_tokens=True))
Evaluated 0-shot with EleutherAI lm-evaluation-harness, except GSM8K at 5-shot, matching the shot conventions stated on the SmolLM2 model card.
| Benchmark | Typakos-140M-base | SmolLM2-135M-8k |
|---|---|---|
| HellaSwag (acc_norm) | 26.5 | 42.1 |
| ARC (avg acc_norm) | 28.3 | 43.9 |
| PIQA (acc_norm) | 56.6 | 68.4 |
| MMLU cloze (acc) | 23.4 | 31.5 |
| CommonsenseQA (acc) | 19.6 | 33.9 |
| TriviaQA (exact_match) | 0.0 |
The gap to SmolLM2-135M-8k below is largely a training-budget gap, not an architecture gap: per its model card, SmolLM2-135M was pretrained on about 2T tokens, roughly 200 times the ~9.33B tokens Typakos-140M-base has seen, on a curated English-only mix (FineWeb-Edu, DCLM, The Stack, plus additional filtered sources). Typakos-140M-base is a single-stage, from-scratch run on a much smaller, bilingual (EN/EL) corpus, so it is comparable in size and architecture but not yet in data scale.
Evaluated 0-shot on the ilsp_greek task suite (65 sub-tasks, including all 57 mmlu_greek subject splits). No SmolLM2 reference exists for these tasks, so they are reported standalone:
| Task | Typakos-140M-base |
|---|---|
| arc_challenge_greek (acc) | 20.2 |
| hellaswag_greek (acc) | 26.5 |
| mmlu_greek (acc, 57 subj.) | 23.0 |
| mgsm_direct_greek (exact_match) | 3.6 |
| winogrande_greek (acc) | 49.7 |
| truthfulqa_greek_mc1 (acc) | 23.9 |
| truthfulqa_greek_mc2 (acc) | 44.5 |
| medical_mcqa_greek (acc) | 15.3 |
| mcqa_greek_asep (acc) |
medical_mcqa_greek and mcqa_greek_asep have no English-benchmark equivalent and are reported for completeness only. winogrande_greek uses a different task formulation than the English winogrande row above (full-candidate-sentence plausibility scoring rather than fill-in-the-blank), so the two are not directly comparable.
MIT
| el |
| synth_table | alexliap/greek-synth-v1 | el |
| synth_tutorial | alexliap/greek-synth-v1 | el |
| 4.1 |
| Winogrande (acc) | 49.6 | 51.3 |
| OpenBookQA (acc_norm) | 33.2 | 34.6 |
| GSM8K 5-shot (exact_match) | 0.2 | 1.4 |
| 18.3 |