GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
alexliap
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
alexliap
Model Tree
alexliap/typakos-140m-base
Input Modalities
Output Modalities
Supported Functionality
Same architecture as the base model, with the vocabulary grown by 3 chat-template special tokens (added in the SFT stage, see below).
| Model Configuration | Value |
|---|---|
| Layers | 18 |
| Hidden size | 768 |
| Intermediate size | 1792 |
| Attention heads | 12 (query) / 6 (KV, grouped-query attention) |
| Head dim | 64 |
| Context length | 2048 |
| Vocab size | 50,261 (base 50,258 + 3 chat-template special tokens) |
| Tied embeddings | yes |
| Precision | bf16 |
| Parameters | ~145M total, ~106M non-embedding |
Starts from the base model's byte-level BPE tokenizer (50,000 merges + 256 byte tokens +
<|begin_of_text|>/<|end_of_text|>). The SFT stage adds three Llama-3-style special tokens
(<|start_header_id|>, <|end_header_id|>, <|eot_id|>) and resizes the (tied) embedding matrix
to match, with the new rows mean/covariance-initialized rather than random.
The chat template (adapted from TRL's llama3_training.jinja) renders each turn as:
markdown
<|start_header_id|>ROLE<|end_header_id|>CONTENT<|eot_id|>
and wraps assistant turns in {% generation %} / {% endgeneration %} markers, which the SFT
stage uses to mask the training loss to assistant tokens only (assistant_only_loss=True).
Trained from scratch, full-parameter, on ~9.33B tokens split roughly 50/50 English/Greek. See
alexliap/typakos-140m-base for the full
pretraining data mix and procedure.
Teaches the base checkpoint the chat format and instruction-following, via trl.SFTTrainer.
Data: alexliap/typakos_sft_dataset:
7 filtered subsets from 4 upstream Hub sources, kept only if well-formed (an optional single
leading system turn, then alternating user/assistant turns ending on assistant) and
token-bounded to fit the 2048-token context, then shuffled and split 90/10 train/validation.
| source | upstream dataset | language | rows (final) |
|---|---|---|---|
| dolci_el | openeurollm/Dolci-Instruct-SFT-translated | el | 448,718 |
| eu_instruct_el | openeurollm/EU-Instruct-Synthetic | el | 137,988 |
| aya_el | CohereLabs/aya_dataset (Greek subset) | el | 623 |
| aya_en | CohereLabs/aya_dataset (English subset) | en | 3,938 |
~ 901M tokens total (~ 394M Greek, ~ 507M English), split into 936,491 train / 104,055 validation conversations (shuffle seed 0, 10% held out).
| Training Configuration | Value |
|---|---|
| Trainer | trl.SFTTrainer |
| Batch size | 16/device |
| Gradient accumulation | 1 |
| Optimizer | AdamW, lr 2e-5, betas (0.9, 0.95), eps 1e-10, weight_decay 0.01 |
| LR schedule | Cosine, warmup 10% of steps |
| Epochs | 1.0 |
| Loss masking | Assistant turns only (assistant_only_loss) |
| Precision |
Aligns the SFT model to preference pairs via trl.DPOTrainer, starting from
Stage 1's last checkpoint.
Data: openeurollm/Dolci-Instruct-DPO-translated
(el + en configs): concatenates both language configs, shuffles (seed 0), and splits off 10% as validation. Rows are
prompt/chosen/rejected conversational triples.
| split | rows |
|---|---|
| train | 423,954 |
| validation | 47,107 |
| Training Configuration | Value |
|---|---|
| Trainer | trl.DPOTrainer |
| Batch size | 8/device |
| Gradient accumulation | 1 |
| Optimizer | AdamW, lr 5e-6, betas (0.9, 0.95), eps 1e-10, weight_decay 0.01 |
| LR schedule | Constant with warmup, 500 warmup steps |
| Loss | sigmoid (standard DPO), beta 0.1 |
| Epochs | 1.0 |
| Precision | bf16 |
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel_id = "alexliap/typakos-140m-it"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id)messages = [{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}]input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")output = model.generate(input_ids, max_new_tokens=200, do_sample=False, temperature=0.7)print(tokenizer.decode(output[0][input_ids.shape[-1] :], skip_special_tokens=True))
Greedy-decoded (do_sample=False) chat completions from the final DPO checkpoint. Chosen to show
both the model's capability and its main failure mode: without a repetition penalty, greedy
decoding tends to lock onto a phrase and loop it.
English -- What is the capital of France?
The capital of France is Paris, which is the capital of France, and it is the capital of the French Empire. The capital is the capital of the French Empire, which is the capital of France. [...loops]
Greek -- Ποια είναι η πρωτεύουσα της Ελλάδας;
Η πρωτεύουσα της Ελλάδας είναι η Αθήνα, η οποία είναι η πρωτεύουσα της Ελλάδας. Η πρωτεύουσα της Ελλάδας είναι η Αθήνα, η οποία είναι η πρωτεύουσα της Ελλάδας. [...loops]
Greek -- Τι είναι η τεχνητή νοημοσύνη;
Η τεχνητή νοημοσύνη (AI) είναι μια μορφή τεχνητής νοημοσύνης που χρησιμοποιείται για την ανάλυση και την ανάλυση δεδομένων. Είναι μια μορφή που χρησιμοποιείται για την ανάλυση δεδομένων, την ανάλυση δεδομένων και την ανάλυση δεδομένων. [...loops]
Both examples get the underlying fact right (Paris, Athens) and stay grammatical throughout, but repeat themselves rather than moving on once they lock onto a clause. Sampling avoids the repetition but trades it for occasional factual drift; see Limitations.
Evaluated 0-shot with EleutherAI lm-evaluation-harness, except BBH (3-shot) and GSM8K (5-shot). The model's own chat template was applied for IFEval; the remaining tasks were run as raw completion/multiple-choice scoring without the chat template, matching Open LLM Leaderboard-style methodology.
| Benchmark | Typakos-140M-it | SmolLM2-135M-Instruct |
|---|---|---|
| IFEval (avg of 4 sub-metrics) | 15.6 | 29.9 |
| HellaSwag (acc_norm) | 26.7 | 40.9 |
| ARC (avg acc_norm) | 28.4 | 37.3 |
| PIQA (acc_norm) | 55.8 | 66.3 |
| MMLU cloze (acc) | 23.5 | 29.3 |
| BBH 3-shot (acc_norm) | 28.6 |
BBH here uses
leaderboard_bbh, a multiple-choice reformulation, rather than the original free-form chain-of-thought task: a model this size cannot yet produce coherent multi-step reasoning text, so this number reflects loglikelihood-based answer ranking, not demonstrated step-by-step reasoning.
Evaluated 0-shot on the ilsp_greek task suite (65 sub-tasks, including all 57 mmlu_greek subject splits), without the chat template. No SmolLM2 reference exists for these tasks, so they are reported standalone:
| Task | Typakos-140M-it |
|---|---|
| arc_challenge_greek (acc) | 19.6 |
| hellaswag_greek (acc) | 26.6 |
| mmlu_greek (acc, 57 subj.) | 22.9 |
| mgsm_direct_greek (exact_match) | 1.6 |
| winogrande_greek (acc) | 50.9 |
| truthfulqa_greek_mc1 (acc) | 24.2 |
| truthfulqa_greek_mc2 (acc) | 44.3 |
| medical_mcqa_greek (acc) | 16.0 |
| mcqa_greek_asep (acc) |
medical_mcqa_greek and mcqa_greek_asep have no English-benchmark equivalent and are reported for completeness only. winogrande_greek uses a different task formulation than the English winogrande row above (full-candidate-sentence plausibility scoring rather than fill-in-the-blank), so the two are not directly comparable.
MIT
| smol_constraints | HuggingFaceTB/smoltalk (smol-constraints) | en | 34,423 |
| smol_magpie_ultra | HuggingFaceTB/smoltalk (smol-magpie-ultra) | en | 361,514 |
| smol_rewrite | HuggingFaceTB/smoltalk (smol-rewrite) | en | 53,342 |
| TOTAL | 1,040,546 |
| bf16 |
| 28.2 |
| GSM8K 5-shot (exact_match) | 1.1 | 1.4 |
| 18.7 |