GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Marcoson320
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Marcoson320
Model Tree
Marcoson320/codeparrot-gpt2-mi50
Input Modalities
Output Modalities
Supported Functionality
The base model was trained following HuggingFace LLM Course Chapter 7.6, whose tokenize function does not insert <|endoftext|> at document boundaries:
python
def tokenize(element): outputs = tokenizer( element["content"], truncation=True, max_length=128, return_overflowing_tokens=True, ... ) # no EOS inserted between documentsGPT-2 paper and karpathy/nanoGPT both insert an end-of-text token between documents; the course's simplification produces a model that does not stop generating at natural boundaries.
This fine-tune attempts to retrofit the EOS signal post-hoc.
| Item | Value |
|---|---|
| Base | Marcoson320/codeparrot-gpt2-mi50 (final checkpoint) |
| Optimizer | AdamW (β₁=0.9, β₂=0.999, weight_decay=0.1) |
| Learning rate | 5×10⁻⁵, cosine schedule, 100 warmup steps |
| Effective batch size | 256 (per_device_bs=32 × grad_accum=4 × world_size=2) |
| Steps | 4,000 |
| Precision | fp16 |
| Parallelism | DistributedDataParallel on 2 × MI50 |
| Wall clock | ~1h 44m |
| final train_loss | 1.162 |
| final eval_loss | 1.569 |
Each training sample is a variable-length token slice (32–120 tokens) of a Python file, with <|endoftext|> (id 0) appended explicitly, then padded to 128 with a label mask of -100 (so padding does not contribute to loss).
This raises EOS signal density from < 0.04% (base training) to ~3%.
Tested on five self-contained prompts, comparing base vs fine-tuned model. EOS emission within 80 generated tokens, greedy decoding, repetition_penalty=1.12, no_repeat_ngram_size=4:
| Prompt | Base | EOS-FT |
|---|---|---|
def add(a, b):\n return a + b\n | ✗ | ✓ (pos 13) |
def square(x):\n return x * x\n\n | ✗ | ✗ |
def greet(name):\n print(f'Hello {name}')\n\n | ✗ | ✓ (pos 47) |
import os\nprint(os.getcwd())\n | ✗ | ✗ |
x = 1\ny = 2\nz = x + y\n | ✗ | ✓ (pos 61) |
| Total emit rate | 0/5 | 3/5 |
FunctionDef / ClassDef blocks via ast.parse so the model only sees EOS at structural endpoints.python
from transformers import pipeline pipe = pipeline( "text-generation", model="Marcoson320/codeparrot-gpt2-mi50-eos-ft", device=0,) out = pipe( "def add(a, b):\n return a + b\n", max_new_tokens=80, do_sample=False, repetition_penalty=1.12, no_repeat_ngram_size=4,)print(out[0]["generated_text"])Training script and full method documentation: bundle on the project HTTP server (LAN only). Source code mirrored alongside this model is the published train_eos_v2.py and test_eos.py.
Marcoson320/codeparrot-gpt2-mi50def tokenize(element): outputs = tokenizer( element["content"], truncation=True, max_length=128, return_overflowing_tokens=True, ... ) # no EOS inserted between documentsfrom transformers import pipeline pipe = pipeline( "text-generation", model="Marcoson320/codeparrot-gpt2-mi50-eos-ft", device=0,) out = pipe( "def add(a, b):\n return a + b\n", max_new_tokens=80, do_sample=False, repetition_penalty=1.12, no_repeat_ngram_size=4,)print(out[0]["generated_text"])