GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Marcoson320
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Marcoson320
Model Tree
Marcoson320/codeparrot-gpt2-mi50
Input Modalities
Output Modalities
Supported Functionality
The base model was trained following HuggingFace LLM Course Chapter 7.6, whose tokenize function does not insert <|endoftext|> at document boundaries:
python
def tokenize(element): outputs = tokenizer( element["content"], truncation=True, max_length=128, return_overflowing_tokens=True, ... ) # no EOS inserted between documentsGPT-2 paper and karpathy/nanoGPT both insert an end-of-text token between documents; the course's simplification produces a model that does not stop generating at natural boundaries.
This fine-tune attempts to retrofit the EOS signal post-hoc.
| Item | Value |
|---|---|
| Base | Marcoson320/codeparrot-gpt2-mi50 (final checkpoint) |
| Optimizer | AdamW (β₁=0.9, β₂=0.999, weight_decay=0.1) |
| Learning rate | 5×10⁻⁵, cosine schedule, 100 warmup steps |
| Effective batch size | 256 (per_device_bs=32 × grad_accum=4 × world_size=2) |
| Steps | 4,000 |
| Precision | fp16 |
| Parallelism | DistributedDataParallel on 2 × MI50 |
| Wall clock | ~1h 44m |
| final train_loss | 1.162 |
| final eval_loss | 1.569 |
Each training sample is a variable-length token slice (32–120 tokens) of a Python file, with <|endoftext|> (id 0) appended explicitly, then padded to 128 with a label mask of -100 (so padding does not contribute to loss).
This raises EOS signal density from < 0.04% (base training) to ~3%.
Tested on five self-contained prompts, comparing base vs fine-tuned model. EOS emission within 80 generated tokens, greedy decoding, repetition_penalty=1.12, no_repeat_ngram_size=4:
| Prompt | Base | EOS-FT |
|---|---|---|
def add(a, b):\n return a + b\n | ✗ | ✓ (pos 13) |
def square(x):\n return x * x\n\n | ✗ | ✗ |
def greet(name):\n print(f'Hello {name}')\n\n | ✗ | ✓ (pos 47) |
import os\nprint(os.getcwd())\n | ✗ | ✗ |
FunctionDef / ClassDef blocks via ast.parse so the model only sees EOS at structural endpoints.python
from transformers import pipeline pipe = pipeline( "text-generation", model="Marcoson320/codeparrot-gpt2-mi50-eos-ft", device=0,) out = pipe( "def add(a, b):\n return a + b\n", max_new_tokens=80, do_sample=False, repetition_penalty=1.12, no_repeat_ngram_size=4,)print(out[0]["generated_text"])Training script and full method documentation: bundle on the project HTTP server (LAN only). Source code mirrored alongside this model is the published train_eos_v2.py and test_eos.py.
Marcoson320/codeparrot-gpt2-mi50x = 1\ny = 2\nz = x + y\n | ✗ | ✓ (pos 61) |
| Total emit rate | 0/5 | 3/5 |