Model details

Files

Use with llama.cpp

bash
Copy code
# Download
wget https://huggingface.co/Thorstin/gpt2-dutch-instruct/resolve/main/dutch-gpt2-q8_0.gguf

# Run
llama-cli -m dutch-gpt2-q8_0.gguf \
  -p "### Instructie:\nWat is de hoofdstad van Nederland?\n### Antwoord:\n" \
  -n 200

Use with Ollama

bash
Copy code
# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./dutch-gpt2-q8_0.gguf
TEMPLATE """### Instructie:
{{ .Prompt }}
### Antwoord:
"""
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.3
PARAMETER num_ctx 512
EOF

ollama create dutch-gpt2 -f Modelfile
ollama run dutch-gpt2

Training

Phase 1 — Pretraining from scratch

• Dataset: CC-100 Dutch (~37 GB raw, ~6.6B tokens), streamed
• Tokenizer: ByteLevel BPE trained on first 500K CC-100 Dutch documents
• Hardware: NVIDIA Tesla T4 (16 GB VRAM)
• Tokens trained: ~5B
• Steps: 154,000
• Final loss: 3.54
• Duration: ~70 GPU hours
• Key settings: fp16=True, gradient_checkpointing=True, batch_size=32, lr=5e-4, cosine scheduler

Phase 2 — Instruction fine-tuning (SFT)

• Dataset: BramVanroy/alpaca-cleaned-dutch — 46,163 Dutch instruction/response pairs
• Framework: TRL 1.6.0 SFTTrainer
• Epochs: 3
• Steps: 4,329
• Loss: 3.31 → 1.14
• Duration: ~1.25 hours

Instruction format

markdown
Copy code
### Instructie:
<vraag of instructie>
### Antwoord:
<antwoord>

Usage

python
Copy code
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Thorstin/gpt2-dutch-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
model.eval()

def chat(instruction: str, max_new_tokens: int = 200) -> str:
    prompt = f"### Instructie:\n{instruction}\n### Antwoord:\n"
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.3,
            pad_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(output[0], skip_special_tokens=True)
    return response.split("### Antwoord:")[-1].strip()

print(chat("Wat is de hoofdstad van Nederland?"))

Benchmark results (lm-evaluation-harness, limit=200)

Random baseline: 50% for BLiMP-NL (binary), 25% for HellaSwag/ARC (4-way).

Sample outputs

Limitations

• 124M parameters is a hard ceiling — expect occasional repetition, factual errors, and shorter coherent responses compared to larger models
• Context window is limited to 512 tokens

Framework versions