achille-fusco

gpt2_ParFindFast_eng_nl_zho

Model Details

Architecture

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Architecture	GPT-2 (`GPT2LMHeadModel`)
Hidden size (`n_embd`)	768
Layers (`n_layer`)	12
Attention heads (`n_head`)	12
Positional capacity (`n_positions`)	1024
Training window length	512
Dropout	0.1
Parameters	119,078,400
Exported vocabulary size	43,276

Tokenizer

The model uses a custom ParadigmFinder tokenizer exported as EnhancedParadigmTokenizerWrapper. In this multilingual configuration, tokenizer training uses a soft per-language vocabulary budget and boundary discovery support designed to remain compatible with space-free scripts such as Chinese.

Tokenizer highlights:

Boundary discovery enabled: true
Boundary discovery mode: space_free_only
Shorter-first sentence ordering: true
Explicit word-boundary insertion during tokenization: disabled (use_word_boundaries=false)
Vocabulary budget: 16,384 token slots per language
Languages: English (eng), Dutch (nld), Chinese (zho)

Special tokens:

<pad> = 0
<unk> = 1
<s> = 2
</s> = 3

This repo includes the tokenizer code and supporting files needed for loading:

tokenizer.py
boundary_discovery.py
preprocessing.py
paradigm_utils.py
paradigms.json
multilingual_meta.json

Training Data

This model was trained on the BabyLM multilingual data selection stored locally as multilingual_selection, with the three languages:

English (eng)
Dutch (nld)
Chinese (zho)

For comparability with the multilingual MoP baseline, the GPT-2 model training uses the same multilingual corpus selection. In this experiment line, tokenizer training and model training were aligned to the same multilingual data source.

Training Procedure

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Training type	`strict`
Epochs	10
Maximum training steps	200,000
Batch size	16
Learning rate	5e-5
Warmup steps	2,000
Weight decay	0
Gradient clipping	1.0

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM repo_id = "achille-fusco/gpt2_ParFindFast_eng_nl_zho" tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True) prompt_en = "The child looked at"inputs = tokenizer(prompt_en, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0])) prompt_nl = "Het kind keek naar"inputs = tokenizer(prompt_nl, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0])) prompt_zh = "孩子看着"inputs = tokenizer(prompt_zh, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0]))

If the environment has a read-only Hugging Face cache, set a writable cache first:

bash

export HF_HOME=/tmp/hf_cache

Evaluation

This model was evaluated with the BabyLM 2026 multilingual evaluation pipeline across zero-shot and fine-tuned tasks.

Summary scores:

Table with columns: Metric, Score
Metric	Score
Zero-shot mean	0.5746
Finetune mean	0.2613
Overall mean	0.3932

Selected zero-shot results:

Table with columns: Task, Score
Task	Score
`blimp`	0.6176
`blimp_nl`	0.6759
`multiblimp_eng`	0.7675
`multiblimp_nld`	0.7336
`zhoblimp`	0.9830
`xcomps_nl`	0.7916

These numbers correspond to the collated submission artifact produced for this export.

Limitations

This is a small multilingual BabyLM-scale model and is not intended as a production general-purpose LM.
The tokenizer relies on custom remote code, so some environments may require extra care when loading.
No explicit language identifier is used; mixed-language prompting may lead to unstable behavior.
Compared with the multilingual MoP baseline, this model is stronger on some Chinese and compositional zero-shot evaluations, but weaker overall after finetuning.

Citation

If you use this model, please cite the model repository and the BabyLM shared task.

bibtex

@misc{fusco2026parfindfast,  author       = {Fusco, Achille},  title        = {{BabyLM 2026 Multilingual GPT-2 (ParadigmFinder)}},  year         = {2026},  howpublished = {\url{https://huggingface.co/achille-fusco/gpt2_ParFindFast_eng_nl_zho}},  note         = {BabyLM 2026 multilingual model release with ParadigmFinder tokenizer.}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

achille-fusco

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Architecture

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Architecture	GPT-2 (`GPT2LMHeadModel`)
Hidden size (`n_embd`)	768
Layers (`n_layer`)	12
Attention heads (`n_head`)	12
Positional capacity (`n_positions`)	1024
Training window length	512
Dropout	0.1
Parameters	119,078,400
Exported vocabulary size	43,276

Tokenizer

Tokenizer highlights:

Boundary discovery enabled: true
Boundary discovery mode: space_free_only
Shorter-first sentence ordering: true
Explicit word-boundary insertion during tokenization: disabled (use_word_boundaries=false)
Vocabulary budget: 16,384 token slots per language
Languages: English (eng), Dutch (nld), Chinese (zho)

Special tokens:

<pad> = 0
<unk> = 1
<s> = 2
</s> = 3

This repo includes the tokenizer code and supporting files needed for loading:

tokenizer.py
boundary_discovery.py
preprocessing.py
paradigm_utils.py
paradigms.json
multilingual_meta.json

Training Data

This model was trained on the BabyLM multilingual data selection stored locally as multilingual_selection, with the three languages:

English (eng)
Dutch (nld)
Chinese (zho)

Training Procedure

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Training type	`strict`
Epochs	10
Maximum training steps	200,000
Batch size	16
Learning rate	5e-5
Warmup steps	2,000
Weight decay	0
Gradient clipping	1.0

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM repo_id = "achille-fusco/gpt2_ParFindFast_eng_nl_zho" tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True) prompt_en = "The child looked at"inputs = tokenizer(prompt_en, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0])) prompt_nl = "Het kind keek naar"inputs = tokenizer(prompt_nl, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0])) prompt_zh = "孩子看着"inputs = tokenizer(prompt_zh, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0]))

If the environment has a read-only Hugging Face cache, set a writable cache first:

bash

export HF_HOME=/tmp/hf_cache

Evaluation

This model was evaluated with the BabyLM 2026 multilingual evaluation pipeline across zero-shot and fine-tuned tasks.

Summary scores:

Table with columns: Metric, Score
Metric	Score
Zero-shot mean	0.5746
Finetune mean	0.2613
Overall mean	0.3932

Selected zero-shot results:

Table with columns: Task, Score
Task	Score
`blimp`	0.6176
`blimp_nl`	0.6759
`multiblimp_eng`	0.7675
`multiblimp_nld`	0.7336
`zhoblimp`	0.9830
`xcomps_nl`	0.7916

These numbers correspond to the collated submission artifact produced for this export.

Limitations

This is a small multilingual BabyLM-scale model and is not intended as a production general-purpose LM.
The tokenizer relies on custom remote code, so some environments may require extra care when loading.
No explicit language identifier is used; mixed-language prompting may lead to unstable behavior.
Compared with the multilingual MoP baseline, this model is stronger on some Chinese and compositional zero-shot evaluations, but weaker overall after finetuning.

Citation

If you use this model, please cite the model repository and the BabyLM shared task.

bibtex

@misc{fusco2026parfindfast,  author       = {Fusco, Achille},  title        = {{BabyLM 2026 Multilingual GPT-2 (ParadigmFinder)}},  year         = {2026},  howpublished = {\url{https://huggingface.co/achille-fusco/gpt2_ParFindFast_eng_nl_zho}},  note         = {BabyLM 2026 multilingual model release with ParadigmFinder tokenizer.}}

gpt2_ParFindFast_eng_nl_zho

README

Model Details

Architecture

Tokenizer

Training Data

Training Procedure

Usage

Evaluation

Limitations

Citation

Explore FriendliAI today

README

Model Details

Architecture

Tokenizer

Training Data

Training Procedure

Usage

Evaluation

Limitations

Citation