Arcoson

distilgpt2-arxiv-csai

Model Details

Table with columns: Property, Value
Property	Value
Base Model	distilgpt2
Parameters	81.9M total, 14.2M trainable (17.3%)
Fine-tuning Strategy	Partial freeze — last 2 transformer blocks + LM head
Training Data	ccdv/arxiv-classification (CS/AI subset)
Training Samples	2,000
Max Sequence Length	128 tokens
License	MIT

Training Details

Dataset

The model was fine-tuned on abstracts from the ccdv/arxiv-classification dataset, filtered for computer science and AI categories:

cs.CV (Computer Vision)
cs.AI (Artificial Intelligence)
cs.SY (Systems and Control)
cs.CE (Computational Engineering)
cs.PL (Programming Languages)
cs.IT (Information Theory)
cs.DS (Data Structures and Algorithms)
cs.NE (Neural and Evolutionary Computing)

Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Learning rate	3e-4
Batch size	8
Max steps	100
Warmup steps	10
Scheduler	Cosine
Weight decay	0.01
Frozen layers	Embedding + blocks 0-3
Trainable layers	Blocks 4-5 + LayerNorm + LM Head

Metrics

Table with columns: Metric, Value
Metric	Value
Training loss	3.82
Eval loss	3.48
Perplexity	32.59
Training time	~7 minutes (CPU)

Usage

python

from transformers import pipeline generator = pipeline("text-generation", model="Arcoson/distilgpt2-arxiv-csai") output = generator(    "arXiv Abstract: We propose a novel approach to",    max_new_tokens=100,    do_sample=True,    temperature=0.8,    top_p=0.9,)print(output[0]["generated_text"])

Or load directly:

python

from transformers import AutoModelForCausalLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("Arcoson/distilgpt2-arxiv-csai")model = AutoModelForCausalLM.from_pretrained("Arcoson/distilgpt2-arxiv-csai") inputs = tokenizer("arXiv Abstract: In this paper, we study the problem of", return_tensors="pt")outputs = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sample Generations

Prompt: "arXiv Abstract: We propose a novel approach to"

Fine-tuned output:

We propose a novel approach to the problem of time-time in the computation model, i.e., and prediction models for (i) continuous intervals which are nonlinear because they cannot be solved with finite random variables or large number functions...

Base DistilGPT-2 output:

We propose a novel approach to the question of whether we can solve this problem by using real-time neural networks in an MRI technique...

The fine-tuned model produces more domain-appropriate language with CS/AI terminology compared to the base model.

Limitations

Short context: Trained on 128-token sequences, so it works best for generating short abstracts or completions.
Limited training: Only 100 steps on 2,000 samples — this is a proof-of-concept fine-tune. More training data and steps would improve quality.
Domain narrow: Outputs are biased toward CS/AI academic language. Not suitable for general-purpose text generation.
No factual accuracy: The model generates plausible-sounding but not necessarily factually correct research text.

Training Infrastructure

Hardware: CPU (2 vCPUs, 8GB RAM)
Framework: HuggingFace Transformers 5.5.0, PyTorch 2.11
Strategy: Partial layer freezing to enable CPU-feasible training

Citation

If you use this model, please cite the base model and dataset:

bibtex

@article{sanh2019distilbert,  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},  journal={arXiv preprint arXiv:1910.01108},  year={2019}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Arcoson

Model Tree

Base

distilbert/distilgpt2

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Table with columns: Property, Value
Property	Value
Base Model	distilgpt2
Parameters	81.9M total, 14.2M trainable (17.3%)
Fine-tuning Strategy	Partial freeze — last 2 transformer blocks + LM head
Training Data	ccdv/arxiv-classification (CS/AI subset)
Training Samples	2,000
Max Sequence Length	128 tokens
License	MIT

Training Details

Dataset

The model was fine-tuned on abstracts from the ccdv/arxiv-classification dataset, filtered for computer science and AI categories:

cs.CV (Computer Vision)
cs.AI (Artificial Intelligence)
cs.SY (Systems and Control)
cs.CE (Computational Engineering)
cs.PL (Programming Languages)
cs.IT (Information Theory)
cs.DS (Data Structures and Algorithms)
cs.NE (Neural and Evolutionary Computing)

Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Learning rate	3e-4
Batch size	8
Max steps	100
Warmup steps	10
Scheduler	Cosine
Weight decay	0.01
Frozen layers	Embedding + blocks 0-3
Trainable layers	Blocks 4-5 + LayerNorm + LM Head

Metrics

Table with columns: Metric, Value
Metric	Value
Training loss	3.82
Eval loss	3.48
Perplexity	32.59
Training time	~7 minutes (CPU)

Usage

python

from transformers import pipeline generator = pipeline("text-generation", model="Arcoson/distilgpt2-arxiv-csai") output = generator(    "arXiv Abstract: We propose a novel approach to",    max_new_tokens=100,    do_sample=True,    temperature=0.8,    top_p=0.9,)print(output[0]["generated_text"])

from transformers import pipeline generator = pipeline("text-generation", model="Arcoson/distilgpt2-arxiv-csai") output = generator(    "arXiv Abstract: We propose a novel approach to",    max_new_tokens=100,    do_sample=True,    temperature=0.8,    top_p=0.9,)print(output[0]["generated_text"])

Or load directly:

python

from transformers import AutoModelForCausalLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("Arcoson/distilgpt2-arxiv-csai")model = AutoModelForCausalLM.from_pretrained("Arcoson/distilgpt2-arxiv-csai") inputs = tokenizer("arXiv Abstract: In this paper, we study the problem of", return_tensors="pt")outputs = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

from transformers import AutoModelForCausalLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("Arcoson/distilgpt2-arxiv-csai")model = AutoModelForCausalLM.from_pretrained("Arcoson/distilgpt2-arxiv-csai") inputs = tokenizer("arXiv Abstract: In this paper, we study the problem of", return_tensors="pt")outputs = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sample Generations

Prompt: "arXiv Abstract: We propose a novel approach to"

Fine-tuned output:

We propose a novel approach to the problem of time-time in the computation model, i.e., and prediction models for (i) continuous intervals which are nonlinear because they cannot be solved with finite random variables or large number functions...

Base DistilGPT-2 output:

We propose a novel approach to the question of whether we can solve this problem by using real-time neural networks in an MRI technique...

The fine-tuned model produces more domain-appropriate language with CS/AI terminology compared to the base model.

Limitations

Short context: Trained on 128-token sequences, so it works best for generating short abstracts or completions.
Limited training: Only 100 steps on 2,000 samples — this is a proof-of-concept fine-tune. More training data and steps would improve quality.
Domain narrow: Outputs are biased toward CS/AI academic language. Not suitable for general-purpose text generation.
No factual accuracy: The model generates plausible-sounding but not necessarily factually correct research text.

Training Infrastructure

Hardware: CPU (2 vCPUs, 8GB RAM)
Framework: HuggingFace Transformers 5.5.0, PyTorch 2.11
Strategy: Partial layer freezing to enable CPU-feasible training

Citation

If you use this model, please cite the base model and dataset:

bibtex

@article{sanh2019distilbert,  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},  journal={arXiv preprint arXiv:1910.01108},  year={2019}}

@article{sanh2019distilbert,  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},  journal={arXiv preprint arXiv:1910.01108},  year={2019}}

distilgpt2-arxiv-csai

README

Model Details

Training Details

Dataset

Hyperparameters

Metrics

Usage

Sample Generations

Limitations

Training Infrastructure

Citation

Explore FriendliAI today

README

Model Details

Training Details

Dataset

Hyperparameters

Metrics

Usage

Sample Generations

Limitations

Training Infrastructure

Citation