pdjamez

Duloch-Pico-125M-Base

README

License: apache-2.0

Model details

Table with columns: Property, Value
Property	Value
Architecture	Decoder-only causal Transformer
Transformers implementation	`LlamaForCausalLM`
Parameters	125,095,680
Layers	18
Hidden width	768
Attention heads	12
Key-value heads	2
Head dimension	64
Feed-forward width	2,048
Context length	512 tokens
Vocabulary	20,000 tokens
Activation	SwiGLU
Normalisation	RMSNorm, epsilon 1e-5
Position encoding	RoPE, theta 10,000
Embeddings	Tied input and output embeddings

Duloch Pico was trained from scratch and contains no Meta Llama weights. It is packaged using the standard Transformers Llama implementation because its decoder architecture, RMSNorm, RoPE, SwiGLU, and grouped-query attention are directly compatible. No custom code or trust_remote_code setting is required.

Intended use

The model is intended for research, education, small-model experiments, base-model evaluation, and further training. It may also be useful for testing resource-constrained inference and Apple Silicon workflows.

The model is not suitable as a factual authority, a safety-critical system, or an unsupervised user-facing assistant. Additional evaluation and appropriate safety controls are required before any deployment.

Training

The model was trained with next-token prediction using a deterministic subset of HuggingFaceFW/fineweb, specifically the sample-10BT configuration.

The local preparation selected 3,600,000 documents, balanced across 15 source Parquet files, then applied exact duplicate removal and trigram SimHash near duplicate filtering. The accepted training split contained 3,560,666 documents. Training stopped within the first epoch after 2,501,935,104 tokens, which is approximately 20 tokens per learned parameter.

Table with columns: Training property, Value
Training property	Value
Objective	Causal next-token prediction
Training tokens	2,501,935,104
Sequence length	512
Global batch	32,768 tokens
Optimizer steps	76,353
Peak learning rate	3e-4
Final learning rate	3e-5
Warmup steps	764
Gradient clipping	1.0

The final scheduled held-out validation loss was 3.1502 at 2,500,001,792 tokens, equivalent to approximately 23.34 perplexity. Each scheduled validation used a fixed maximum of 64 windows rather than the entire validation corpus.

Training data attribution and terms

FineWeb was created and published by Hugging Face from Common Crawl material. The source dataset is available at HuggingFaceFW/fineweb under the Open Data Commons Attribution License, ODC-By 1.0. Use of FineWeb is also subject to the Common Crawl Terms of Use.

FineWeb is derived from public web crawls. It can contain copyrighted text, personal information, factual errors, unsafe material, bias, malformed pages, and residual duplicates despite upstream and local filtering. FineWeb reports that email addresses and public IP addresses were anonymised, while also warning that other personally identifiable information is likely to remain. The Apache-2.0 licence for these model weights does not grant rights to third-party training content.

Evaluation

The following results use the EleutherAI Language Model Evaluation Harness at commit b954108c9baaaa934b4ad842033b31a97ee30816. Evaluation was zero-shot, float32, batch size 32, and limited to the model's 512-token context. No BOS token was added. The full benchmark splits were used.

Table with columns: Benchmark, Metric, Score, Standard error, Samples
Benchmark	Metric	Score	Standard error	Samples
PIQA	Normalised accuracy	62.84%	1.13%	1,838
HellaSwag	Normalised accuracy	30.80%	0.46%	10,042
ARC-Easy	Normalised accuracy	37.58%	0.99%	2,376

These scores describe a small base-model evaluation suite. They do not establish broad capability, factual reliability, instruction following, or safety. The aggregate machine-readable results and evaluation provenance are included in evaluation/summary.json.

Usage with Transformers

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pdjamez/Duloch-Pico-125M-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=False,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=False,
    dtype=torch.float32,
)

prompt = "The weather in Scotland is"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Keep the prompt and generated continuation within the supported 512-token context. This base model has no chat template.

The direct loading and generation path above has been tested with Transformers 4.57.1 and 5.14.1. Transformers 5.14.1 can omit the display space between the prompt and continuation when its pipeline() convenience API separately decodes and then joins SentencePiece text. The generated token IDs and direct full-sequence decoding remain correct. The example above avoids that rendering issue.

The canonical repository uses standard Transformers files. Conversion to MLX BF16 has been validated with mlx-lm 0.31.3, but the MLX package is planned as a separate derivative repository.

The canonical Transformers model was also compared directly with the native Swift/MLX checkpoint using all 20,000 final-position logits at deterministic input lengths 1, 2, 17, 511 and 512. Every probe passed the declared bfloat16 cross-framework tolerance, and the top-five token set matched in every case. Native MLX and Transformers produced identical 64-token greedy generations, including Transformers generation with its key-value cache. They also produced the same next token from a 511-token prompt at the supported 512-token boundary.

One artificial one-token probe reversed two near-tied top candidates within the declared numerical tolerance; the top-five set remained identical. This means the validation establishes tolerance-aware numerical and operational parity, not bit-exact equality of every bfloat16 calculation. A separate-process reload after saving the FP32 Transformers model passed the same gate and produced byte-identical Safetensors weights.

Limitations

The 512-token context is short by current standards.
The model can generate false, contradictory, biased, offensive, unsafe, or private-looking content.
It has limited factual recall, arithmetic, reasoning, and long-form coherence.
Training was English-focused, although the web corpus contains some other languages.
The full selected corpus received automated screening, but it was not exhaustively inspected by people. Memorisation has not been ruled out.
The model has not been instruction tuned, preference aligned, red teamed, or safety tuned.
Benchmark results can be affected by prompt format, tokenizer behaviour, harness version, precision, and context truncation.

Please use the repository's Community tab to report reproducibility problems, suspected memorised personal information, or correction and removal requests. Requests concerning records in the original FineWeb dataset should also use the removal route linked from the FineWeb dataset card. Selective deletion from released model weights may not be technically possible. A confirmed issue may instead require withdrawing or replacing an affected release.

Licence

The model weights and accompanying release files are provided under the Apache License 2.0. See LICENSE for the full terms. Training data remains subject to its original source terms and any applicable rights.

Acknowledgements

This model was trained using FineWeb data published by Hugging Face and was evaluated using the EleutherAI Language Model Evaluation Harness.

References

Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, NeurIPS 2024 Datasets and Benchmarks Track.
FineWeb dataset card.
Common Crawl Terms of Use.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

pdjamez

Model Tree

Base

this model

Input Modalities

Text

Output Modalities