Gpeik

Sophira-360M-base

Deploy Dedicated

README

License: apache-2.0

Model Summary

Model name: Sophira-360M-base
Intended Hugging Face repository: Gpeik/Sophira-360M-base
Internal frozen release candidate: Sophira-360M-base-v0.1
Architecture: decoder-only Llama-like Transformer
Parameter count: 376.075.200
Language: Only Italian
Canonical checkpoint: iter_1.104.964
Tokenizer: Gpeik/Sophira-tokenizer-64k-v0
License target: Apache-2.0

Intended Use

This model is the first published base-pretraining artifact for the Sophira project.

Intended uses:

Italian language modeling research
downstream evaluation and benchmarking
initialization for later instruction tuning or task adaptation
reproducibility work around open Italian foundation-model pretraining

Out-of-Scope Use

This artifact is not yet documented or evaluated as suitable for:

safety-critical production deployment
legal, medical, or financial decision support
factual-reliability-sensitive assistant use without downstream evaluation
multilingual production use outside Italian-first evaluation

Training Data

Canonical training sources:

uonlp/CulturaX Italian subset
PleIAs/Italian-PD

Full-pass mixture used for this run:

82.977% CulturaX Italian
17.023% Italian-PD

Measured source-token counts:

CulturaX_it: 135.197.223.323
Italian_PD: 27.736.341.018
Total: 162.933.564.341

The project license policy and source-license references remain tracked in DATA_LICENSES.md.

Training Procedure

Training Framework: Megatron-LM
Runtime: .venv-apex
Cluster: CINECA Leonardo. We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support.
Topology: 3 nodes / 12 GPUs
Sequence length: 2048
Micro-batch size: 3
Global batch size: 72
Tokens per step: 147.456
Target steps: 1.104.964
Checkpoint interval:

Final successful completion occurred on Sunday, July 26, 2026.

Evaluation

End-of-training validation:

iteration: 1.104.964
validation loss: 2.910531E+00
validation perplexity: 1.836655E+01

This is the validated Megatron end-of-training metric for the completed full-pass run.

Release validation and benchmark highlights:

Hugging Face controlled-generation summary:
- empty_generation_rate: 0.0
- avg_repeated_bigram_fraction: 0.1148
- avg_repeated_trigram_fraction: 0.0809
- avg_topic_keyword_overlap: 0.23
stronger benchmark groups (BLiMP-IT):
- verbal_class_and_argument_structure: 0.9500
- pronouns: 0.7188
- agreement_and_inflection: 0.7454 with appended EOS

Usage

Transformers example:

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Gpeik/Sophira-360M-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Il governo ha annunciato che"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Pipeline example:

python
from transformers import pipeline
import torch

generator = pipeline(
    "text-generation",
    model="Gpeik/Sophira-360M-base",
    tokenizer="Gpeik/Sophira-360M-base",
    model_kwargs={"dtype": torch.bfloat16},
    device_map="auto",
)

result = generator(
    "Milano, 19 luglio - La Borsa europea ha chiuso",
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(result[0]["generated_text"])

Evaluation Snapshot

Completed:

end-of-training Megatron validation
checkpoint save/resume validation across multiple Slurm jobs
quota-recovery and checkpoint-pruning procedure
Hugging Face export
native vs Hugging Face first-token parity validation
corrected Hugging Face controlled-generation rerun
corrected controlled-generation analysis rerun
BLiMP-IT benchmark
BLiMP-IT appended-EOS benchmark

Areas for Improvement

This is a base model, not an instruction-tuned model.
Controlled generation still shows visible topic drift and clause repetition on some prompts.
BLiMP-IT performance is only modestly above chance overall, especially on non-local dependency phenomena.
Broader robustness, bias, and safety evaluation is still pending.
This release should be treated as an open Italian base-model baseline, not as a polished assistant model.

Release Artifacts

The canonical release candidate checkpoint is:

iter_1104964

The published Hugging Face repository is intended to contain:

the frozen Hugging Face model weights
the tokenizer files needed for same-repo loading
this model card as the public README.md
the release image asset used for repository presentation

License

Code: Apache-2.0
Tokenizer: Apache-2.0
Model weights target: Apache-2.0

Source dataset licenses remain governed by their upstream terms and by the repository tracking in DATA_LICENSES.md.

Citation

If you use this model in academic work, please cite it as follows:

bibtex
@misc{peikos2026sophira360mbase,
  title        = {Sophira-360M-base},
  author       = {Peikos, Georgios},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Gpeik/Sophira-360M-base}},
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.