Gpeik

Sophira-1B-base

README

License: apache-2.0

Model Summary

Model name: Sophira-1B-base
Intended Hugging Face repository: Gpeik/Sophira-1B-base
Internal frozen release candidate: Sophira-1B-base-v0.1
Architecture: decoder-only Llama-like Transformer
Nominal size: 1B
Language: Only Italian
Canonical checkpoint: iter_1243085
Tokenizer: Gpeik/Sophira-tokenizer-64k-v0
License target: Apache-2.0

This release is the first completed 1B Italian base model from the Sophira project. It is intended as an open research and benchmarking artifact, and as a starting point for later downstream adaptation work.

Intended Use

This model is the first completed 1B base-pretraining artifact for the Sophira project.

Intended uses:

Italian language modeling research
downstream evaluation and benchmarking
initialization for later instruction tuning or task adaptation
reproducibility work around open Italian foundation-model pretraining

Out-of-Scope Use

This artifact is not documented or evaluated as suitable for:

safety-critical production deployment
legal, medical, or financial decision support
factual-reliability-sensitive assistant use without downstream evaluation
multilingual production use outside Italian-first evaluation

Training Data

Canonical training sources:

uonlp/CulturaX Italian subset
PleIAs/Italian-PD

Full-pass mixture used for this run:

82.977% CulturaX Italian
17.023% Italian-PD

Measured source-token counts:

CulturaX_it: 135.197.223.323
Italian_PD: 27.736.341.018
Total: 162.933.564.341

The project license policy and source-license references remain tracked in DATA_LICENSES.md.

Training Procedure

Training framework: Megatron-LM
Runtime: .venv-apex
Cluster: CINECA Leonardo. We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support.
Topology: 16 nodes / 64 GPUs
Sequence length: 2048
Micro-batch size: 1
Global batch size: 64
Tokens per step: 131072
Target steps: 1243085
Checkpoint interval:

Final successful completion occurred on Tuesday, August 11, 2026.

Evaluation

End-of-training validation:

iteration: 1243085
validation loss: 2.756790E+00
validation perplexity: 1.574921E+01

Export validation:

Hugging Face export completed successfully
native vs Hugging Face next-token parity passed
parity prompt:
- Roma, 19 luglio - Il vertice europeo sull'energia si è aperto oggi
parity result:
- greedy next token match restored
- top-20 overlap restored to 20 / 20

Controlled-generation summary:

prompts: 10
generations: 30
average output tokens: 65.7667
average output words: 58.3667
average repeated bigram fraction: 0.1017
average repeated trigram fraction: 0.0755
average topic keyword overlap: 0.1386
empty generation count: 0
malformed output count: 0

Controlled-generation analysis summary:

severity counts:
- mixed: 23
- good: 6
- problematic: 1
dominant issue types:
- topic_drift: 20
- repeated_clause: 6
- high_trigram_repetition: 4
- repeated_percentage_pattern: 1

The main residual weakness in this release-facing prompt set is topic drift under sampled decoding, with one repetition-heavy greedy continuation on the prompt Il governo ha annunciato che.

Comparison against the released 360M baseline:

repetition is lower in 1B than in 360M
- bigram repetition:
  - 1B: 0.1017
  - 360M: 0.1148
- trigram repetition:
  - 1B: 0.0755
  - 360M: 0.0809
malformed outputs on the current prompt suite:
- 1B: 0

Release-facing takeaway:

1B is the stronger overall base-model release candidate on the current controlled-generation evidence
the clearest gains over 360M are lower repetition and the absence of malformed outputs on this prompt suite
the current evidence does not justify a strong claim that topical adherence is solved

Usage

Transformers example:

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Gpeik/Sophira-1B-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Il governo ha annunciato che"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Snapshot

Completed:

end-of-training Megatron validation
checkpoint save/resume validation across multiple Slurm jobs
final Hugging Face export
native vs Hugging Face first-token parity validation
controlled-generation evaluation
controlled-generation analysis
comparison against the published 360M baseline
final checkpoint pruning to the retained release set

Areas for Improvement

This is a base model, not an instruction-tuned model.
Broader robustness, bias, and safety evaluation is still pending.
This release should be treated as an open Italian base-model baseline, not as a polished assistant model.

Release Artifacts

The canonical release checkpoint is:

iter_1243085

The canonical exported model bundle is:

.../models/releases/Sophira-1B-base-v0.1/model

License

Code: Apache-2.0
Tokenizer: Apache-2.0
Model weights target: Apache-2.0

Source dataset licenses remain governed by their upstream terms and by the repository tracking in DATA_LICENSES.md.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.