LumiOpen

Poro-34B

README

License: apache-2.0

Model Overview

NOTE: In addition to being an early research release, Poro is a base model which needs further fine tuning for most use cases.

Poro is a generative pretrained transformer using a BLOOM architecture, and makes use of ALiBi embeddings to support context length extrapolation at inference time.

Table with columns: Hyperparameter, Value
Hyperparameter	Value
n_parameters	34.2B
n_layers	54
n_heads	56
d_model	7168
vocab_size	128000
sequence_length	2048

Poro Research Checkpoints

Checkpoints are available as branches in the repository. Checkpoints will be released roughly every 100B tokens. The main branch will always point to the latest checkpoint. The following checkpoints are available:

The transformers library allows you to load a checkpoint from a branch as follows:

python
branch = "200B"
model = transformers.AutoModelForCausalLM.from_pretrained(
    "LumiOpen/Poro-34B",
    torch_dtype=torch.bfloat16,
    revision=branch,
)

Training

Poro was trained on the LUMI supercomputer, using 512 AMD MI250X GPUs. Each MI250X GPU has two Graphics Complex Dies (GCDs) for a world size of 1024 during training, using activation checkpointing, a micro batch size of 1, gradient accumulation of 16, and a 3D parallelism strategy of TP=2, PP=4, DP=128.

Training began in September 2023 using a custom fork of the Megatron-Deepspeed framework. Our code is available here.

Training Hyperparameters

Table with columns: Hyperparameter, Value, Comment
Hyperparameter	Value	Comment
Precision	bfloat16
Optimizer	AdamW
Learning rate	1.5e-4	10B tokens warm-up, cosine decay to 2e-5
Weight decay	1e-1
Batch size	2048	2048 samples x 2048 tokens = 4194304 tokens

Tokenizer

Poro uses a custom 128K Bloom tokenizer trained on the same English, Finnish and Code dataset used to train the model.

Dataset

Poro is being trained on a 1 trillion token mixed dataset of English, Finnish and Code.

Table with columns: Dataset, Notes, Percentage, Epochs, Tokens
Dataset	Notes	Percentage	Epochs	Tokens
SlimPajama	Excluding books3 data	54.16%	1x	541.7B
Finnish	TurkuNLP Finnish dataset	13.05%	4x	131.5B
Tatoeba	English/Finnish sentence pairs	0.81%	1x	8.0B

The Finnish dataset is a combination of many Finnish resources:

Evaluation Results

Full evaluations for each checkpoint are available on our Github repo.

Ethical Considerations and Limitations

Poro is an advanced language model, primarily optimized for English, Finnish and code, with no meaningful proficiency in any other languages. As with most AI-driven systems, Poro is a product of the vast data it has been trained on, which may reflect the imperfections, biases, and idiosyncrasies of the wider web. Poro may, at times, produce outputs that can be considered inaccurate, prejudiced, or controversial. Users and developers engaging with Poro should exercise discretion and consider additional evaluation and customization to ensure the model's responses align with their specific needs and ethical standards.

License

Poro is released under the Apache 2.0 license.

Citation

markdown
@misc{luukkonen2024poro,
      title={Poro 34B and the Blessing of Multilinguality}, 
      author={Risto Luukkonen and Jonathan Burdge and Elaine Zosa and Aarne
Talman and Ville Komulainen and Väinö Hatanpää and Peter Sarlin and Sampo
Pyysalo},
      year={2024},
      eprint={2404.01856},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider