GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
aisingapore
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
aisingapore
Model Tree
Input Modalities
Output Modalities
Supported Functionality
The SEA-LION model is a significant leap forward in the field of Natural Language Processing, specifically trained to understand the SEA regional context.
SEA-LION-v1-3B is built on the robust MPT architecture and has a vocabulary size of 256K.
For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.
The training data for SEA-LION-v1-3B encompasses 980B tokens.
SEA-LION-v1-3B has an average performance on general tasks in English (as measured by Hugging Face's LLM Leaderboard):
| Model | ARC | HellaSwag | MMLU | TruthfulQA | Average |
|---|---|---|---|---|---|
| SEA-LION 3B | 36.26 | 64.59 | 24.07 | 36.46 | 40.35 |
SEA-LION-v1-3B was trained on 980B tokens of the following data:
| Data Source | Unique Tokens | Multiplier | Total Tokens | Percentage |
|---|---|---|---|---|
| RefinedWeb - English | 571.3B | 1 | 571.3B | 58.20% |
| mC4 - Chinese | 91.2B | 1 | 91.2B | 9.29% |
| mC4 - Indonesian | 3.68B | 4 | 14.7B | 1.50% |
SEA-LION-v1-3B was trained using MosaicML Composer on the following hardware:
| Training Details | SEA-LION-v1-3B |
|---|---|
| AWS EC2 p4d.24xlarge | 30 instances |
| Nvidia A100 40GB GPU | 240 |
| Training Duration | 14 days |
| HyperParameter | SEA-LION-v1-3B |
|---|---|
| Precision | bfloat16 |
| Optimizer | decoupled_adamw |
| Scheduler | cosine_with_warmup |
| Learning Rate | 1.6e-4 |
| Global Batch Size | 1200 |
| Micro Batch Size | 5 |
SEA-LION-v1-3B is a decoder model using the MPT architecture.
| Parameter | SEA-LION-v1-3B |
|---|---|
| Layers | 32 |
| d_model | 2560 |
| head_dim | 20 |
| Vocabulary | 256000 |
| Sequence Length | 2048 |
We sample 20M lines from the training data to train the tokenizer. The framework for training is SentencePiece. The tokenizer type is Byte-Pair Encoding (BPE).
Lam Wen Zhi Clarence Leong Wei Qi Li Yier Liu Bing Jie Darius Lovenia Holy Montalan Jann Railey Ng Boon Cheong Raymond Ngui Jian Gang Nguyen Thanh Ngan Ong Tat-Wee David Rengarajan Hamsawardhini Susanto Yosephine Tai Ngee Chia Tan Choon Meng Teo Jin Howe Teo Eng Sipp Leslie Teo Wei Yi Tjhi William Yeo Yeow Tong Yong Xianbin
AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
For more info, please contact us using this SEA-LION Inquiry Form
Link to SEA-LION's GitHub repository
This the repository for the base model. The model has not been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights and codes.
bibtex
@misc{lowphansirikul2021wangchanberta, title={WangchanBERTa: Pretraining transformer-based Thai Language Models}, author={Lalita Lowphansirikul and Charin Polpanumas and Nawat Jantrakulchai and Sarana Nutanong}, year={2021}, eprint={2101.09635}, archivePrefix={arXiv}, primaryClass={cs.CL}}| mC4 - Malay | 0.72B | 4 | 2.9B | 0.29% |
| mC4 - Filipino | 1.32B | 4 | 5.3B | 0.54% |
| mC4 - Burmese | 1.2B | 4 | 4.9B | 0.49% |
| mC4 - Vietnamese | 63.4B | 1 | 63.4B | 6.46% |
| mC4 - Thai | 5.8B | 2 | 11.6B | 1.18% |
| WangChanBERTa - Thai | 5B | 2 | 10B | 1.02% |
| mC4 - Lao | 0.27B | 4 | 1.1B | 0.12% |
| mC4 - Khmer | 0.97B | 4 | 3.9B | 0.40% |
| mC4 - Tamil | 2.55B | 4 | 10.2B | 1.04% |
| the Stack - Python | 20.9B | 2 | 41.8B | 4.26% |
| the Stack - Javascript | 55.6B | 1 | 55.6B | 5.66% |
| the Stack - Shell | 1.2B5 | 2 | 2.5B | 0.26% |
| the Stack - SQL | 6.4B | 2 | 12.8B | 1.31% |
| the Stack - Markdown | 26.6B | 1 | 26.6B | 2.71% |
| RedPajama - StackExchange | 21.2B | 1 | 21.2B | 2.16% |
| RedPajama - ArXiv | 30.6B | 1 | 30.6B | 3.12% |
@misc{lowphansirikul2021wangchanberta, title={WangchanBERTa: Pretraining transformer-based Thai Language Models}, author={Lalita Lowphansirikul and Charin Polpanumas and Nawat Jantrakulchai and Sarana Nutanong}, year={2021}, eprint={2101.09635}, archivePrefix={arXiv}, primaryClass={cs.CL}}