Model Details
- Parameters: 102,558,240 (~102.6M)
- Architecture: Llama-style decoder-only Transformer
- Layers: 12
- Hidden size: 720
- Intermediate size: 1,920
- Attention heads: 12
- KV heads: 4
- Vocabulary size: 50,257
- Context length: 1,024 tokens
- Tokenizer: GPT-2 tokenizer
Pretraining
The model was initialized with newly created model weights
and pretrained on approximately 2 billion Indonesian-language
tokens.
Training hardware:
Dataset
The training corpus was prepared from
HuggingFace FineWeb-2,
using the Indonesian Latin (ind_Latn) subset.
Approximately:
- 2 billion training tokens
- 10 million validation tokens
The original dataset is not redistributed with this repository.
Users should refer to the upstream dataset documentation,
license, and terms.
Source Repository
Technical documentation and supporting resources:
https://github.com/novrian6/novai
Training
The model was pretrained using causal language modeling.
- Training tokens: ~2 billion
- Validation tokens: ~10 million
- Training steps: 15,259
Full pretraining logs and training source code are available in the project repository:
https://github.com/novrian6/novai
Important
This model is not a fine-tuned copy of Llama, GPT, Claude,
Gemini, or another pretrained foundation model.
The architecture uses the Llama implementation available
through Hugging Face Transformers, while the model weights
were newly initialized and pretrained separately for NovAI.
Training Evidence
The documented training run completed at:
15,259 / 15,259 steps
Training loss:
1.7229
Validation loss:
1.7012
Approximate training tokens:
2,000,027,648
Tokenizer
The model uses the GPT-2 tokenizer with a vocabulary size
of 50,257 tokens.
Tokenizer:
https://huggingface.co/openai-community/gpt2
Intended Use
NovAI Base 102M is intended for:
- Indonesian NLP research
- language-model experimentation
- downstream fine-tuning
- local inference experimentation
- educational and research purposes
Limitations
NovAI Base 102M is a relatively small language model and
should not be compared directly with substantially larger
commercial or open-weight foundation models.
The model may generate incorrect, repetitive, biased, or
factually unreliable content.
This repository contains the base pretrained model and
is not an instruction-tuned conversational model.
License
The NovAI Base 102M model weights are released under the
Apache License 2.0.
The training dataset, tokenizer, and third-party software
remain subject to their respective licenses and terms.
Attribution
Model: NovAI Base 102M
Developer: KaryaVirtual
Author: Nova Novriansyah, MSc, MBA
Website:
https://karyavirtual.com/