num1notsvn

wicara-56m-base

README

License: apache-2.0

Architecture Specifications

Table with columns: Component, Specification
Component	Specification
Total Parameters	56,027,520
Non-Embedding Parameters	45,541,120
Layers (`n_layer`)	10
Hidden Dimension (`d_model`)	640
Attention Mechanism	Grouped-Query Attention (10 query heads / 5 KV heads, GQA 2:1)
Feed-Forward Dimension (`d_ffn`)	1,728 (SwiGLU)
Head Dimension (`head_dim`)	64
Context Length	512 tokens
Vocabulary Size	16,384 byte-level BPE (4.26 chars/token on Indonesian text)
Positional Embedding	RoPE ( $θ = 10000.0$ )
Normalization	Pre-norm RMSNorm ( $ϵ = 1 0 - 5$ )
Embedding Tying	Yes (`lm_head.weight == tok_emb.weight`)

Pretraining Details

Dataset Composition

Trained on 1.12 billion tokens (3,985,535 documents) curated exclusively from natural Indonesian texts:

Table with columns: Source, Role, Tokens, Share, Filter Pass Rate
Source	Role	Tokens	Share	Filter Pass Rate
OpenSubtitles v2024	Conversational dialogue	484M	43.2%	93.9%
FineWeb-2 `ind_Latn`	Informal web writing	210M	18.7%	99.4%

Training Progression & Hardware

Hardware: Single laptop with NVIDIA GeForce RTX 4050 Laptop GPU (6 GB VRAM).
Duration: 11.1 hours (17,010 steps, 1 epoch, 100% corpus coverage).
Throughput: 28,017 tokens/second.
Precision: bf16 mixed precision with AdamW ( $β 1 = 0.9, β 2 = 0.95$ , cosine decay with warmup).
Validation Loss: 3.0505 (Perplexity: 21.2).

Quickstart & Usage

[!NOTE] As a base model, wicara-56m-base is trained for next-token prediction (text completion), not multi-turn instruction following. If you are looking for an interactive conversational assistant, please use num1notsvn/wicara-56m-chat.

Text Completion Example

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "num1notsvn/wicara-56m-base"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
).to(device)

# Provide a prefix prompt for completion
prompt = "Indonesia adalah sebuah negara kepulauan yang"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate continuation
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    do_sample=True,
    eos_token_id=tokenizer.eos_token_id,
)

continuation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(continuation)

Intended Use & Fine-Tuning

This base model is intended as a foundation for:

Downstream Fine-Tuning: Supervised fine-tuning (SFT) for Indonesian NLP tasks (sentiment analysis, intent classification, named entity recognition, or custom domain adaptation).
Edge SLM Research: Exploring the boundaries of lightweight language models (<100M parameters) under strict compute and VRAM budgets.
Academic & Educational Use: Studying transformer pretraining dynamics from scratch.

Limitations

No Instruction Tuning: This model does not understand conversational turns or system prompts out of the box.
Hallucination & Factual Consistency: Small-capacity models (56M parameters) prioritize surface-level linguistic fluency and cannot reliably serve as factual knowledge repositories without external retrieval (RAG).
Context Length: The native maximum context length is 512 tokens.

Citation & License

Released under the Apache-2.0 License.

bibtex
@misc{ardin2026wicara,
  title={Wicara: A Weight-efficient Indonesian Conversational Architecture Built from Scratch},
  author={Bagus Ardin Prayoga},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/num1notsvn/wicara-56m-base}}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

num1notsvn

Model Tree

Base

this model

Input Modalities

Text

Output Modalities