num1notsvn

wicara-56m-chat

README

License: apache-2.0

Architecture Specifications

Table with columns: Component, Specification
Component	Specification
Total Parameters	56,027,520
Non-Embedding Parameters	45,541,120
Layers (`n_layer`)	10
Hidden Dimension (`d_model`)	640
Attention Mechanism	Grouped-Query Attention (10 query heads / 5 KV heads, GQA 2:1)
Feed-Forward Dimension (`d_ffn`)	1,728 (SwiGLU)
Head Dimension (`head_dim`)	64
Context Length	512 tokens
Vocabulary Size	16,384 byte-level BPE (4.26 chars/token on Indonesian text)
Positional Embedding	RoPE ( $θ = 10000.0$ )
Normalization	Pre-norm RMSNorm ( $ϵ = 1 0 - 5$ )
Embedding Tying	Yes (`lm_head.weight == tok_emb.weight`)

Training Details

Pretraining (`wicara-56m-base`)

Data: 1.12 billion tokens (3,985,535 documents) curated exclusively from Indonesian texts:
- OpenSubtitles v2024 (43.2%)
- FineWeb-2 ind_Latn (18.7%)
- Indonesian Wikipedia (15.5%)
- Cendol v2 (11.5%)
- Aya Collection (10.8%)
- TED2020 (0.3%)
Compute: Single RTX 4050 6 GB laptop (~11.1 hours, 28,017 tokens/sec).
Validation Loss: 3.0505 (Perplexity: 21.2).

Supervised Fine-Tuning (SFT v5 - `wicara-56m-chat`)

Base Checkpoint: Initialized directly from wicara-56m-base (step 17,010).
Data: ~40,000 conversational and instruction examples mined from Aya Collection, Cendol v2, and subtitles, augmented with 591 curated handwritten samples to anchor persona ("Wicara") and constrain hallucinations.
Loss Masking: Masked cross-entropy loss computed strictly over assistant turn tokens (user and system prompts are unmasked).
Safety: Built-in deterministic guardrails for crisis prompts and special-token input sanitization (src/infer/pencegat.py).

Quickstart & Usage

You can run Wicara using the Hugging Face transformers library directly:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "num1notsvn/wicara-56m-chat"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).to(device)

# Prepare conversation
messages = [
    {"role": "system", "content": "Kamu adalah asisten AI berbahasa Indonesia yang ramah."},
    {"role": "user", "content": "Halo! Siapa namamu dan kamu bisa apa?"},
]

# Apply chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate response
outputs = model.generate(
    **inputs,
    max_new_tokens=160,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    do_sample=True,
    eos_token_id=[tokenizer.eos_token_id, 2],
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Limitations & Intended Use

Resource-Constrained Research: Wicara 56M is designed to demonstrate efficient small-language-model architecture for Indonesian dialogue running locally on edge and consumer devices.
Factual Knowledge: Due to its compact size (56M parameters), the model should not be used as an authoritative factual reference without external verification or RAG (Retrieval-Augmented Generation).
Context Length: The context window is 512 tokens.

Citation & License

Released under the Apache-2.0 License.

bibtex
@misc{ardin2026wicara,
  title={Wicara: A Weight-efficient Indonesian Conversational Architecture Built from Scratch},
  author={Bagus Ardin Prayoga},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/num1notsvn/wicara-56m-chat}}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

num1notsvn

Model Tree

Base

num1notsvn/wicara-56m-base

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Architecture Specifications

Table with columns: Component, Specification
Component	Specification
Total Parameters	56,027,520
Non-Embedding Parameters	45,541,120
Layers (`n_layer`)	10
Hidden Dimension (`d_model`)	640
Attention Mechanism	Grouped-Query Attention (10 query heads / 5 KV heads, GQA 2:1)
Feed-Forward Dimension (`d_ffn`)	1,728 (SwiGLU)
Head Dimension (`head_dim`)	64
Context Length	512 tokens
Vocabulary Size	16,384 byte-level BPE (4.26 chars/token on Indonesian text)
Positional Embedding	RoPE ( $θ = 10000.0$ )
Normalization	Pre-norm RMSNorm ( $ϵ = 1 0 - 5$ )
Embedding Tying	Yes (`lm_head.weight == tok_emb.weight`)

Training Details

Pretraining (`wicara-56m-base`)

Data: 1.12 billion tokens (3,985,535 documents) curated exclusively from Indonesian texts:
- OpenSubtitles v2024 (43.2%)
- FineWeb-2 ind_Latn (18.7%)
- Indonesian Wikipedia (15.5%)
- Cendol v2 (11.5%)
- Aya Collection (10.8%)
- TED2020 (0.3%)
Compute: Single RTX 4050 6 GB laptop (~11.1 hours, 28,017 tokens/sec).
Validation Loss: 3.0505 (Perplexity: 21.2).

Supervised Fine-Tuning (SFT v5 - `wicara-56m-chat`)

Base Checkpoint: Initialized directly from wicara-56m-base (step 17,010).
Data: ~40,000 conversational and instruction examples mined from Aya Collection, Cendol v2, and subtitles, augmented with 591 curated handwritten samples to anchor persona ("Wicara") and constrain hallucinations.
Loss Masking: Masked cross-entropy loss computed strictly over assistant turn tokens (user and system prompts are unmasked).
Safety: Built-in deterministic guardrails for crisis prompts and special-token input sanitization (src/infer/pencegat.py).

Quickstart & Usage

You can run Wicara using the Hugging Face transformers library directly:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "num1notsvn/wicara-56m-chat"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).to(device)

# Prepare conversation
messages = [
    {"role": "system", "content": "Kamu adalah asisten AI berbahasa Indonesia yang ramah."},
    {"role": "user", "content": "Halo! Siapa namamu dan kamu bisa apa?"},
]

# Apply chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate response
outputs = model.generate(
    **inputs,
    max_new_tokens=160,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    do_sample=True,
    eos_token_id=[tokenizer.eos_token_id, 2],
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Limitations & Intended Use

Resource-Constrained Research: Wicara 56M is designed to demonstrate efficient small-language-model architecture for Indonesian dialogue running locally on edge and consumer devices.
Factual Knowledge: Due to its compact size (56M parameters), the model should not be used as an authoritative factual reference without external verification or RAG (Retrieval-Augmented Generation).
Context Length: The context window is 512 tokens.

Citation & License

Released under the Apache-2.0 License.

bibtex
@misc{ardin2026wicara,
  title={Wicara: A Weight-efficient Indonesian Conversational Architecture Built from Scratch},
  author={Bagus Ardin Prayoga},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/num1notsvn/wicara-56m-chat}}
}

wicara-56m-chat

README

Architecture Specifications

Training Details

Pretraining (wicara-56m-base)

Supervised Fine-Tuning (SFT v5 - wicara-56m-chat)

Quickstart & Usage

Limitations & Intended Use

Citation & License

Explore FriendliAI today

README

Architecture Specifications

Training Details

Pretraining (wicara-56m-base)

Supervised Fine-Tuning (SFT v5 - wicara-56m-chat)

Quickstart & Usage

Limitations & Intended Use

Citation & License

Pretraining (`wicara-56m-base`)

Supervised Fine-Tuning (SFT v5 - `wicara-56m-chat`)

Pretraining (`wicara-56m-base`)

Supervised Fine-Tuning (SFT v5 - `wicara-56m-chat`)