neosantara

wader-100m

README

License: apache-2.0

Architecture

This model implements key DeepSeek-V4 innovations at a miniature scale:

Table with columns: Component, Details
Component	Details
Parameters	~110M total (41M embeddings, 69M non-embedding)
Hidden size	320
Layers	8
Attention heads	8 (1 KV head — MQA-style)
MLA	Multi-head Latent Attention with q_lora_rank=160
MoE	4 routed experts + 1 shared, top-2 routing
Hyper-Connections	hc_mult=4, Sinkhorn routing (replacing residual connections)
MTP	1 next-token prediction layer
Vocab	129,280 (DeepSeek-V4 tokenizer)
Context	2,048 tokens

Training

Stage 1: Pretraining

Dataset: neosantara/IndoTalk (396k dialogs)
Steps: 5,000 | Tokens: ~327M
Batch: 32 effective | Seq length: 2,048
Hardware: 1× NVIDIA RTX 5090 Blackwell

SFT Training Metrics

Table with columns: Metric, Value
Metric	Value
Final Train Loss	~2.60 (Cross-Entropy at Step 3000)
Token Accuracy	~51.71% (Predicting assistant responses)

Stage 2: SFT (this model)

Dataset: neosantara/IndoTalkSFT (~160K conversations & instructions)
Steps: 3,000 | Tokens: ~96M
Batch: 16 effective | Seq length: 2,048
Hardware: 1× NVIDIA RTX 5090 Blackwell

SFT Training Metrics

Table with columns: Metric, Value
Metric	Value
Final Train Loss	~2.60 (Cross-Entropy at Step 3000)
Token Accuracy	~51.71% (Predicting assistant responses)

Usage

python
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

# IMPORTANT: This model requires a modern GPU (Ampere/Ada/Hopper) that supports bfloat16.
# Running on CPU or older GPUs (like T4) will result in NaN generation errors.
device = "cuda" if torch.cuda.is_available() else "cpu"

repo_id = "neosantara/wader-100m"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, 
    trust_remote_code=True, 
    torch_dtype=torch.bfloat16
).to(device)

# Chat
messages = [{"role": "user", "content": "Halo, apa kabar?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    input_ids, 
    max_new_tokens=200, 
    temperature=0.7, 
    top_p=0.9, 
    pad_token_id=tokenizer.eos_token_id,
    do_sample=True
)
print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))

Limitations

Tiny model: 110M params with 129K vocabulary — most capacity goes to embeddings. Generations are often incoherent or factually wrong.
Undertrained: Only 5K pretrain + 3K SFT steps. Production models train for 100K+ steps on trillions of tokens.
bf16 NaN: Use fp32 — the Hyper-Connections architecture produces values that overflow bf16 range at this scale.
Custom code: Requires trust_remote_code=True.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

neosantara

Model Tree

Base

neosantara/wader-100m-base

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Architecture

This model implements key DeepSeek-V4 innovations at a miniature scale:

Table with columns: Component, Details
Component	Details
Parameters	~110M total (41M embeddings, 69M non-embedding)
Hidden size	320
Layers	8
Attention heads	8 (1 KV head — MQA-style)
MLA	Multi-head Latent Attention with q_lora_rank=160
MoE	4 routed experts + 1 shared, top-2 routing
Hyper-Connections	hc_mult=4, Sinkhorn routing (replacing residual connections)
MTP	1 next-token prediction layer
Vocab	129,280 (DeepSeek-V4 tokenizer)
Context	2,048 tokens

Training

Stage 1: Pretraining

Dataset: neosantara/IndoTalk (396k dialogs)
Steps: 5,000 | Tokens: ~327M
Batch: 32 effective | Seq length: 2,048
Hardware: 1× NVIDIA RTX 5090 Blackwell

SFT Training Metrics

Table with columns: Metric, Value
Metric	Value
Final Train Loss	~2.60 (Cross-Entropy at Step 3000)
Token Accuracy	~51.71% (Predicting assistant responses)

Stage 2: SFT (this model)

Dataset: neosantara/IndoTalkSFT (~160K conversations & instructions)
Steps: 3,000 | Tokens: ~96M
Batch: 16 effective | Seq length: 2,048
Hardware: 1× NVIDIA RTX 5090 Blackwell

SFT Training Metrics

Table with columns: Metric, Value
Metric	Value
Final Train Loss	~2.60 (Cross-Entropy at Step 3000)
Token Accuracy	~51.71% (Predicting assistant responses)

Usage

python
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

# IMPORTANT: This model requires a modern GPU (Ampere/Ada/Hopper) that supports bfloat16.
# Running on CPU or older GPUs (like T4) will result in NaN generation errors.
device = "cuda" if torch.cuda.is_available() else "cpu"

repo_id = "neosantara/wader-100m"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, 
    trust_remote_code=True, 
    torch_dtype=torch.bfloat16
).to(device)

# Chat
messages = [{"role": "user", "content": "Halo, apa kabar?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    input_ids, 
    max_new_tokens=200, 
    temperature=0.7, 
    top_p=0.9, 
    pad_token_id=tokenizer.eos_token_id,
    do_sample=True
)
print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))

Limitations

Tiny model: 110M params with 129K vocabulary — most capacity goes to embeddings. Generations are often incoherent or factually wrong.
Undertrained: Only 5K pretrain + 3K SFT steps. Production models train for 100K+ steps on trillions of tokens.
bf16 NaN: Use fp32 — the Hyper-Connections architecture produces values that overflow bf16 range at this scale.
Custom code: Requires trust_remote_code=True.