jbomdev

AlterEgo

README

License: apache-2.0

Introduction

AlterEgo is a small, decoder-only language model built from the ground up - not a fine-tune of an existing model. Every part was written from zero: the transformer architecture, the training loop, the tokenizer wiring, and the KV-cached inference engine. It was pre-trained on ~10B tokens of high-quality educational web text and then instruction-tuned for chat.

It is the model at the heart of LLME, a self-hosted, end-to-end-encrypted LLM platform (think LM Studio / Open WebUI / Ollama, also built from scratch). LLME can serve AlterEgo alongside llama.cpp GGUF models and the Gemini API; AlterEgo is the "house" model it was designed around.

This repository contains the model. The training and architecture code lives in the AlterEgo repo; the serving platform lives in the LLME repo.

Two formats are published. This repo is the Hugging Face LlamaForCausalLM conversion, for drop-in use with transformers, vLLM, and GGUF tooling. The original checkpoint - in AlterEgo's own from-scratch architecture, exactly as trained - is published separately as jbomdev/alterego_raw. This version is a numerically-lossless conversion of it (verified: max logit difference ~1e-6).

What it is and isn't. AlterEgo is a research / learning artifact - a demonstration of the full modern LLM pipeline (architecture → pretraining → SFT → serving) at a scale one person can train on a single GPU. It is not a production assistant and won't compete with billion-parameter models. See Limitations.

Architecture

A modern Llama-style decoder (and, thanks to that, it loads as a standard LlamaForCausalLM).

Table with columns: Component, Choice
Component	Choice
Type	Decoder-only transformer (autoregressive)
Parameters	~373M (input/output embeddings tied)
Layers	24
Model dimension	1024
Attention	Grouped-Query Attention - 16 query heads / 4 KV heads (head dim 64)
Positional encoding	Rotary embeddings (RoPE), θ = 10,000
Normalization	RMSNorm (pre-norm)
Feed-forward	, hidden dim 2816

Training

AlterEgo was trained in two stages on a single NVIDIA RTX 4090.

Stage 1 - Pretraining

Pre-trained on FineWeb-Edu (HuggingFaceFW), a quality-filtered educational subset of CommonCrawl.

Pretraining loss

Training dynamics

The grad-norm settling to ~0.26 and the smooth cosine-shaped loss indicate stable training with no divergence.

Stage 2 - Supervised fine-tuning

Instruction-tuned on UltraChat-200K (HuggingFaceH4), formatted as multi-turn ChatML.

SFT loss

Hyperparameters

Table with columns: Pretraining, SFT
	Pretraining	SFT
Dataset	FineWeb-Edu	UltraChat-200K
Tokens / steps	~10B / 19,073	~64M / 244
Global batch	524,288 tokens (micro 2 × 2048 × 128 grad-accum)	same scheme
Optimizer	AdamW (β = 0.9, 0.95; ε = 1e-8; fused)	same
Weight decay	0.1 (decoupled; excluded from norms/biases)	same
LR schedule	linear warmup (1,900 steps) → cosine decay

Evaluation

Benchmarked with EleutherAI's lm-evaluation-harness (0-shot).

Table with columns: Benchmark, Metric, AlterEgo-373M, Random
Benchmark	Metric	AlterEgo-373M	Random
lambada_openai	acc	31.6%	~0%
hellaswag	acc_norm	38.0%	25%
arc_easy	acc_norm	52.7%	25%
arc_challenge	acc_norm	27.3%	25%
piqa

For a 373M model trained on ~10B tokens these are solid: clearly above chance on science and commonsense (SciQ, PIQA, BoolQ, ARC-easy, HellaSwag) and on next-word prediction (LAMBADA — perplexity 62.3), with the expected near-chance results on the hardest reasoning sets (ARC-challenge, WinoGrande).

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizerimport torch tok = AutoTokenizer.from_pretrained("jbomdev/AlterEgo")model = AutoModelForCausalLM.from_pretrained("jbomdev/AlterEgo", torch_dtype=torch.bfloat16) messages = [    {"role": "system", "content":     "You are Alter Ego, a small AI built from scratch. You're casual and direct. "     "You're not great with facts, math, or current events - when you don't know "     "something, just say so. You're better at chatting than at answering questions."},    {"role": "user", "content": "Tell me something interesting about the ocean."},]ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") out = model.generate(    ids,    max_new_tokens=200,    do_sample=True,    temperature=0.7,    top_k=50,    top_p=1.0,    repetition_penalty=1.1,)print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Recommended generation settings

These are the defaults AlterEgo was tuned and served with in LLME:

Table with columns: Parameter, Value
Parameter	Value
`temperature`	0.7
`top_k`	50
`top_p`	1.0
`repetition_penalty`	1.1
`max_new_tokens`	200

Lower the temperature toward 0.3–0.5 for steadier, more focused replies; it stops on <|im_end|> or <|endoftext|>.

Chat format

AlterEgo uses ChatML:

text

<|im_start|>system{system prompt}<|im_end|><|im_start|>user{message}<|im_end|><|im_start|>assistant

Run it locally (GGUF)

Feel free to use my pre-made GGUF's and quants by visiting The GGUF's and quants page. Or running the model with ollama.

Also, Because it's standard Llama format, you can convert to GGUF for Ollama / LM Studio / llama.cpp yourself:

bash

python llama.cpp/convert_hf_to_gguf.py ./AlterEgo --outfile alterego-f16.gguf --outtype f16

Limitations

AlterEgo is a 373M-parameter model trained on a modest token budget, and it behaves like one:

Capability - it can be factually wrong, repeat itself, and lose coherence on long or complex prompts. By its own (default) system prompt, it is "better at chatting than at answering questions."
Language - English only.
Safety - it is not safety- or preference-tuned (no RLHF/DPO). It can produce incorrect, biased, or undesirable content and must not be deployed in user-facing settings without additional safeguards.
Bias - it inherits biases from FineWeb-Edu (web text) and UltraChat.

License

Released under the Apache 2.0 license. Training data is governed by the respective licenses of FineWeb-Edu and UltraChat-200K.

Citation

bibtex

@misc{alterego2026,  title  = {AlterEgo: A 373M language model trained from scratch},  author = {J-bom},  year   = {2026},  url    = {https://github.com/J-bom/AlterEgo}}

Credits - datasets: FineWeb-Edu (HuggingFaceFW), UltraChat-200K (HuggingFaceH4). Architecture follows the modern Llama-style design (RoPE, GQA, SwiGLU, RMSNorm); implementation, training, and serving by the author.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.