alephn-1 API & Inference Endpoint

Table

Architecture	GPT-2-style decoder, dense attention in every block, sequential (non-parallel) blocks
Parameters	~111M (embeddings tied with the output head)
Layers / hidden / heads	10 / 768 / 12 (head dim 64)
FFN size	3072, exact GELU
Positions	Learned absolute, context length 2048
Vocabulary	GPT-2 BPE, 50,257 tokens
Dropout	None
Weights format	`safetensors`, fp32

Table

Data	KnowSpread
Tokens	~2.2B (about 20 tokens per parameter, compute-optimal)
Batch size	~246K tokens (120 sequences x 2048)
Optimizer	AdamW, betas (0.9, 0.95), eps 1e-8
Weight decay	0.1 (2D weight matrices only; biases and LayerNorm excluded)
Peak LR	6e-4, linear warmup over 375M tokens, then linear decay to 10% of peak
Gradient clipping	1.0 (global norm)
Precision

Zero-shot accuracy using EleutherAI's lm-evaluation-harness.

Alephn 1 scores higher than Cerebras-GPT 111M on 5 of the 7 tasks reported in the Cerebras-GPT paper.

This is a small base model trained on about 2.2B tokens. Expect fluent-looking but frequently incoherent or factually wrong text.
Scores on commonsense and reasoning benchmarks are near chance level, as is typical at this scale.
It has had no safety training or alignment, and it can produce biased, offensive, or inappropriate text reflecting its training data.
Intended for research, education, and experimentation (for example as a starting point for fine-tuning). It is not suitable for production use or for any decision-making.

Our Article on Alephn 1: Introducing Alephn 1

Table

Architecture	GPT-2-style decoder, dense attention in every block, sequential (non-parallel) blocks
Parameters	~111M (embeddings tied with the output head)
Layers / hidden / heads	10 / 768 / 12 (head dim 64)
FFN size	3072, exact GELU
Positions	Learned absolute, context length 2048
Vocabulary	GPT-2 BPE, 50,257 tokens
Dropout	None
Weights format	`safetensors`, fp32

Table

Data	KnowSpread
Tokens	~2.2B (about 20 tokens per parameter, compute-optimal)
Batch size	~246K tokens (120 sequences x 2048)
Optimizer	AdamW, betas (0.9, 0.95), eps 1e-8
Weight decay	0.1 (2D weight matrices only; biases and LayerNorm excluded)
Peak LR	6e-4, linear warmup over 375M tokens, then linear decay to 10% of peak
Gradient clipping	1.0 (global norm)
Precision

Zero-shot accuracy using EleutherAI's lm-evaluation-harness.

Alephn 1 scores higher than Cerebras-GPT 111M on 5 of the 7 tasks reported in the Cerebras-GPT paper.

This is a small base model trained on about 2.2B tokens. Expect fluent-looking but frequently incoherent or factually wrong text.
Scores on commonsense and reasoning benchmarks are near chance level, as is typical at this scale.
It has had no safety training or alignment, and it can produce biased, offensive, or inappropriate text reflecting its training data.
Intended for research, education, and experimentation (for example as a starting point for fine-tuning). It is not suitable for production use or for any decision-making.

Our Article on Alephn 1: Introducing Alephn 1

alephn-1