TevunahAi

chochmah-350m

README

License: apache-2.0

Model Details

Table with columns: Property, Value
Property	Value
Developer	TevunahAi
Model Type	Decoder-only transformer, Llama architecture, pretrained from scratch
Parameters	358.7M total (308.3M non-embedding)
Training Tokens	29.99B (57,200 steps x 524,288 tokens)
Context Length	2,048 tokens
Tokenizer	SmolLM2 (49,152 BPE)
Training Precision	bf16 autocast, fp32 master weights
Release Precision	fp32 safetensors (1.43 GB)
Training Hardware	1x NVIDIA RTX 5000 Ada 32 GB
Training Time	~10.5 days wall-clock (August 12–22, 2026)
Training Compute	~6.5 x 10^19 FLOPs
Stage	Base — pretrained only
Loads With	`AutoModelForCausalLM` / `LlamaForCausalLM`, no custom code
License	Apache-2.0

Architecture Breakdown

A standard modern decoder, with no exotic choices — every design decision here is the 2024-era default so the model loads everywhere Llama loads.

Layer Composition (24 layers)

24 Transformer Blocks, pre-norm residual
- Multi-Head Attention: 16 heads, head dimension 64, no GQA (16 KV heads), no biases
- SwiGLU MLP: intermediate size 2,816, down(silu(gate(x)) * up(x)), no biases
- RMSNorm before attention and before the MLP, eps 1e-5
Rotary position embeddings (RoPE), theta 10,000, applied to Q and K
Tied input/output embeddings: the 49,152 x 1,024 embedding matrix is also the LM head
No dropout, no bias terms anywhere, init std 0.02

Parameter Accounting

Table with columns: Component, Parameters
Component	Parameters
Attention (24 x 4 x 1024^2)	100.7M
SwiGLU MLP (24 x 3 x 1024 x 2816)	207.6M
Norms	~0.05M
Non-embedding total	308.3M
Tied embedding (49,152 x 1024)	50.3M
Total	358.7M

Why This Matters

Non-embedding count drives compute. 308M is why this run took ten days on one GPU rather than a month.
Two-thirds of the parameters are in the MLPs. That is where the model's factual and stylistic knowledge lives, which matters for quantization planning.
Llama-compatible weights. Conversion from the training implementation is a pure key rename (no weight permutation); logit equivalence between the original PyTorch model and LlamaForCausalLM was verified to ~1e-6 in fp32. Everything downstream — vLLM, GGUF, GPTQModel — works out of the box.

Training

Optimizer and Schedule

Table with columns: Field, Value
Field	Value
Optimizer	AdamW, betas (0.9, 0.95), eps 1e-8, fused
Weight Decay	0.1 (none on norms or embeddings)
Peak Learning Rate	4e-4
Schedule	WSD (warmup–stable–decay): 2,000-step linear warmup → constant → decay to ~0 across the final 5,700 steps
Gradient Clipping	1.0
Global Batch	524,288 tokens (8 micro-batch x 32 grad-accum x 2,048)
Total Steps	57,200

The learning-rate decay is aligned to the data curriculum on purpose: the LR stays flat while data quality ramps through phases 1 and 2, then decays to zero exactly across the anneal phase. That alignment is what lets the highest-quality 10% of the data set the model's final voice instead of being averaged away by later updates.

Loss trajectory. 10.98 at initialization → 2.24 entering the anneal (step 51,500) → ~1.75 at the end of training (measured on the anneal mix; the phases use different data, so losses are not directly comparable across boundaries). Gradient norm held at ~0.15 through the entire decay with no instability. The run survived one unplanned reboot by resuming from checkpoint with no discontinuity in the loss curve, and both phase transitions occurred automatically without a restart.

Three-Phase Curriculum

Table with columns: Phase, Steps, Tokens, Purpose
Phase	Steps	Tokens	Purpose
1 — Foundation	0 – 34,300	18B	Broad language competence. Deliberately unclever.
2 — Domain build	34,300 – 51,500	9B	The six target domains enter at full weight.
3 — Anneal	51,500 – 57,200	3B	Highest-quality material only, LR decaying to zero. Sets the model's voice.

Each phase was pre-mixed into shards at fixed ratios (no runtime sampling), with a consumption manifest tracking per-source byte offsets and head hashes so that the three phases read disjoint ranges of every source. Phase 1 holds every source under one epoch (0.91). Only the small hand-built corpora repeat in later phases: philosophy 2.0 epochs, textbooks 3.4 epochs; FineWeb-Edu wraps slightly in phase 3. Separate per-domain validation sets were carved from the unused tail of each source.

Phase 1 — Foundation (18B tokens)

Table with columns: Source, Share
Source	Share
FineWeb-Edu	62%
StarCoderData	14%
Wikipedia (en)	8%
Project Gutenberg	7%
OpenWebMath	5%
Cosmopedia v2	4%

Phase 2 — Domain build (9B tokens)

Table with columns: Source, Share
Source	Share
FineWeb-Edu	28%
Math (FineMath-4+, OpenWebMath, AlgebraicStack)	20%
Law (hand-built, broad)	18.8%
StarCoderData	15%
peS2o (science)	8%
Cosmopedia v2	6%
Philosophy (hand-built)	4%
Textbooks (OpenStax)	0.2%

Phase 3 — Anneal (3B tokens, LR → 0)

Table with columns: Source, Share
Source	Share
Cosmopedia v2	37%
FineMath-4+	18%
Philosophy (hand-built)	12%
StarCoderData	12%
Law canon (hand-built, most-cited opinions)	10%
FineWeb-Edu	10%
Textbooks (OpenStax)	1%

Data Sources

Table with columns: Source, Origin, License, Role
Source	Origin	License	Role
FineWeb-Edu	`HuggingFaceFW/fineweb-edu` (sample-100BT)	ODC-By	Foundation, all phases
StarCoderData	`bigcode/starcoderdata`	Permissive-licensed code	Code, all phases
Wikipedia	`wikimedia/wikipedia` (20231101.en)	CC BY-SA 4.0	Phase 1

The Hand-Built Corpora

Law — 2,556,649 opinions (~9.92B tokens) broad, 101,688 opinions (~0.80B tokens) canon. Built from the 2026-06-30 CourtListener bulk export. Courts were selected by jurisdiction level, not by name (New York's "Supreme Court" is a trial court): the Supreme Court of the United States, the 13 modern federal circuits, and state courts of last resort; state intermediate appellate courts and the pre-1911 circuit trial courts were excluded. Opinions that were too short (1.6M) or carried West headnote markers were rejected. The canon split — every opinion cited 160 or more times, plus all SCOTUS opinions — is what the anneal phase trained on; it is 13% SCOTUS by token count and 73% from the 1980s–2010s. The broad corpus spans 66 courts with 42% of tokens pre-1980. Supply exceeded demand by roughly an order of magnitude: training consumed 0.17 epochs of the broad corpus and 0.37 epochs of the canon, so the law the model saw is a random cross-section, not a skim of the beginning.

Philosophy — 3,072 works (~0.353B tokens), 126 curated authors across seven eras. Ancient through early twentieth century, deliberately broadened to political theory and intellectual history, including Eastern texts, Douglass, Du Bois, Spooner, and the public-domain women's canon (Wollstonecraft, Fuller, Stanton, Harriet Taylor Mill, Anna Julia Cooper, Gilman, Goldman, Addams, de Gouges, Martineau). Nietzsche is present in full (22 works, the complete Levy edition), alongside Voltaire (23 works), Marx, Aristotle, Kant, Machiavelli, Confucius; the largest contributors by volume are William James, Hume, Henry George, T. H. Green, and Augustine. Matched by author, then Library of Congress classification, then subject; Gutenberg license boilerplate stripped; duplicate editions collapsed to the longest. Consumed at 2.0 epochs across phases 2 and 3.

Textbooks — 19 OpenStax STEM and government titles (~14M tokens). Fetched as CNXML from the openstax/osbooks-* source repositories, stripped of MathML, media, and metadata to clean prose. Openly licensed real textbooks are scarce (~100–300M tokens exist in total), so this corpus supplies authenticity while Cosmopedia supplies textbook-style volume. Consumed at 3.4 epochs.

Tokenizer

HuggingFaceTB/SmolLM2-360M — 49,152 BPE with digit splitting and code-friendly merges, chosen for a LaTeX- and code-heavy mix. Documents were concatenated with <|endoftext|> as the separator. No BOS token was prepended during training — do not add one at inference.

Performance Benchmarks

Standard Benchmarks

0-shot, lm-evaluation-harness, fp32, batch size 32. All three models were evaluated with the identical command on the same machine so the columns are directly comparable. Pythia-410M is the closest compute-matched reference (~302M non-embedding parameters vs. 308M here) and saw 10x the tokens; SmolLM2-360M shares this model's tokenizer and saw 133x the tokens.

Table with columns: Task, Chochmah-350M (30B tokens), Pythia-410M (300B tokens), SmolLM2-360M (4T tokens)
Task	Chochmah-350M (30B tokens)	Pythia-410M (300B tokens)	SmolLM2-360M (4T tokens)
ARC-Easy (acc / acc_norm)	59.0 / 54.2	51.9 / 45.7	70.4 / 68.0
HellaSwag (acc / acc_norm)	34.8 / 41.8	33.7 / 40.6	43.2 / 56.3
PIQA (acc / acc_norm)	68.6 / 68.3	66.7 / 67.2	72.1 / 72.0
LAMBADA-OpenAI (acc / ppl ↓)	40.2 / 23.8

Standard errors: ARC-Easy ±1.0, HellaSwag ±0.5, PIQA ±1.1, LAMBADA ±0.7.

How to read it. With a tenth of Pythia-410M's tokens, Chochmah is clearly ahead on ARC-Easy (+7 points, ~5 standard errors) — the signature of the education-weighted mix (FineWeb-Edu, Cosmopedia, OpenStax), the same effect the SmolLM family demonstrated. HellaSwag and PIQA are on par with Pythia-410M within error. LAMBADA, which tests final-word prediction on modern novel excerpts, trails Pythia by ~11 points: the mix contains ~1.3B tokens of public-domain books in phase 1 and essentially no fiction afterward, a deliberate trade for the target domains. The SmolLM2 column shows what 133x the tokens buys on the same kind of data — most of it on HellaSwag, which scales with token count almost on rails.

Per-Domain Held-Out Loss

Validation loss (nats/token) at the final step on held-out sets carved from the unused tail of each source:

Table with columns: Domain, Loss
Domain	Loss
FineWeb-Edu	—
StarCoderData	—
Wikipedia	—
Gutenberg	—
OpenWebMath	—
Cosmopedia	—
FineMath	—
Law	—
Philosophy	—

Sample Outputs

Verbatim, unedited, from the first prompts run on the final checkpoint (temperature 0.7, top-k 50, top-p 0.95, repetition penalty 1.1, 200 new tokens). The first two are shown together deliberately: they are the same model, one prompt apart, and together they are an accurate description of what a 350M base model is.

Prompt: The Supreme Court of the United States has long held that

The Supreme Court of the United States has long held that the Due Process Clause of the Fourteenth Amendment requires that a State provide an opportunity for an impartial adjudication before deprivation occurs. See, e. g., Mullane v. Central Hanover Bank & Trust Co., 339 U. S. 306 (1950); Goldberg v. Kelly, 397 U. S. 254 (1970). This Court has never, however, required the government to provide a hearing where the rights guaranteed by the Constitution are not implicated. In the absence of an express constitutional command, the Government is free to place all or some of those rights in any "general rule" which, when applied to particular cases, would be constitutionally permissible. This general rule is one which we have approved. The Court today adopts a new standard under which the Due Process Clause must be satisfied at the time of the deprivation. The Court finds that a hearing must be provided if the State fails to afford an opportunity

Both citations are real, correctly reported, and actually support the proposition — Mullane and Goldberg are the two cases for pre-deprivation process. Note the Court's own typographic conventions ("U. S.", "See, e. g.,"), learned from the SCOTUS opinions in the canon. Also note the drift: the final paragraph opens in the voice of a dissent.

Prompt: In Marbury v. Madison, Chief Justice Marshall reasoned that

In Marbury v. Madison, Chief Justice Marshall reasoned that the right of a citizen to be tried by his peers, rather than by a jury of his peers, would not serve as a check on the judiciary. The Chief Justice further stated that in order to prevent judicial oppression, the Constitution could not require a trial by jury unless it was provided by statute, but he did not believe that this provision was necessary because the judiciary was so numerous and well endowed that the process of trying an individual is within its jurisdiction. While this court had earlier recognized that Article III, Section 2 of the Constitution gave a right to a jury trial, Chief Justice Marshall concluded that the process of determining guilt or innocence was not constitutionally required. He stated that "the right to a trial by jury is of course limited by the law of the land, and that it does not extend to a trial by a jury of any particular number."

Every substantive claim in this sample is false, including the quotation. Marbury is about judicial review and original jurisdiction; the model followed the wrong clause of Article III § 2 and produced fluent, confident, well-formed legal prose about juries. A repeated citation pattern (Mullane + Goldberg) is memorized; a holding is a concept, and at this scale the model stores forms, not concepts. This is the failure mode to expect from every small base model, and it is why this model must not be used as a source of facts.

Prompt: Newton's second law states that

markdown
Newton's second law states that the net external force acting on an object is equal to its mass times its acceleration:
$$\text{net external force} = m \times a_f$$
where $m$ is the mass of the object, $a_f$ is the acceleration due to gravity ($9.81 \text{ m/s^2}$), and $a_f$ represents the acceleration of the object's center of mass relative to its center. By rearranging the equation above for the net external force, we can solve for the unknown variable:
$$\text{net external force} = m \times a_f = m \times g \times h$$
Now that we have solved for $g$, we can substitute it back into our original equation to find the object's acceleration:
$$a_f = \frac{GM}{r^2}$$
where $G$ is the gravitational constant ($6.67 \times 10^{

"Net external force" is OpenStax's exact phrasing. Every formula and constant is real and the LaTeX is well-formed; the derivation connecting them is not physics.

Prompt: Nietzsche's critique of morality begins with the observation that

[...] So why should middle school students care about Nietzschean thought? One reason is that many of his ideas continue to influence modern philosophy and culture. Nietzsche's emphasis on individual freedom and self-expression has been

The audience address is the fingerprint of Cosmopedia's synthetic textbook prose, which at 37% of the anneal is the model's default expository voice. Prompting in the register of a primary text rather than a summary about one shifts the output toward the philosophy canon's voice.

Usage

Transformers

python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "TevunahAi/Chochmah-350M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).cuda()

prompt = "The Supreme Court of the United States has long held that"
ids = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **ids,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
    repetition_penalty=1.15,
    no_repeat_ngram_size=4,
)
print(tok.decode(out[0], skip_special_tokens=True))

Prompting a Base Model

Continue, don't instruct. Give it the opening of the document you want; it will not follow a question or a command.
The prompt's register selects the corpus. A SCOTUS-style opening produces opinion prose with citations; a textbook sentence produces textbook prose; a def produces code. Write the first line the way the target document would be written.
No BOS token, no chat template. The tokenizer is saved without one; do not add either.
Sample tightly. Temperature 0.5–0.7, top-p 0.9, repetition penalty 1.1–1.15, no_repeat_ngram_size=4 (or min_p=0.05), and 100–150 new tokens. Base models do not know when to stop, and most drift lives in the second half of a long generation.

vLLM

bash
pip install vllm
vllm serve TevunahAi/Chochmah-350M --max-model-len 2048 --dtype bfloat16

Fine-Tuning

The model is a stock LlamaForCausalLM and works unmodified with TRL, Axolotl, LLaMA-Factory, Unsloth, and PEFT. Its main intended use is as an SFT / DPO base and as a small, clean, fully documented target for quantization research.

Known Issues and Limitations

It will state false things fluently. At this scale the model learns how legal, philosophical, mathematical, and scientific text is written far better than it learns what is true. It must not be used as a source of legal, medical, financial, or factual answers. It is not a substitute for legal advice, and nothing it writes about a case should be assumed to be in that case.
Base model. No instruction following, no safety tuning, no chat template, no stop behavior beyond <|endoftext|>.
Code is the weakest register. Indentation can break after a docstring and logic is unreliable; expect HumanEval near zero. This is expected at 350M / 30B tokens.
Formatting artifacts from the data. Two are visible in generations: Gutenberg-derived text is reproduced with its original hard line wraps, and case names sometimes appear on their own lines as in CourtListener's HTML-to-text conversion. A re-annealed revision on cleaned phase-3 data is planned and will be pushed to this repository as a new commit with a changelog.
Narrative fiction is underweighted, which is what the LAMBADA result reflects.
English only, 2,048-token context. Some Wikipedia-derived knowledge is dated 2023-11.
Repetition at long generation lengths, as with all small models; see the sampling guidance above.

Memory Requirements

Inference

fp32 (as released): 1.43 GB weights; ~2 GB VRAM at full 2,048 context
bf16: ~0.72 GB weights; ~1.2 GB VRAM at full context
Runs comfortably on CPU; a few hundred tokens per second on a modern GPU without a KV cache, far more with one

Training (Reproduction)

Used: RTX 5000 Ada (32 GB) + Dual Xeon Max 9480 (128 GB HBM2e + 256 GB DDR5) for the data pipeline
Peak VRAM: 26.4 GB at micro-batch 8 x 2,048 with torch.compile; micro-batch 16 does not fit in 32 GB
Tokenized data: ~60 GB (uint16 shards) — fits in page cache on 256 GB of RAM, so the GPU never waits on data
Raw downloads: ~500 GB, dominated by the 53 GB CourtListener bulk export and FineWeb-Edu

Technical Specifications

Table with columns: Specification, Value
Specification	Value
Model Family	TevunahAi Chochmah
Variant	350M (v1.0, base)
Total Parameters	358,663,168
Non-Embedding Parameters	308,331,520
Layers	24
Hidden Size	1,024
Attention Heads	16
KV Heads	16 (MHA)
Head Dimension	64

Roadmap

v1.1 — re-anneal of the stable-phase checkpoint on cleaned phase-3 data (hard wraps and citation line breaks removed, Cosmopedia share reduced), pushed to this repo as a new commit with a changelog
Chochmah-350M-Instruct — supervised fine-tuning on an open instruction set
TevunahAi Ultra-Hybrid GPTQ variant of the instruct model
Chochmah-1B — same pipeline and corpora, ~1B parameters, 40–50B tokens

License

Weights: Apache-2.0.

Training data is a mix of ODC-By (FineWeb-Edu, FineMath, OpenWebMath, peS2o), Apache-2.0 (Cosmopedia v2), CC BY 4.0 (OpenStax), CC BY-SA 4.0 (Wikipedia), permissively licensed code (StarCoderData), and public-domain material (Project Gutenberg; United States judicial opinions, which are edicts of government). No data under non-commercial terms was used. Users are responsible for their own compliance with the licenses of the underlying datasets.

Citation

bibtex
@misc{chochmah350m_2026,
  title  = {Chochmah-350M: a from-scratch pretrained base model on one GPU},
  author = {TevunahAi},
  year   = {2026},
  url    = {https://huggingface.co/TevunahAi/Chochmah-350M},
  note   = {359M parameters, 30B tokens, three-phase WSD curriculum with hand-built case-law and philosophy corpora}
}

Acknowledgments

Hugging Face / SmolLM team for the SmolLM2 tokenizer, FineWeb-Edu, FineMath, and Cosmopedia — and for the SmolLM reports that made the education-weighted recipe a known quantity
Free Law Project for the CourtListener bulk data that made the law corpus possible
Project Gutenberg and the sedthh/gutenberg_english packaging
OpenStax for publishing textbook sources under CC BY
BigCode (StarCoderData), Allen AI (peS2o), Wikimedia
EleutherAI for lm-evaluation-harness, Proof-Pile-2, and the Pythia suite used as the compute-matched reference

Pretrained by TevunahAi AI Model Quantization and On-Premises AI Infrastructure From-scratch models, hybrid-architecture quantization (Mamba, MoE, SSM), and enterprise hardware bring-up

https://huggingface.co/TevunahAi

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

TevunahAi

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table with columns: Property, Value
Property	Value
Developer	TevunahAi
Model Type	Decoder-only transformer, Llama architecture, pretrained from scratch
Parameters	358.7M total (308.3M non-embedding)
Training Tokens	29.99B (57,200 steps x 524,288 tokens)
Context Length	2,048 tokens
Tokenizer	SmolLM2 (49,152 BPE)
Training Precision	bf16 autocast, fp32 master weights
Release Precision	fp32 safetensors (1.43 GB)
Training Hardware	1x NVIDIA RTX 5000 Ada 32 GB
Training Time	~10.5 days wall-clock (August 12–22, 2026)
Training Compute	~6.5 x 10^19 FLOPs
Stage	Base — pretrained only
Loads With	`AutoModelForCausalLM` / `LlamaForCausalLM`, no custom code
License	Apache-2.0

Architecture Breakdown

A standard modern decoder, with no exotic choices — every design decision here is the 2024-era default so the model loads everywhere Llama loads.

Layer Composition (24 layers)

24 Transformer Blocks, pre-norm residual
- Multi-Head Attention: 16 heads, head dimension 64, no GQA (16 KV heads), no biases
- SwiGLU MLP: intermediate size 2,816, down(silu(gate(x)) * up(x)), no biases
- RMSNorm before attention and before the MLP, eps 1e-5
Rotary position embeddings (RoPE), theta 10,000, applied to Q and K
Tied input/output embeddings: the 49,152 x 1,024 embedding matrix is also the LM head
No dropout, no bias terms anywhere, init std 0.02

Parameter Accounting

Table with columns: Component, Parameters
Component	Parameters
Attention (24 x 4 x 1024^2)	100.7M
SwiGLU MLP (24 x 3 x 1024 x 2816)	207.6M
Norms	~0.05M
Non-embedding total	308.3M
Tied embedding (49,152 x 1024)	50.3M
Total	358.7M

Why This Matters

Non-embedding count drives compute. 308M is why this run took ten days on one GPU rather than a month.
Two-thirds of the parameters are in the MLPs. That is where the model's factual and stylistic knowledge lives, which matters for quantization planning.
Llama-compatible weights. Conversion from the training implementation is a pure key rename (no weight permutation); logit equivalence between the original PyTorch model and LlamaForCausalLM was verified to ~1e-6 in fp32. Everything downstream — vLLM, GGUF, GPTQModel — works out of the box.

Training

Optimizer and Schedule

Table with columns: Field, Value
Field	Value
Optimizer	AdamW, betas (0.9, 0.95), eps 1e-8, fused
Weight Decay	0.1 (none on norms or embeddings)
Peak Learning Rate	4e-4
Schedule	WSD (warmup–stable–decay): 2,000-step linear warmup → constant → decay to ~0 across the final 5,700 steps
Gradient Clipping	1.0
Global Batch	524,288 tokens (8 micro-batch x 32 grad-accum x 2,048)
Total Steps	57,200

Three-Phase Curriculum

Table with columns: Phase, Steps, Tokens, Purpose
Phase	Steps	Tokens	Purpose
1 — Foundation	0 – 34,300	18B	Broad language competence. Deliberately unclever.
2 — Domain build	34,300 – 51,500	9B	The six target domains enter at full weight.
3 — Anneal	51,500 – 57,200	3B	Highest-quality material only, LR decaying to zero. Sets the model's voice.

Phase 1 — Foundation (18B tokens)

Table with columns: Source, Share
Source	Share
FineWeb-Edu	62%
StarCoderData	14%
Wikipedia (en)	8%
Project Gutenberg	7%
OpenWebMath	5%
Cosmopedia v2	4%

Phase 2 — Domain build (9B tokens)

Table with columns: Source, Share
Source	Share
FineWeb-Edu	28%
Math (FineMath-4+, OpenWebMath, AlgebraicStack)	20%
Law (hand-built, broad)	18.8%
StarCoderData	15%
peS2o (science)	8%
Cosmopedia v2	6%
Philosophy (hand-built)	4%
Textbooks (OpenStax)	0.2%

Phase 3 — Anneal (3B tokens, LR → 0)

Table with columns: Source, Share
Source	Share
Cosmopedia v2	37%
FineMath-4+	18%
Philosophy (hand-built)	12%
StarCoderData	12%
Law canon (hand-built, most-cited opinions)	10%
FineWeb-Edu	10%
Textbooks (OpenStax)	1%

Data Sources

Table with columns: Source, Origin, License, Role
Source	Origin	License	Role
FineWeb-Edu	`HuggingFaceFW/fineweb-edu` (sample-100BT)	ODC-By	Foundation, all phases
StarCoderData	`bigcode/starcoderdata`	Permissive-licensed code	Code, all phases
Wikipedia	`wikimedia/wikipedia` (20231101.en)	CC BY-SA 4.0	Phase 1

The Hand-Built Corpora

Tokenizer

Performance Benchmarks

Standard Benchmarks

Table with columns: Task, Chochmah-350M (30B tokens), Pythia-410M (300B tokens), SmolLM2-360M (4T tokens)
Task	Chochmah-350M (30B tokens)	Pythia-410M (300B tokens)	SmolLM2-360M (4T tokens)
ARC-Easy (acc / acc_norm)	59.0 / 54.2	51.9 / 45.7	70.4 / 68.0
HellaSwag (acc / acc_norm)	34.8 / 41.8	33.7 / 40.6	43.2 / 56.3
PIQA (acc / acc_norm)	68.6 / 68.3	66.7 / 67.2	72.1 / 72.0
LAMBADA-OpenAI (acc / ppl ↓)	40.2 / 23.8

Standard errors: ARC-Easy ±1.0, HellaSwag ±0.5, PIQA ±1.1, LAMBADA ±0.7.

Per-Domain Held-Out Loss

Validation loss (nats/token) at the final step on held-out sets carved from the unused tail of each source:

Table with columns: Domain, Loss
Domain	Loss
FineWeb-Edu	—
StarCoderData	—
Wikipedia	—
Gutenberg	—
OpenWebMath	—
Cosmopedia	—
FineMath	—
Law	—
Philosophy	—

Sample Outputs

Prompt: The Supreme Court of the United States has long held that

The Supreme Court of the United States has long held that the Due Process Clause of the Fourteenth Amendment requires that a State provide an opportunity for an impartial adjudication before deprivation occurs. See, e. g., Mullane v. Central Hanover Bank & Trust Co., 339 U. S. 306 (1950); Goldberg v. Kelly, 397 U. S. 254 (1970). This Court has never, however, required the government to provide a hearing where the rights guaranteed by the Constitution are not implicated. In the absence of an express constitutional command, the Government is free to place all or some of those rights in any "general rule" which, when applied to particular cases, would be constitutionally permissible. This general rule is one which we have approved. The Court today adopts a new standard under which the Due Process Clause must be satisfied at the time of the deprivation. The Court finds that a hearing must be provided if the State fails to afford an opportunity

Prompt: In Marbury v. Madison, Chief Justice Marshall reasoned that

In Marbury v. Madison, Chief Justice Marshall reasoned that the right of a citizen to be tried by his peers, rather than by a jury of his peers, would not serve as a check on the judiciary. The Chief Justice further stated that in order to prevent judicial oppression, the Constitution could not require a trial by jury unless it was provided by statute, but he did not believe that this provision was necessary because the judiciary was so numerous and well endowed that the process of trying an individual is within its jurisdiction. While this court had earlier recognized that Article III, Section 2 of the Constitution gave a right to a jury trial, Chief Justice Marshall concluded that the process of determining guilt or innocence was not constitutionally required. He stated that "the right to a trial by jury is of course limited by the law of the land, and that it does not extend to a trial by a jury of any particular number."

Prompt: Newton's second law states that

markdown
Newton's second law states that the net external force acting on an object is equal to its mass times its acceleration:
$$\text{net external force} = m \times a_f$$
where $m$ is the mass of the object, $a_f$ is the acceleration due to gravity ($9.81 \text{ m/s^2}$), and $a_f$ represents the acceleration of the object's center of mass relative to its center. By rearranging the equation above for the net external force, we can solve for the unknown variable:
$$\text{net external force} = m \times a_f = m \times g \times h$$
Now that we have solved for $g$, we can substitute it back into our original equation to find the object's acceleration:
$$a_f = \frac{GM}{r^2}$$
where $G$ is the gravitational constant ($6.67 \times 10^{

"Net external force" is OpenStax's exact phrasing. Every formula and constant is real and the LaTeX is well-formed; the derivation connecting them is not physics.

Prompt: Nietzsche's critique of morality begins with the observation that

[...] So why should middle school students care about Nietzschean thought? One reason is that many of his ideas continue to influence modern philosophy and culture. Nietzsche's emphasis on individual freedom and self-expression has been

Usage

Transformers

python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "TevunahAi/Chochmah-350M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).cuda()

prompt = "The Supreme Court of the United States has long held that"
ids = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **ids,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
    repetition_penalty=1.15,
    no_repeat_ngram_size=4,
)
print(tok.decode(out[0], skip_special_tokens=True))

Prompting a Base Model

Continue, don't instruct. Give it the opening of the document you want; it will not follow a question or a command.
The prompt's register selects the corpus. A SCOTUS-style opening produces opinion prose with citations; a textbook sentence produces textbook prose; a def produces code. Write the first line the way the target document would be written.
No BOS token, no chat template. The tokenizer is saved without one; do not add either.
Sample tightly. Temperature 0.5–0.7, top-p 0.9, repetition penalty 1.1–1.15, no_repeat_ngram_size=4 (or min_p=0.05), and 100–150 new tokens. Base models do not know when to stop, and most drift lives in the second half of a long generation.

vLLM

bash
pip install vllm
vllm serve TevunahAi/Chochmah-350M --max-model-len 2048 --dtype bfloat16

Fine-Tuning

Known Issues and Limitations

It will state false things fluently. At this scale the model learns how legal, philosophical, mathematical, and scientific text is written far better than it learns what is true. It must not be used as a source of legal, medical, financial, or factual answers. It is not a substitute for legal advice, and nothing it writes about a case should be assumed to be in that case.
Base model. No instruction following, no safety tuning, no chat template, no stop behavior beyond <|endoftext|>.
Code is the weakest register. Indentation can break after a docstring and logic is unreliable; expect HumanEval near zero. This is expected at 350M / 30B tokens.
Formatting artifacts from the data. Two are visible in generations: Gutenberg-derived text is reproduced with its original hard line wraps, and case names sometimes appear on their own lines as in CourtListener's HTML-to-text conversion. A re-annealed revision on cleaned phase-3 data is planned and will be pushed to this repository as a new commit with a changelog.
Narrative fiction is underweighted, which is what the LAMBADA result reflects.
English only, 2,048-token context. Some Wikipedia-derived knowledge is dated 2023-11.
Repetition at long generation lengths, as with all small models; see the sampling guidance above.

Memory Requirements

Inference

fp32 (as released): 1.43 GB weights; ~2 GB VRAM at full 2,048 context
bf16: ~0.72 GB weights; ~1.2 GB VRAM at full context
Runs comfortably on CPU; a few hundred tokens per second on a modern GPU without a KV cache, far more with one

Training (Reproduction)

Used: RTX 5000 Ada (32 GB) + Dual Xeon Max 9480 (128 GB HBM2e + 256 GB DDR5) for the data pipeline
Peak VRAM: 26.4 GB at micro-batch 8 x 2,048 with torch.compile; micro-batch 16 does not fit in 32 GB
Tokenized data: ~60 GB (uint16 shards) — fits in page cache on 256 GB of RAM, so the GPU never waits on data
Raw downloads: ~500 GB, dominated by the 53 GB CourtListener bulk export and FineWeb-Edu

Technical Specifications

Table with columns: Specification, Value
Specification	Value
Model Family	TevunahAi Chochmah
Variant	350M (v1.0, base)
Total Parameters	358,663,168
Non-Embedding Parameters	308,331,520
Layers	24
Hidden Size	1,024
Attention Heads	16
KV Heads	16 (MHA)
Head Dimension	64

Roadmap

v1.1 — re-anneal of the stable-phase checkpoint on cleaned phase-3 data (hard wraps and citation line breaks removed, Cosmopedia share reduced), pushed to this repo as a new commit with a changelog
Chochmah-350M-Instruct — supervised fine-tuning on an open instruction set
TevunahAi Ultra-Hybrid GPTQ variant of the instruct model
Chochmah-1B — same pipeline and corpora, ~1B parameters, 40–50B tokens

License

Weights: Apache-2.0.

Citation

bibtex
@misc{chochmah350m_2026,
  title  = {Chochmah-350M: a from-scratch pretrained base model on one GPU},
  author = {TevunahAi},
  year   = {2026},
  url    = {https://huggingface.co/TevunahAi/Chochmah-350M},
  note   = {359M parameters, 30B tokens, three-phase WSD curriculum with hand-built case-law and philosophy corpora}
}

Acknowledgments

Hugging Face / SmolLM team for the SmolLM2 tokenizer, FineWeb-Edu, FineMath, and Cosmopedia — and for the SmolLM reports that made the education-weighted recipe a known quantity
Free Law Project for the CourtListener bulk data that made the law corpus possible
Project Gutenberg and the sedthh/gutenberg_english packaging
OpenStax for publishing textbook sources under CC BY
BigCode (StarCoderData), Allen AI (peS2o), Wikimedia
EleutherAI for lm-evaluation-harness, Proof-Pile-2, and the Pythia suite used as the compute-matched reference

Pretrained by TevunahAi AI Model Quantization and On-Premises AI Infrastructure From-scratch models, hybrid-architecture quantization (Mamba, MoE, SSM), and enterprise hardware bring-up

https://huggingface.co/TevunahAi