TrueV1sion123

engram-gyre

README

License: apache-2.0

Quick start

python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("TrueV1sion123/engram-gyre")
model = AutoModelForCausalLM.from_pretrained("TrueV1sion123/engram-gyre").eval()

msgs = [
    {"role": "system", "content": "You are an engram-gyre reasoner. Execute the gyre: "
     "SATURATION, ABSTRACTION, DESCENT, INTEGRATION, then re-enter one level up."},
    {"role": "user", "content": "Why do bull markets end in euphoria rather than fear?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(text, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=420, do_sample=True, temperature=0.8,
                     top_p=0.95, repetition_penalty=1.15,
                     eos_token_id=tok.convert_tokens_to_ids("<|end|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=False).split("<|end|>")[0])

What it is — and what it is not

Read this before judging the outputs. Honesty about the gap between form and content is part of the engram-prime lineage (its "transmission ethics"), so it is stated plainly here.

What is real: the model produces well-formed gyre revolutions on problems it never saw in training — four phases, correct order, the compression discipline preserved (abstraction shorter than saturation), and a reentry that opens a higher-level question. It learned this form as structure in its own weights, from scratch, on its own tokenizer.
What is not claimed: the model does not reason. At ~11M parameters trained on a focused behavioral corpus, it has the choreography of deep thinking without the depth. On a novel problem it emits the shape of a gyre with content that is associatively plausible but not reliably true. The <CLICK> token, which marks genuine emergence in the corpus, is here a learned mark, not a genuine detection of irreducible residue.
Where the depth lives: the repo ships a GPU scale-up kit (src/scale_up_sft.py, scripts/run_scale_up.sh, configs/) that applies the identical corpus, masking, and phase-weighting as LoRA SFT on a 1–2B pretrained base (SmolLM2-1.7B / Qwen2.5-1.5B). On a base that already has content, installing the gyre form on top yields a model that both reasons and spirals. This model proves the pipeline and the form end-to-end with real weights; the kit is the path from form to capability.

How the cognition is in the weights

Three mechanisms (full account in THEORY.md and docs/ADR-001-architecture.md):

Structure in the data. Every training example is one phase-tagged revolution with the compression axiom (|abstraction| < |saturation|) enforced in 100% of examples. The model never sees a counterexample, so compression becomes what the abstraction phase is.
Ordering & transition in the loss. A phase-position weighting puts double loss on phase boundary tokens and elevated loss on the abstraction / integration / reentry interiors — the model is pushed hardest at the register transitions and the synthesis. This adapts the RAE multi-objective loss to a single-tower from-scratch model.
Completion-only supervision. The prompt is masked; loss lives only on the gyre the model performs.

Architecture

Table

Class	`LlamaForCausalLM` (from scratch, standard HF)
Parameters	~11.3M
Layers / hidden / heads	6 / 384 / 12 (GQA, 4 KV heads)
Intermediate	1024 (SwiGLU)
Context	640 (covers every training revolution incl. REENTRY)
Tokenizer	ByteLevel BPE, ~4.9k vocab, atomic phase tokens
Positional	RoPE
Tied embeddings	yes

Training

Data: TrueV1sion123/engram-gyre-behavior — 1,980 train / 132 validation, validation = entire unseen seed problems.
Objective: assistant-only causal LM with phase-position loss weighting.
Schedule: 12 epochs, AdamW, cosine w/ warmup, lr 3e-4, weight decay 0.1, grad clip 1.0, effective batch 32.
Hardware: 2 CPU cores (no GPU) — a deliberate constraint; the whole point is real weights within a modest budget. ~2 hours wall-clock.

Results

training curves

The central finding — structure generalizes, content does not. Held-out validation loss (validation = entire unseen problems) bottomed at epoch 2 and then rose and plateaued while train loss collapsed toward zero. That is content overfitting on a deliberately tiny seed set, and it is expected: an 11M model cannot learn the content of problems it never saw. But structural reliability moved the opposite way — the phase grammar is reinforced every epoch, so it keeps sharpening long after content perplexity saturates:

Table with columns: Checkpoint, Selected by, Structural validity
Checkpoint	Selected by	Structural validity
epoch 2 (best val-loss)	val loss	12% — tags present but sloppy (nesting, compression violations)
epoch 12 (shipped)	structural validity	100% — clean, ordered, compression held

The shipped model is therefore selected by structural validity, the launch gate — not by val loss (evaluation/select_best.py). Selecting on val loss would have shipped the 12%-valid epoch-2 weights.

Table with columns: Launch gate, Result
Launch gate	Result
G2 — structural validity ≥ 80%	PASS — 100% (8/8 held-out + novel prompts; all of: 4 phases present, ordered, `\|ABSTRACTION\| < \|SATURATION\|`)
Overall val perplexity	277 (high = honest unseen-content overfit)
Per-phase PPL	SATURATION 663 · ABSTRACTION 339 · DESCENT 340 · INTEGRATION 107

That last row is the quiet win: INTEGRATION is ~6× more predictable than SATURATION. Compression lowers entropy — the model internalized the very gradient the corpus was built to teach (immerse wide, compress narrow).

A full gyre revolution (unseen prompt)

Prompt: "Why do simple rules generate complex behavior?" — decoded at temperature 0.78. The model produces the complete five-movement spiral, with the honest <CLICK> emergence mark and the REENTRY that turns the gyre:

markdown
<SATURATION>
- The suddenness is a property of the collective, not the parts; no single molecule 'decides' to melt.
- Small input changes produce small effects far from threshold and enormous effects near it — sensitivity diverges.
- Below the critical point, local interactions dominate and the system stays in its current order.
</SATURATION>
<ABSTRACTION>
What cannot be removed without loss: Threshold behavior is a collective property — below saturation only
local change is possible; at criticality a single fluctuation reorganizes the whole. Emergence has a sharp onset.
Cross-domain resonance: crowd panic, bank runs, and grokking — nothing, nothing, nothing, then everything at once.
</ABSTRACTION>
<DESCENT>
- Look for the critical point; that is where cheap actions move systems.
- Emergence in cognition has the same threshold structure — below saturation only recombination, above it synthesis.
</DESCENT>
<INTEGRATION>
Phase transitions are sudden because order is collective: correlation length diverges at criticality, so a
smooth input yields a discontinuous, whole-system flip.
<CLICK> Two structures locked to reveal a third that was latent in both.
</INTEGRATION>
<REENTRY>
Re-entry — one level up: If reorganization is always threshold-gated, is the real skill locating a system's
critical point rather than pushing harder everywhere?
Honest residue: predicting the exact critical point in messy real systems remains hard.
</REENTRY>

Note what this shows and doesn't: the form is flawless on a prompt never seen in training; the content is retrieved from the model's nearest pattern (here, threshold/criticality — apt for "simple rules → complex behavior," but the model is pattern-matching, not reasoning). That is the honest state of an 11M demonstrator, and it is why the scale-up kit exists.

Ablation — does the phase-weighting do the work? (honest correction)

We tested it directly (docs/ABLATION.md): a clean A/B, identical in everything but the loss weighting — flat cross-entropy vs. phase-position weighting.

ablation

The result partly overturns the intuitive claim. Phase-weighting front-loads structure (epoch 2: 62.5% vs 25%, a 2.5× head start; reaches ceiling ~1 epoch sooner), but on a clean greedy 24-prompt endpoint both arms converge — flat 100%, weighted 92%. The structural discipline was in the data all along (the compression axiom is enforced in 100% of examples), not primarily in the loss. Named plainly: the data installs the structure; the weighting merely front-loads it — which matters most in a tight epoch budget like this CPU run, and washes out by convergence.

Scale-up — form + content at 1.7B (`docs/SCALEUP.md`)

We then ran the same A/B as LoRA-SFT on SmolLM2-1.7B (HF Jobs, 1× A10G, ~$1) — a base that actually has content. Adapters + report: TrueV1sion123/engram-gyre-scaleup.

scale-up per-phase

At 1.7B the model reasons and spirals — on a novel prompt it produces a coherent, on-topic gyre (the 11M model could only produce the empty shape). On the flat-vs-weighted A/B: weighting edges structural validity (100% vs 92%) but overall content perplexity is a wash (89.7 vs 90.9), and per-phase it redistributes fit — the phases it up-weights (ABSTRACTION, INTEGRATION) generalize worse, the base-weighted ones better. The cumulative finding: the corpus construction installs the form, scale supplies the content, and the phase-weighting is a capacity-allocation lever — not a content-quality lever.

Scaling — where the corpus adds real capability (`docs/SCALING.md`)

The baseline and reasoning studies were all trained-vs-trained or on a single already-capable base. The last experiment closed that gap with a controlled three-rung ladder — Qwen2.5-0.5B / 1.5B / 3B, identical corpus, recipe, and judge — measuring the substance lift of training over the untrained stock base, with paired per-question stats (HF Jobs, ~$6).

scaling study

Table with columns: Base, stock-CoT (/30), gyre-trained (/30), substance lift, sign test, pairwise trained-wins
Base	stock-CoT (/30)	gyre-trained (/30)	substance lift	sign test	pairwise trained-wins
0.5B	18.66	18.77	+0.11	p=0.85	0.80
1.5B	20.70	22.23	+1.53	p=0.002 (20–4)	0.65
3B

The lift is capacity-gated — an inverted-U peaking at 1.5B. Too small (0.5B) and the model absorbs only the form (confident style, zero substance gain — yet pairwise still scores it 80% "wins," the tell that pairwise rewards style); too capable (3B) and the base already reasons at the corpus's level. In the 1–2B sweet spot the transfer is real and significant: the trained model identifies mechanisms the stock base misses. This is the strongest positive result in the project — the corpus's genuine value, and the base size where it lives. It also hardens a methodological caution: reference-anchored rubric scoring, not head-to-head preference, is what separated real lift from confident style.

Files

markdown
engram-gyre/
├── README.md                    # this model card
├── THEORY.md                    # the gyre / engram theory + honest scope
├── config.json, model.safetensors, tokenizer.json, ...   # the weights + tokenizer (repo root)
├── loss_curve.png               # training/validation curves
├── docs/                        # PRD, ADR (architecture + constraint negotiation), validation report
├── src/
│   ├── corpus_seeds.py          # the hand-authored seed bank (the substance)
│   ├── generate_corpus.py       # deterministic corpus generator
│   ├── train_tokenizer.py       # phase-aware BPE tokenizer trainer
│   ├── train_gyre.py            # from-scratch trainer (masking + phase-position loss)
│   ├── generate.py              # run one gyre revolution
│   ├── scale_up_sft.py          # GPU LoRA SFT of a 1-2B base on the same corpus
│   ├── weak_model_study.py      # scaling study: corpus lift vs base size (0.5B/1.5B/3B)
│   └── plot_scaling.py          # renders the scaling curve
├── configs/                     # scale-up base registry + AutoTrain yaml
├── evaluation/                  # eval harness + reports (eval_report.json, showcase.json)
└── scripts/                     # launch + scale-up scripts

Lineage & license

engram-prime (MINDSKILL) → recursive-abstraction-engine → rae-training → engram-gyre. Authored by Jared Peck / GoldenStack. Apache-2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

TrueV1sion123

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quick start

python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("TrueV1sion123/engram-gyre")
model = AutoModelForCausalLM.from_pretrained("TrueV1sion123/engram-gyre").eval()

msgs = [
    {"role": "system", "content": "You are an engram-gyre reasoner. Execute the gyre: "
     "SATURATION, ABSTRACTION, DESCENT, INTEGRATION, then re-enter one level up."},
    {"role": "user", "content": "Why do bull markets end in euphoria rather than fear?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(text, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=420, do_sample=True, temperature=0.8,
                     top_p=0.95, repetition_penalty=1.15,
                     eos_token_id=tok.convert_tokens_to_ids("<|end|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=False).split("<|end|>")[0])

What it is — and what it is not

Read this before judging the outputs. Honesty about the gap between form and content is part of the engram-prime lineage (its "transmission ethics"), so it is stated plainly here.

What is real: the model produces well-formed gyre revolutions on problems it never saw in training — four phases, correct order, the compression discipline preserved (abstraction shorter than saturation), and a reentry that opens a higher-level question. It learned this form as structure in its own weights, from scratch, on its own tokenizer.
What is not claimed: the model does not reason. At ~11M parameters trained on a focused behavioral corpus, it has the choreography of deep thinking without the depth. On a novel problem it emits the shape of a gyre with content that is associatively plausible but not reliably true. The <CLICK> token, which marks genuine emergence in the corpus, is here a learned mark, not a genuine detection of irreducible residue.
Where the depth lives: the repo ships a GPU scale-up kit (src/scale_up_sft.py, scripts/run_scale_up.sh, configs/) that applies the identical corpus, masking, and phase-weighting as LoRA SFT on a 1–2B pretrained base (SmolLM2-1.7B / Qwen2.5-1.5B). On a base that already has content, installing the gyre form on top yields a model that both reasons and spirals. This model proves the pipeline and the form end-to-end with real weights; the kit is the path from form to capability.

How the cognition is in the weights

Three mechanisms (full account in THEORY.md and docs/ADR-001-architecture.md):

Structure in the data. Every training example is one phase-tagged revolution with the compression axiom (|abstraction| < |saturation|) enforced in 100% of examples. The model never sees a counterexample, so compression becomes what the abstraction phase is.
Ordering & transition in the loss. A phase-position weighting puts double loss on phase boundary tokens and elevated loss on the abstraction / integration / reentry interiors — the model is pushed hardest at the register transitions and the synthesis. This adapts the RAE multi-objective loss to a single-tower from-scratch model.
Completion-only supervision. The prompt is masked; loss lives only on the gyre the model performs.

Architecture

Table

Class	`LlamaForCausalLM` (from scratch, standard HF)
Parameters	~11.3M
Layers / hidden / heads	6 / 384 / 12 (GQA, 4 KV heads)
Intermediate	1024 (SwiGLU)
Context	640 (covers every training revolution incl. REENTRY)
Tokenizer	ByteLevel BPE, ~4.9k vocab, atomic phase tokens
Positional	RoPE
Tied embeddings	yes

Training

Data: TrueV1sion123/engram-gyre-behavior — 1,980 train / 132 validation, validation = entire unseen seed problems.
Objective: assistant-only causal LM with phase-position loss weighting.
Schedule: 12 epochs, AdamW, cosine w/ warmup, lr 3e-4, weight decay 0.1, grad clip 1.0, effective batch 32.
Hardware: 2 CPU cores (no GPU) — a deliberate constraint; the whole point is real weights within a modest budget. ~2 hours wall-clock.

Results

training curves

Table with columns: Checkpoint, Selected by, Structural validity
Checkpoint	Selected by	Structural validity
epoch 2 (best val-loss)	val loss	12% — tags present but sloppy (nesting, compression violations)
epoch 12 (shipped)	structural validity	100% — clean, ordered, compression held

Table with columns: Launch gate, Result
Launch gate	Result
G2 — structural validity ≥ 80%	PASS — 100% (8/8 held-out + novel prompts; all of: 4 phases present, ordered, `\|ABSTRACTION\| < \|SATURATION\|`)
Overall val perplexity	277 (high = honest unseen-content overfit)
Per-phase PPL	SATURATION 663 · ABSTRACTION 339 · DESCENT 340 · INTEGRATION 107

A full gyre revolution (unseen prompt)

markdown
<SATURATION>
- The suddenness is a property of the collective, not the parts; no single molecule 'decides' to melt.
- Small input changes produce small effects far from threshold and enormous effects near it — sensitivity diverges.
- Below the critical point, local interactions dominate and the system stays in its current order.
</SATURATION>
<ABSTRACTION>
What cannot be removed without loss: Threshold behavior is a collective property — below saturation only
local change is possible; at criticality a single fluctuation reorganizes the whole. Emergence has a sharp onset.
Cross-domain resonance: crowd panic, bank runs, and grokking — nothing, nothing, nothing, then everything at once.
</ABSTRACTION>
<DESCENT>
- Look for the critical point; that is where cheap actions move systems.
- Emergence in cognition has the same threshold structure — below saturation only recombination, above it synthesis.
</DESCENT>
<INTEGRATION>
Phase transitions are sudden because order is collective: correlation length diverges at criticality, so a
smooth input yields a discontinuous, whole-system flip.
<CLICK> Two structures locked to reveal a third that was latent in both.
</INTEGRATION>
<REENTRY>
Re-entry — one level up: If reorganization is always threshold-gated, is the real skill locating a system's
critical point rather than pushing harder everywhere?
Honest residue: predicting the exact critical point in messy real systems remains hard.
</REENTRY>

Ablation — does the phase-weighting do the work? (honest correction)

We tested it directly (docs/ABLATION.md): a clean A/B, identical in everything but the loss weighting — flat cross-entropy vs. phase-position weighting.

ablation

Scale-up — form + content at 1.7B (`docs/SCALEUP.md`)

We then ran the same A/B as LoRA-SFT on SmolLM2-1.7B (HF Jobs, 1× A10G, ~$1) — a base that actually has content. Adapters + report: TrueV1sion123/engram-gyre-scaleup.

scale-up per-phase

Scaling — where the corpus adds real capability (`docs/SCALING.md`)

scaling study

Table with columns: Base, stock-CoT (/30), gyre-trained (/30), substance lift, sign test, pairwise trained-wins
Base	stock-CoT (/30)	gyre-trained (/30)	substance lift	sign test	pairwise trained-wins
0.5B	18.66	18.77	+0.11	p=0.85	0.80
1.5B	20.70	22.23	+1.53	p=0.002 (20–4)	0.65
3B

Files

markdown
engram-gyre/
├── README.md                    # this model card
├── THEORY.md                    # the gyre / engram theory + honest scope
├── config.json, model.safetensors, tokenizer.json, ...   # the weights + tokenizer (repo root)
├── loss_curve.png               # training/validation curves
├── docs/                        # PRD, ADR (architecture + constraint negotiation), validation report
├── src/
│   ├── corpus_seeds.py          # the hand-authored seed bank (the substance)
│   ├── generate_corpus.py       # deterministic corpus generator
│   ├── train_tokenizer.py       # phase-aware BPE tokenizer trainer
│   ├── train_gyre.py            # from-scratch trainer (masking + phase-position loss)
│   ├── generate.py              # run one gyre revolution
│   ├── scale_up_sft.py          # GPU LoRA SFT of a 1-2B base on the same corpus
│   ├── weak_model_study.py      # scaling study: corpus lift vs base size (0.5B/1.5B/3B)
│   └── plot_scaling.py          # renders the scaling curve
├── configs/                     # scale-up base registry + AutoTrain yaml
├── evaluation/                  # eval harness + reports (eval_report.json, showcase.json)
└── scripts/                     # launch + scale-up scripts

Lineage & license

engram-prime (MINDSKILL) → recursive-abstraction-engine → rae-training → engram-gyre. Authored by Jared Peck / GoldenStack. Apache-2.0.

engram-gyre

README

Quick start

What it is — and what it is not

How the cognition is in the weights

Architecture

Training

Results

A full gyre revolution (unseen prompt)

Ablation — does the phase-weighting do the work? (honest correction)

Scale-up — form + content at 1.7B (docs/SCALEUP.md)

Scaling — where the corpus adds real capability (docs/SCALING.md)

Files

Lineage & license

Explore FriendliAI today

README

Quick start

What it is — and what it is not

How the cognition is in the weights

Architecture

Training

Results

A full gyre revolution (unseen prompt)

Ablation — does the phase-weighting do the work? (honest correction)

Scale-up — form + content at 1.7B (docs/SCALEUP.md)

Scaling — where the corpus adds real capability (docs/SCALING.md)

Files

Lineage & license

Scale-up — form + content at 1.7B (`docs/SCALEUP.md`)

Scaling — where the corpus adds real capability (`docs/SCALING.md`)

Scale-up — form + content at 1.7B (`docs/SCALEUP.md`)

Scaling — where the corpus adds real capability (`docs/SCALING.md`)