ajaxdavis

alpha-60m-chat

README

License: apache-2.0

Use

python
from transformers import pipeline

chat = pipeline("text-generation", model="ajaxdavis/alpha-60m-chat")
result = chat(
    [{"role": "user", "content": "Hey, how's your day going?"}],
    max_new_tokens=64,
    do_sample=False,
)
print(result[0]["generated_text"][-1]["content"])

The repository is a standard LlamaForCausalLM export with model.safetensors, tokenizer.json, and a Jinja chat template. It needs no custom code and no trust_remote_code=True.

The tokenizer template ends generation prompts exactly at <|assistant|>. Do not manually append a space after that marker; the first generated byte-BPE content token owns its leading space.

Architecture

Table with columns: Property, Value
Property	Value
Parameters	57,688,576
Layers	16
Hidden width	512
Attention heads / KV heads	8 / 8
SwiGLU intermediate width	1,408
Context length	512
Vocabulary	12,288-token Alpha byte-level BPE
Positions / normalization	RoPE (`theta=10000`) / RMSNorm (`eps=1e-5`)
Embeddings	input/output weights tied
Export dtype	float32

Corrective training

The release starts from the clean terminal pre-SFT base rather than continuing the failed SFT epoch.

Clean base SHA-256: 08e14fa9604bf1b46ebcd5df37933c84d2496c1d05d9e4b32ebad98792cc6049.
Training source: 57c065e35c7564688726dafb404efaff952d860b.
Prompt-boundary fix: cf4ad61.
2,200 finite steps, batch 32, context 512.
AdamW, learning rate 5e-5 cosine-decayed to 5e-6, 100 warmup steps.
Deterministic epoch shuffle.
Equal total supervised weight per conversation.
First four assistant content tokens weighted 8x.
Terminal EOS weighted 2x and excluded from answer-start weighting.
Full float32 training through Helios Vulkan on one RTX 4090; CPU fallback forbidden for model-sized GPU ops.

The full run took 4,421 seconds wall-clock. Its 2,200 measured iterations had median throughput 8,733 tokens/s, 10th percentile 8,091 tokens/s, and zero allocator free-range overflow. The measured iteration time corresponds to about $0.81 a tt h e p o d' s$ 0.69/hour price; setup, evaluation, and storage are excluded.

Checkpoint 1,200 was selected by deterministic free generation. Step 2,000 reached the best teacher-forced validation loss (1.6814) but failed to stop on two development prompts and had substantially worse maximum repetition. Lower validation loss did not override worse conversation.

Corrective data

The compact repair corpus contains 34,880 unique conversations:

Table with columns: Source, Conversations, Terms noted by source
Source	Conversations	Terms noted by source
SODA	30,000	CC-BY-4.0
SmolTalk2 everyday conversation	2,260	Apache-2.0 generated dataset; upstream terms may apply
Smol concise subset	1,436	constituent-source terms may apply
OpenAssistant/oasst2	1,184	Apache-2.0

The deterministic split contains 33,113 training and 1,767 development conversations. Exact duplicate rendered conversations were removed. All selected rows fit the 512-token limit without cutting an assistant response, and independent assistant-only mask audits passed.

SODA supplies about 86% of this corrective mixture. That helped short conversational turn-taking but also biases the model toward fictional human-to-human roleplay, invented personal circumstances, and vague continuation. Future work should test a more direct assistant mixture rather than blindly continuing this run.

No AlphaCorpus synthetic-curriculum candidate was used in this checkpoint.

Why the old release often returned empty

The old result combined two independent problems.

First, one ordered SFT epoch let long answer interiors dominate conversation starts. Teacher-forced loss could look healthy while EOS still won the first generated-token decision.

Second, the evaluator, native API, and exported chat template appended a literal space after the terminal assistant marker. With Alpha's byte BPE:

Table with columns: Text, Token IDs
Text	Token IDs
`<	assistant
`<	assistant
`<	assistant

The standalone token 32 never occupied that boundary during SFT and could activate code-fence/forum loops. Changing only the boundary fixed such a loop on the selected checkpoint under both the fast and reference native inference paths. Commit cf4ad61 adds regression checks across evaluation, serving, suite construction, and HF export. Historical assistant turns retain their required space before known content.

Evaluation

All listed generations use temperature-zero greedy decoding.

Repair-development checkpoint sweep

Table with columns: Step, Nonempty, Structural / EOS, Role leaks, Loops, Mean / max 4-gram repetition
Step	Nonempty	Structural / EOS	Role leaks	Loops	Mean / max 4-gram repetition
1,200	48/48	48/48	0	5	0.04034 / 0.400
1,600	48/48	46/48	0	6	0.07994 / 0.928
1,800	48/48

Untouched frozen suite

Table with columns: Metric, Selected step 1,200
Metric	Selected step 1,200
Structural pass	55 / 100
Nonempty	70 / 100
EOS terminated	56 / 100
Empty	30 / 100
Role leaks	0 / 100
Degenerate loops	31 / 100
Mean / maximum 4-gram repetition	0.1414 / 0.840
Closed-book QA exact / answer contained	0 / 200 / 0 / 200
Closed-book QA mean token F1

The development suite substantially overestimated generalization. The selected checkpoint is materially more responsive than the archived release, but the untouched result still contains 30 empty replies, 44 replies that do not stop on EOS, and 31 detected repetition loops. It is therefore not a dependable conversational model. The frozen chat and QA output SHA-256 values are 3f1a178299468be0549f32f7c871445de2113ed652bfd82c3068588445311570 and 137a3981401e0563dd1bdde2e2fc86aa04112363deb10a879d10b3fb495c9300.

The historical terminal checkpoint's 2/100 structural and 92/100 empty result describes the exact old published runtime. Because that runtime also contained the prompt-boundary defect, it is not a clean model-only control for the corrected checkpoint. Both records are preserved rather than rewriting the old one.

No model judge supplied the semantic assessment in this card. The development examples below were inspected directly, and the full final frozen outputs are published as evidence.

Exact output examples

Table with columns: User input, Greedy output, Assessment
User input	Greedy output	Assessment
“Hey, how's your day going?”	“It's going well, thanks. How about you?”	Natural, direct turn-taking
“You won't believe what happened at lunch.”	“What happened?”	Concise and responsive
“Continue this in two sentences: The elevator opened onto a beach.”	“The elevator opened onto a beach.”	Fluent but only repeats the seed and ignores the instruction
resolved financial-bank follow-up	“Well, it's possible that the bank was crowded. But it's not a riverbank.”	Retains the sense, but does not answer the question
“The plan is held together with string. What does that imply?”	“It means that the plan is held together with string. It's a string that is used to hold the plan together with string.”

Stock Transformers message-list generation independently returned: " It's going well. I'm doing alright. ".

Limitations

Responses are often shallow, irrelevant, contradictory, or roleplay-like even when they are grammatical.
Repetition is reduced but not solved.
Multi-turn stability beyond the short frozen cases is not established.
Closed-book factual recall is weak by design and by training history; use retrieval for facts.
The model is not safety-aligned and must not be used for medical, legal, financial, security, or other consequential advice.
The web and dialogue sources can contain errors, bias, unsafe content, personal information, and copyrighted text. The model may reproduce or transform such material.
Contexts above 512 tokens are unsupported.
Greedy frozen evaluation is useful for format, stopping, and repetition but is not a comprehensive capability or safety benchmark.

Reproducibility

Selected native checkpoint SHA-256: 399f776b49acc0c8834ff8a7f2390454e2c5f2d833a264e3f83ff546e973cfec
Standard model.safetensors SHA-256: a5214ebad501b8bd3b09f7552c0db67417d18c3b66432f66f847de0e723dd688
Training corpus SHA-256: 298ca5332fc23e19bdd5f8cdb2a1a94e54dadfb48579ab6a58097f20135e1744
Development corpus SHA-256: 412d9dcc5bd36af38140f288fce9a94b9de9287fd0bdc5e9bc10101c1c2f4645
Repair manifest SHA-256: 792256f22e43777fbc188141b11358d29c25773bea7f50583fd0ba213081bfa6
NVIDIA gate: 46/46 executed and passed, zero skipped.
Native/Transformers parity: 87/87 top-1 positions, 5/5 exact tokenizer prompts, maximum logit difference 5.531e-05 at tolerance 1e-3.

The native optimizer-bearing checkpoint, exact run contract, metrics, corrected evaluations, and restart notes are archived in ajaxdavis/alpha-60m-training-checkpoints.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ajaxdavis

Model Tree

Base

ajaxdavis/alpha-60m-base

Fine-tuned

this model

Input Modalities

Text

Output Modalities