laskar-ks

alcyone-v1-dpo

README

License: apache-2.0

What it does

python

from transformers import AutoTokenizer, AutoModelForCausalLMimport torch model_id = "laskar-ks/alcyone-v1-dpo"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id).eval() def chat(message, temperature=0.7):    prompt = f"<|user|>\n{message}\n<|assistant|>\n"    ids = tok(prompt, return_tensors="pt", add_special_tokens=False)    out = model.generate(**ids, max_new_tokens=220, do_sample=True,                         temperature=temperature, top_p=0.95,                         repetition_penalty=1.05,                         eos_token_id=tok.eos_token_id,                         pad_token_id=tok.pad_token_id)    full = tok.decode(out[0], skip_special_tokens=False)    return full.split("<|assistant|>")[-1].replace(tok.eos_token, "").strip() print(chat("Write a story about a mountain."))print(chat("Features: Dialogue\nWords: cat, ball, tree"))print(chat("Introduce yourself"))

It accepts three prompt styles, because all three appear in its SFT data:

Table with columns: Style, Example
Style	Example
structured	`Features: Dialogue` + `Words: cat, ball, tree`
natural	`Write a short story using the words: moon and boat.`
free topic	`Write a story about a mountain.`

Results

Measured on 200 held-out instruction prompts, three random seeds, identical scorer for all three models. The scorer is rule-based and fully reproducible — no LLM judge.

Table with columns: Metric, Base (pretrained), + SFT, + DPO
Metric	Base (pretrained)	+ SFT	+ DPO
Required words used	0.232 ± 0.011	0.784 ± 0.011	0.959 ± 0.006
Stops cleanly	0.852 ± 0.019	0.825 ± 0.009	0.987 ± 0.006
No fabricated user turn	1.000 ± 0.000	1.000 ± 0.000	1.000 ± 0.000
Reasonable length	0.528 ± 0.033	1.000 ± 0.000

Two things worth reading carefully.

Each stage fixes something different. SFT contributes almost all of the gain in required words (+55 points) and length (+47 points). DPO contributes the rest of required words (+17) and nearly all of stops cleanly (+16). Neither stage is redundant.

SFT slightly degraded one capability. Stops cleanly went from 0.852 at base to 0.825 after SFT — mild catastrophic forgetting, since the base model had been trained with an EOS token at the end of every story and SFT taught it a new output shape. DPO recovered it and went past both.

What the stages look like

Prompt:

text

Words: order, speed, ancientRandom sentence: He was very fast and had lots of speed.

Base — reads the field label as story content and invents a character called "Randomone":

One day, Randomone met a small boy. The boy asked Randomone: "What do you have to say?" Randomone replied: "I order an ancient book!" …

After SFT — writes a proper story, uses ancient, misses order and speed:

Once upon a time, there was a little car named Red. Red loved to go fast on the road. One day, Red saw an ancient car that was very old. …

After DPO — uses all three required words and embeds the requested sentence verbatim:

Once upon a time, there was an ancient car. It was very old and needed to be ordered. … He was very fast and had lots of speed. …

Training

Stage 1 — Pretraining from scratch

Table

Architecture	GPT-2 style, 12 layers / 12 heads / 768 dim
Parameters	97.7M
Tokenizer	byte-level BPE, 16,000 vocab, trained on TinyStories
Context length	512
Data	TinyStories, 462.6M tokens × 2 epochs = 925M tokens
Precision	bf16 mixed precision + SDPA attention
Optimizer	AdamW, lr 6e-4, cosine schedule, 1% warmup
Final eval loss	1.3108 (perplexity 3.71)
Wall time

Train loss 1.2599 vs eval loss 1.3108 — a gap of 0.05 after two full epochs. The model is data-bound, not capacity-bound: it still had room to learn when TinyStories ran out. Scaling the model further without more data would not have helped.

Stage 2 — Supervised fine-tuning

Table

Data	TinyStoriesInstruct, ~143k instruction/story pairs
Chat template	`<\|user\|>` / `<\|assistant\|>` added as new tokens (vocab 16,000 → 16,008)
Prompt styles	structured 40%, natural 35%, free topic 25%
Persona data	1,500 identity examples (~1% of the mix)
Loss masking	loss computed on the response only, prompt tokens set to -100
Optimizer	AdamW, lr 1e-4, 2 epochs
Final eval loss	?

The two chat tokens were initialized by mean resizing — drawn from a distribution matching the existing embeddings' mean and covariance, rather than pure noise.

The persona ("I am Alcyone, a tiny helper who writes little stories") is trained into the weights rather than supplied by a system prompt, which costs zero context tokens. It was built from 25 question phrasings × 7 answers, paired at random, so the model learns the concept rather than memorizing one string.

Persona generalization: ? on 8 phrasings never seen during training. Held-out evaluation, not a recall test.

Stage 3 — Preference optimization (DPO)

Table

Preference pairs	2,615 (from 3,000 prompts × 4 on-policy samples)
Mean score margin	0.2752
β	0.1
Optimizer	AdamW, lr 1e-5, 1 epoch
Final loss	0.556 (from 0.693 at init)
Preference accuracy	0.704
Wall time	0.9 min

Negatives are on-policy: the model generated four candidates per prompt at temperature 0.9, a rule-based scorer ranked them, and best-vs-worst became the pair. Synthetically corrupted negatives were deliberately avoided — the model already assigns them near-zero probability, so correcting them teaches nothing.

The DPO loss was implemented directly rather than through a library, with a frozen reference copy of the SFT model as the anchor.

DPO is a closed-form reformulation of the RLHF objective. This model was not trained with PPO or a learned reward model.

The scorer

Five rules, all machine-checkable, weighted:

Table with columns: Rule, Weight
Rule	Weight
Fraction of required words present	0.45
Response ends on terminal punctuation	0.20
No fabricated user turn	0.15
Length between 40 and 300 words	0.10
No verbatim repeated sentence	0.10

The same function produced the DPO preference pairs and the evaluation table above.

Limitations

It is a story writer, not an assistant. It has no world knowledge. Ask it the capital of France and it will write a children's story. Nothing in TinyStories could have given it facts, and no amount of fine-tuning adds knowledge that was never there.

English only. The 16k tokenizer was trained purely on TinyStories; other languages shatter into meaningless fragments.

512-token context. That budget is shared by the prompt, any conversation history, and the answer. Multi-turn conversation is possible in form but not useful in practice — two or three turns fill the window.

It uses words without knowing their part of speech. Outputs contain artifacts like "the independent trees" or "looking at the fair". The model learned that a word occurs in a context, not what kind of word it is. This is a ceiling of the 98M scale, not a training bug.

The scorer measures part of the instruction, not all of it. It checks the Words field only. Features and Random sentence compliance is not measured, so 0.978 is not a claim of full instruction compliance.

Word matching is prefix-based. game matches games and reach matches reached — intended, since the model writes inflected forms. The side effect is that bad also matches badge. This slightly inflates the word-rate metric for all three models equally.

Source-data vocabulary. TinyStoriesInstruct occasionally includes words such as "gun" in its required-word lists. Stories remain children's stories, but the vocabulary is inherited from the source corpus, not curated here.

The persona is narrow by design — name and role only. It makes no claims about its own architecture, because a model this size would repeat such claims without understanding them.

Running it

Measured at 72.8 tokens/sec on one A100, batch size 1, eager HF generate. That number is latency-bound by kernel launches at batch 1, not compute-bound — the GPU is mostly idle.

Weights are ~200 MB in bf16, ~400 MB in fp32. It runs comfortably on CPU. Quantized to GGUF it fits well under 100 MB.

Project stages

Table with columns: Repo, Stage
Repo	Stage
`alcyone-v0`	first experiment, 30.5M, eval loss 1.81
`alcyone-base-100m`	pretrained base, 97.7M, eval loss 1.3108
`alcyone-v1`	+ SFT, chat template, persona
`alcyone-v1-dpo`

The v1 label was reserved from the start for the first chat-capable model. Everything before it is the v0 stage.

About the name

Alcyone (η Tauri) is the brightest star in the Pleiades cluster. Part of a star-named family alongside Parallax, Altair, Fornax and others.

Author

Trained by Laskar as part of an AI engineering portfolio exploring agentic systems and foundational ML.

Notebooks for all three stages are in this repository under notebooks/.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.