jaweed123

TinyJLLM-Instruct-DPO

README

License: apache-2.0

Model family

Table with columns: Model, Stage, Link
Model	Stage	Link
`TinyJLLM`	Base	jaweed123/TinyJLLM
`TinyJLLM-Instruct`	SFT	jaweed123/TinyJLLM-Instruct
`TinyJLLM-Instruct-DPO`	DPO (this model)	jaweed123/TinyJLLM-Instruct-DPO

Usage

Same marker format and the same response-masked generation protocol as the SFT model (plain causal model.generate() degrades into repetition):

markdown
### Instruction:
What is the capital of France?

### Response:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "jaweed123/TinyJLLM-Instruct-DPO"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()

@torch.no_grad()
def respond(instruction, max_new_tokens=40):
    prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
    ids = tok(prompt)["input_ids"]
    L = len(ids)
    seq = ids + [tok.pad_token_id]          # dummy keeps response positions aligned
    for _ in range(max_new_tokens):
        inp = torch.tensor([seq])
        T = inp.shape[1]
        mask = torch.tril(torch.ones(1, 1, T, T, dtype=torch.bool))
        mask[:, :, L:, :] = False           # response rows ...
        mask[:, :, L:, :L] = True           # ... see the prompt only
        nxt = int(model(inp, attention_mask=mask).logits[0, -1].argmax())
        if nxt == tok.eos_token_id:
            break
        seq.append(nxt)
    return tok.decode(seq[L + 1:])

print(respond("What is the capital of France?"))

For sampling (temperature 0.8, top-k 50, top-p 0.95, repetition penalty 1.15 — the values chosen by the project's sampling sweep), use the project's learnllm.inference.generate.generate(..., response_mask_start=...).

Training details

Table

Base / reference	TinyJLLM-Instruct (frozen reference = same weights)
Dataset	32,648 pairs: Orca DPO (Apache-2.0) + HH-RLHF (MIT, filtered to single-turn answer-like pairs), 2 epochs
Beta / LR	0.2 / 3e-6, warmup + cosine
Compute trick	reference log-probabilities precomputed once (halves training compute)
Batch / steps	32 pairs effective, 3,882 steps
Hardware	RTX 4060 8 GB, ~2 h

Two earlier DPO runs (unfiltered data, lower LR) and the reasoning for the final recipe are documented in DPO_TRAINING.md, including a data-integrity incident that was found and fixed.

Limitations

Still a 100M model. DPO improves preference ranking and answer shape; it does not add knowledge. Math and code remain unreliable.
Multi-turn conversations fail. The model echoes earlier context instead of answering the last turn (measured; see the repository docs).
Safety tuning is minimal — a small, mostly-English preference set. Do not rely on this model for safety-critical filtering.

Citation

bibtex
@misc{tinyjllm,
  title  = {TinyJLLM: A 100M-Parameter Small Language Model Built From Scratch},
  author = {Jaweed, Abdul},
  year   = {2026},
  url    = {https://github.com/Abdul-Jaweed/TinyLLM}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

jaweed123

Model Tree

Base

jaweed123/TinyJLLM-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities