Introduction
Lythri-Wink-2B-pt is the pre-trained base model behind Lythri-Wink-2B. It has 2.52B total parameters and 1.48B activated parameters.
Lythri-Wink-2B-pt is structurally sheared from google/gemma-4-E2B and recovered through teacher distillation on approximately 813M tokens at an 8K training context length. It is intended for continued pre-training, supervised fine-tuning and research—not direct assistant-style conversation.
Architecture
Table with columns: Gemma4-E2B, Lythri-Wink-2B-pt | Gemma4-E2B | Lythri-Wink-2B-pt |
|---|
| Total parameters | 4.63B | 2.52B |
| Activated parameters | 2.28B | 1.48B |
| Decoder layers | 35 | 31 |
| PLE dimension | 256 | 128 |
The FFN width is reduced to approximately 60% of the original model. The language-model hidden size and attention-head structure are retained.
Distillation Recovery
At the final validation point before export (step 1500):
Table with columns: Metric, Value| Metric | Value |
|---|
| Validation KL | 0.3241 |
| Student CE | 2.552 |
| Teacher CE | 2.270 |
The released checkpoint was exported at step 1550, corresponding to approximately 812.6M training tokens.
Base-Model Benchmarks
Lythri-Wink-2B-pt is evaluated without a chat template and with a BOS token prepended, Lythri-Wink-2B-pt acheives similar intelligence compare Gemma3-1B-pt, while holding a larger knowlege scale and a more permissive license.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_path = "Lythri/Lythri-Wink-2B-pt"tokenizer = AutoTokenizer.from_pretrained(model_path)model = AutoModelForCausalLM.from_pretrained( model_path, dtype=torch.bfloat16, device_map="auto",) prompt = "Once upon a time"inputs = tokenizer( prompt, return_tensors="pt", add_special_tokens=True,).to(model.device) with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=128, do_sample=True, temperature=0.8, top_p=0.95, ) print(tokenizer.decode(output[0], skip_special_tokens=True))
Training Your Own Model
If You dislike our instruct model Lythri/Lythri-Wink-2B, you could train a friend from our pt model!
First, prepare a JSONL dataset with one text field per sample:
{"text": "Your training text goes here."}
Here's an example about the code:
from datasets import load_datasetfrom transformers import ( AutoModelForCausalLM, AutoTokenizer, DataCollatorForLanguageModeling, Trainer, TrainingArguments,)import torch model_id = "Lythri/Lythri-Wink-2B-pt" tokenizer = AutoTokenizer.from_pretrained(model_id)tokenizer.add_bos_token = Truetokenizer.pad_token = tokenizer.eos_token model = AutoModelForCausalLM.from_pretrained( model_id, dtype=torch.bfloat16, device_map="auto",)model.config.use_cache = False dataset = load_dataset("json", data_files="train.jsonl", split="train")dataset = dataset.map( lambda x: tokenizer( x["text"], truncation=True, max_length=2048, add_special_tokens=True, ), remove_columns=dataset.column_names,) trainer = Trainer( model=model, args=TrainingArguments( output_dir="./wink-2b-output", per_device_train_batch_size=1, gradient_accumulation_steps=16, learning_rate=2e-5, num_train_epochs=1, bf16=True, gradient_checkpointing=True, logging_steps=10, save_steps=200, ), train_dataset=dataset, data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),) trainer.train()trainer.save_model("./wink-2b-output")tokenizer.save_pretrained("./wink-2b-output")
Important: Keep the BOS token enabled when evaluating or fine-tuning this model.
Intended Use
- Continued pre-training
- Domain adaptation
- Supervised fine-tuning
- Research on structured pruning and distillation
For direct conversation, use Lythri/Lythri-Wink-2B.
Limitations
- This is a base model and is not instruction-tuned.
- The model inherits limitations and biases from its base model, teacher and training data.
- It can produce inaccurate or inappropriate content.
License
Lythri-Wink-2B-pt is built on Gemma 4 and released under the Apache License 2.0.
Citation
@techreport{li2026lythri, title = {Lythri Technical Report}, author = {Li, Jiawen}, year = {2026}, institution = {Zenodo}, doi = {10.5281/zenodo.23179311}, url = {https://doi.org/10.5281/zenodo.23179311}}
Support
The whole training process is self-funded. If you like our model, please click a free like — that means a lot to me as a student!