GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Timur-th
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Timur-th
Model Tree
HuggingFaceTB/SmolLM-135M-Instruct
Input Modalities
Output Modalities
Supported Functionality
Домашнее задание 2 курса по LLM: алайнмент SmolLM-135M-Instruct на датасете преференсов HumanLLMs/Human-Like-DPO-Dataset: chosen-ответы в нём эмоциональные и «человечные», rejected — сухие ответы в духе «As an AI language model…».
AutoModelForSequenceClassification с линейной головой) обучена RewardTrainer из TRL.PPOTrainer из TRL (trl==1.11.0, модуль trl.experimental.ppo) с этой reward model.Данные разбиты 90/10 (train_test_split(test_size=0.1, seed=42)): 9795 пар для обучения и 1089 отложенных пар для оценки; разбиение общее для всех трёх моделей.
Policy и reference — исходная SmolLM-135M-Instruct; reward и value model инициализированы из обученной reward model. Ответ обрезается после первого <|im_end|>, награда считается на нём; ответы без EOS штрафуются.
| параметр | значение |
|---|---|
| learning rate | 5e-06 |
| total episodes | 8192 |
| rollout batch | 64 |
| PPO epochs per batch | 4 |
| KL coefficient | 0.05 |
| response length | 128 |
| sampling temperature | 0.7 |
| missing EOS penalty | 1.0 |
| clip range | 0.2 |
| precision | fp32-веса, bf16 autocast |
| training time, min | 23.1 |
| метрика | значение |
|---|---|
| reward (последние 10 обновлений) | -0.553 |
| KL к референсу | 1.602 |
| RLHF reward (reward − KL) | -0.633 |
| reward первого обновления | -0.674 |
| run | lr | kl_coef | episodes | response_len | train_scores | train_kl | rm_score | emoji_rate | emoji_per_response | ai_phrase_rate | mean_tokens | eos_rate | minutes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| base | 5e-06 | 0.050 | 8192 | 128 | -0.553 | 1.602 | -2.822 | 0.000 | 0.000 | 0.320 | 128.000 | 0.000 | 23.102 |
128 отложенных промптов, сэмплирование с temperature 0.7, до 128 новых токенов. rm_score — средняя награда reward model из этой коллекции, emoji_rate — доля ответов с эмодзи, ai_phrase_rate — доля ответов с шаблонами вроде «as an AI», mean_tokens — средняя длина, eos_rate — доля ответов, закончившихся EOS.
| model | rm_score | emoji_rate | emoji_per_response | ai_phrase_rate | mean_tokens | eos_rate |
|---|---|---|---|---|---|---|
| SFT (ref) | -3.071 | 0.000 | 0.000 | 0.352 | 125.867 | 0.031 |
| DPO · beta_0.1 | -1.844 | 0.320 | 1.320 | 0.008 | 117.375 | 0.156 |
| PPO · base | -2.822 | 0.000 | 0.000 | 0.320 | 128.000 | 0.000 |
Средняя по токенам лог-вероятность ответа (300 пар на набор): обучающая и отложенная части датасета, а также внешний tatsu-lab/alpaca.
| model | train · chosen | train · rejected | held-out · chosen | held-out · rejected | alpaca (external) |
|---|---|---|---|---|---|
| SFT (ref) | -1.901 | -1.662 | -1.956 | -1.688 | -2.215 |
| DPO · beta_0.1 | -1.802 | -2.733 | -1.848 | -2.783 | -2.422 |
| PPO · base | -1.907 | -1.673 | -1.962 | -1.699 | -2.255 |
Жадное декодирование, до 128 новых токенов.
Prompt: What's your morning routine like?
SmolLM-135M-Instruct:
I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:
Morning Routine (10-15 minutes)
- Hydrate: Drink a full glass of water or a herbal tea to start the day.
- Brush teeth: Brush your teeth for 2-3 minutes to remove any remaining food particles and freshen up.
- **Get dressed
Эта модель (PPO):
I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:
Morning Routine (10-15 minutes)
- Hydrate: Drink a full glass of water or a herbal tea (e.g., chamomile, peppermint) to start the day.
- Brush teeth: Brush your teeth for 2-3 minutes to remove any remaining food particles
Prompt: Do you have a favorite movie?
SmolLM-135M-Instruct:
What a great question! As a digital AI assistant, I don't have personal preferences, but I can provide you with some popular and influential movies that have had a significant impact on popular culture. Keep in mind that opinions on movies can vary, and opinions on movies can be subjective.
Here are some of the most influential movies of all time:
- Star Wars (1977) - The first Star Wars film, directed by George Lucas, has become a cultural phenomenon, inspiring countless adaptations, franchises, and cosplay.
- The Matrix (1999) - The
Эта модель (PPO):
What a great question! As a digital AI assistant, I don't have personal preferences, but I can provide you with some popular and influential movies that have had a significant impact on popular culture. Here are a few:
- Star Wars (1977) - The first Star Wars film, directed by George Lucas, was a groundbreaking sci-fi epic that introduced the galaxy-spanning universe to a global audience.
- The Matrix (1999) - The first film in the Matrix, directed by Joseph Heller, was a thought-provoking and visually stunning sci-fi thr
Prompt: I'm feeling stressed about my exams. Any advice?
SmolLM-135M-Instruct:
I totally understand! Exam stress can be overwhelming, but don't worry, I've got some tips to help you manage your exam anxiety:
- Create a study plan: Break down your study material into manageable chunks, and allocate specific times for each topic. Set realistic goals and deadlines to stay on track.
- Get enough sleep: Aim for 7-9 hours of sleep each night to help you feel rested and focused during the exam.
- Eat a balanced diet: Fuel your body with nutritious food, including fruits, vegetables, whole grains, and lean proteins. Avoid sugary and processed foods that
Эта модель (PPO):
I totally understand! Exam stress can be overwhelming! Here are some tips to help you manage your exam-related stress:
- Create a study plan: Break down your study material into manageable chunks, and allocate specific times for each topic or subject. Set realistic goals and deadlines to stay on track.
- Get enough sleep: Aim for 7-9 hours of sleep each night to help you feel rested and focused during the exam.
- Exercise regularly: Regular physical activity can help reduce stress and anxiety. Try to incorporate some form of exercise into your daily routine, such as a short wal…
python
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Timur-th/llm-course-hw2-ppo"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo) messages = [{"role": "user", "content": "What's your morning routine like?"}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)output = model.generate(**inputs, max_new_tokens=128, do_sample=False)print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Timur-th/llm-course-hw2-ppo"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo) messages = [{"role": "user", "content": "What's your morning routine like?"}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)output = model.generate(**inputs, max_new_tokens=128, do_sample=False)print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))