Highlights
- Follows instructions 55% better than V1-Instruct on IFEval (0.129 against 0.083), clearing its registered gate by 2.56 standard errors.
- Writes some working code: 7 of 164 HumanEval problems solved and 4.2% of MBPP, where V1-Instruct solved none.
- Small: 201M parameters and a 4,096-token context. It runs comfortably on a laptop.
- Loads in Transformers with no remote code, with its chat template included.
Results
Table with columns: Task, V2-Instruct, V1-Instruct| Task | V2-Instruct | V1-Instruct |
|---|
| IFEval | 0.129 | 0.083 |
| GSM8K chat | 0.022 | 0.021 |
| HumanEval chat | 0.043 | 0.000 |
| MBPP chat | 0.042 | 0.000 |
Every benchmark, including comparisons with other chat models: results.
How it compares
LiteGPT V2 Instruct doesn't reach the top score among these chat models. What it brings is a chat model built entirely from scratch, small enough for a laptop, and open down to every run record.
Table with columns: Model, IFEval, GSM8K chat, HumanEval chat, MBPP chat| Model | IFEval | GSM8K chat | HumanEval chat | MBPP chat |
|---|
| Qwen3-0.6B | 0.564 | 0.310 | 0.372 | 0.310 |
| Gemma 3 270M-it | 0.251 | 0.099 | 0.122 | 0.154 |
| SmolLM2-135M-Instruct | 0.220 | 0.021 | 0.000 | 0.098 |
| LiteGPT V2 Instruct | 0.129 | 0.022 | 0.043 | 0.042 |
| LiteGPT V1 Instruct | 0.083 | 0.021 | 0.000 | 0.000 |
Higher is better.
Model
Use
GGUF files for llama.cpp and LM Studio (BF16, Q8_0, Q4_K_M), checked against this model: kmkrworks/LiteGPT-Llama-V2-Instruct-GGUF. Try it in your browser: LiteGPT V2 Chat.
With Transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer name = "kmkrworks/LiteGPT-Llama-V2-Instruct"tokenizer = AutoTokenizer.from_pretrained(name)model = AutoModelForCausalLM.from_pretrained(name)inputs = tokenizer.apply_chat_template([{"role": "user", "content": "What is the capital of France?"}], add_generation_prompt=True, return_tensors="pt", return_dict=True)output = model.generate(**inputs, max_new_tokens=48)print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
With LiteGPT's own engine, from a clone of the repository:
uv sync --lockeduv run litegpt chat --model LiteGPT-Llama-V2-Instruct
Benchmarks were scored through LiteGPT's engine.
Intended use: research and learning. It has no safety tuning, so add your own safeguards before putting it in front of people.
Training
3 epochs, 14,000 steps, assistant-only loss, peak learning rate 6e-4. Held-out loss fell every epoch: 1.1323, 1.0518, 1.0238. Details: training.md.
Training data
Everything in the Base card, then these chat conversations (385,677 in all, 497M tokens). Attributions: the NOTICE file.
Table with columns: Source, Dataset card, Generated by| Source | Dataset card | Generated by |
|---|
| smol-magpie-ultra | Apache-2.0 | Llama-3.1-405B-Instruct |
| smol-summarize, smol-rewrite, smol-constraints (reserved part) | Apache-2.0 | Qwen2.5-72B-Instruct |
Licence
Apache-2.0 for the weights, MIT for the code, with the NOTICE file.
Built with Llama. Built with Qwen. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Use of this model should follow the Llama Acceptable Use Policy.