Model
Table | |
|---|
| Parameters | 20,405,376 |
| Architecture | Llama-style decoder (RMSNorm, SwiGLU, RoPE, tied embeddings) |
| Layers / hidden / MLP | 9 / 384 / 1152 |
| Attention heads | 6 (head dim 64) |
| Context | 1,024 tokens |
| Vocabulary | 8,192 (byte-level BPE) |
| Language | English |
Files
model.safetensors, config.json, tokenizer files: Hugging Face transformers format (LlamaForCausalLM).
tavik.gguf: GGUF (f16) for Ollama and llama.cpp.
Modelfile: Ollama model file with the chat format and stop tokens.
Usage
Ollama
Download tavik.gguf and Modelfile into one folder, then:
ollama create tavik-1 -f Modelfileollama run tavik-1
It is small enough for a Raspberry Pi 4 or 5.
from transformers import AutoTokenizer, AutoModelForCausalLM tok = AutoTokenizer.from_pretrained("Convergent-ch/Tavik-1")model = AutoModelForCausalLM.from_pretrained("Convergent-ch/Tavik-1") ids = tok.apply_chat_template([{"role": "user", "content": "What is your name?"}], add_generation_prompt=True, return_tensors="pt")out = model.generate(ids, max_new_tokens=60, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))# My name is Tavik. I'm a small AI language model made by Convergent.
<|bos|><|user|>Hello!<|end|><|assistant|>
Generation stops at <|end|>. Temperatures around 0.3 with a light repetition penalty work best.
Examples
Real outputs (temperature 0.3):
User: Hi! Who are you?
Tavik: I'm Tavik, a small AI made by Convergent. I like to chat and explain simple things!
User: What is the capital of France?
Tavik: The capital of France is Paris. It is a city known for its historical landmarks, cultural institutions, and cultural institutions.
User: I feel a bit lonely today.
Tavik: That's okay, I'm here to chat and explain things. Let me know if you need help with anything!
User: Why is the sky blue?
Tavik: The sky blue is red because it's made up of lots of colors. When sunlight hits the sky, it creates a brightness that's almost like a sunshine.
The last one shows the typical failure: fluent, but wrong.
Training
Three stages, all on one RTX 3060 Ti (8 GB) with MakeAI, bf16, AdamW, cosine schedule, 131,072 tokens per step:
- Pretraining from scratch on 355M tokens of English educational web pages from FineWeb-Edu (sample-10BT, first shard, pages of 150-3,000 words). 6,000 steps (about 786M tokens, 2.2 epochs), learning rate 1.5e-3. 2 h 53 min. Validation loss 3.09.
- Chat training on 586K conversations (279M tokens), loss only on the assistant's replies. 3,000 steps (about 1.4 epochs), learning rate 5e-4. 1 h 23 min. Validation loss 1.82.
- Identity tuning on 37K conversations: about 1,000 chats about Tavik written locally by Qwen3 8B (Apache 2.0) and 4,000 template chats, mixed with 30K ordinary chats so nothing is forgotten. 450 steps at 32,768 tokens per step, learning rate 1e-4. Final validation loss 1.76 (perplexity 5.8) on held-out chats.
Chat data:
- Smol-SmolTalk (Apache 2.0), all four training shards, without code and LaTeX. Parts of it were generated with Llama 3.1 by the dataset authors.
- TinyDialogues (MIT): only one-on-one conversations between a child and one other person, the child as the user. Generated with GPT-4 by the dataset authors.
- Everyday Conversations (Apache 2.0): generated with Llama 3.1 70B Instruct by the dataset authors.
Limitations
- Facts are unreliable. It often sounds confident and is wrong.
- Lists and advice frequently contain invented or nonsensical items.
- No math, no code, no reasoning over several steps.
- English only, context of 1,024 tokens.
- No safety tuning. Do not use it in products or for advice of any kind.
License
Apache 2.0.