Model Details
- Developed by: Rakesh44
- Model type: Decoder-only transformer (LlamaForCausalLM)
- Parameters: ~471M (24 layers × 1280 hidden, tied embeddings)
- Context length: 2048 tokens
- Vocabulary size: 50,257 (GPT-2 tokenizer)
- Precision: float32
- Language: English
- License: Apache 2.0
Architecture
Table with columns: Component, Value| Component | Value |
|---|
| Hidden size | 1280 |
| Layers | 24 |
| Attention heads | 16 |
| KV heads | 16 (no GQA — full multi-head attention) |
| Head dim | 80 |
| Intermediate (MLP) size | 3456 |
| Activation | SiLU |
| Norm | RMSNorm (eps 1e-5) |
| Positional encoding | RoPE (theta 10000) |
| Attention bias | None |
| Tied embeddings | Yes |
| Max position embeddings | 2048 |
Scope
Odyssey is a base language model pretrained on OpenWebText, so its native capability is fluent English text continuation rather than instruction following. Out of the box it is suited to simple language tasks: text completion, short-form generation, style and tone continuation, and serving as a foundation for further training.
Its main value is as a starting point for fine-tuning. Because it has already learned general English structure and web-scale language patterns, it can be adapted to focused downstream tasks with a relatively small labeled dataset. Suitable fine-tuning targets include:
- Financial sentiment analysis — classifying the sentiment of financial text such as earnings reports, filings, or analyst notes.
- Market sentiment analysis — gauging bullish, bearish, or neutral tone in news headlines, social posts, or commentary.
- General text classification — topic tagging, intent detection, and similar sentence- or document-level labeling tasks.
- Domain-specific text generation — adapting the model's output style to a narrow domain after task-specific tuning.
Out of scope
Not suitable for high-stakes, factual-authority, or safety-critical use. As a small model, factual reliability and reasoning depth are limited.
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Rakesh44/odyssey")
model = AutoModelForCausalLM.from_pretrained("Rakesh44/odyssey")
inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training
- Data: OpenWebText (~8M documents, ~40GB)
- Tokenizer: GPT-2 BPE (50,257 tokens)
- Hardware: 1× RTX 4090
- Hyperparameters:
- Training time: 100 hours
Limitations and Bias
Odyssey is a small model and may produce inaccurate, repetitive, or biased text reflecting its training data. Verify outputs before use. The 2048-token context and float32 weights mean memory use is higher than a quantized deployment would require; consider converting to bf16/fp16 for inference.