Model Details
- Developed by: Hadeeqa Al Islam
- Model type: Causal Language Model
- Language: English
- Finetuned from: TinyLlama/TinyLlama-1.1B-Chat-v1.0
- Training data: yahma/alpaca-cleaned (20,000 samples)
- LoRA config: r=8, lora_alpha=16, target_modules=[q_proj, v_proj]
- Training: lr=2e-4, batch=4, epochs=2, fp16=True
How to Get Started
tokenizer = AutoTokenizer.from_pretrained("Haldi247/TinyLlama-SFT-Alpaca")
model = AutoModelForCausalLM.from_pretrained("Haldi247/TinyLlama-SFT-Alpaca")
messages = [{"role": "user", "content": "What is photosynthesis?"}]
Training Details
- Dataset: yahma/alpaca-cleaned (20,000 samples after filtering)
- Hardware: NVIDIA RTX 5070 Ti (16GB VRAM) via WSL2
- Training time: ~14 minutes
Training Procedure
Training Hyperparameters
- Training regime: fp16 mixed precision
- Learning Rate: 2e-4
- Batch Size: 4
- Epochs: 2
Evaluation
Testing Data, Factors & Metrics
Metrics
- BLEU Score: Used to evaluate the overlap of n-grams between the model output and the ground truth.
- BERTScore: Used to compute semantic similarity between the generated text and reference text using BERT embeddings.
Results
- Average BLEU Score: 0.4303
- Average BERTScore: 0.7236
Limitations
- Tokenization Constraints: The training process used sequence packing to manage memory effectively. Because the model was trained with packed sequences without special tokens (to prevent cross-contamination), it may exhibit formatting issues if the inference environment does not strictly adhere to the expected chat template.
- Data Bias: The model is fine-tuned on the Alpaca dataset, which may inherit biases present in the synthetic instruction data.
- Inference Stability: Due to the aggressive packing strategy, users may observe repetition or formatting artifacts if the prompt structure deviates significantly from the training format.
Summary
Model Examination [optional]
[More Information Needed]
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: [More Information Needed]
- Hours used: [More Information Needed]
- Cloud Provider: [More Information Needed]
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
[More Information Needed]
Compute Infrastructure
[More Information Needed]
Hardware
[More Information Needed]
Software
[More Information Needed]
Citation [optional]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
Glossary [optional]
[More Information Needed]
[More Information Needed]
Model Card Authors [optional]
[More Information Needed]
[More Information Needed]