Limitation — Read Before Using
This model has 0% recall on toxic content. It was trained on an
imbalanced dataset (4.4% toxic examples) and learned to always predict
"not toxic" regardless of input. It will not flag actual toxic content.
This is documented honestly rather than hidden — see Evaluation Results
below for the full diagnosis. Do not use this model for actual content
moderation in its current state.
Training Details
- Base model: unsloth/mistral-7b-instruct-v0.3-bnb-4bit
- Method: LoRA (r=16, alpha=16), targeting q/k/v/o/gate/up/down projections
- Training data: 2,500 examples from Jigsaw Toxic Comments (111 toxic, 2,389 not toxic)
- Training steps: ~100, batch size 2, gradient accumulation 4
Evaluation Results
Tested on 20 held-out examples (no overlap with training data):
Table with columns: Metric, Score| Metric | Score |
|---|
| Accuracy | 95.0% |
| Precision | 0.0% |
| Recall | 0.0% |
| F1 Score | 0.0% |
Confusion Matrix:
What I'd Do Differently
- Balance the training set (equal toxic / not-toxic examples)
- Evaluate with precision/recall/F1 during training, not just after
- Check class balance during dataset preparation, before training starts
Why This Is Documented Instead of Hidden
A model reporting fake high accuracy while having zero real capability
is worse than no model — it creates false confidence. This card exists
so anyone using this model understands its actual (current) limitations.
Intended Use
Currently: Portfolio demonstration only.
Not suitable for production content moderation without retraining on
a balanced dataset.