Overview
Merged student model for the Kaggle LLM Classification Finetuning task.
This model was trained with a custom mixed loss over A/B/C next-token logits: hard-label CE plus soft-label CE.
It is intended to preserve winner classification stability while improving probability calibration for multi-class log loss.
Base Model
unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
Training Data
- Distilled source:
safe_plus_filtered_plus_hard_plus_raw_partial
- Raw Kaggle train mix mode:
partial
- Hard loss weight:
0.8
- Soft loss weight:
0.2
- Soft label source:
blended
- Soft label temperature:
1.0
- Soft label floor:
0.005
- Raw label smoothing:
0.04
- Prompt contains
Prompt / Response A / Response B / Winner:
- Teacher rationale is not injected into prompt-side input
- Loss is computed on the next-token logits for
A, B, and C
- Hard CE uses
target_label_abc / raw gold label
- Soft CE uses
blended_soft_label / smoothed raw gold label
C means tie
- Adapter repo:
tussiiiii/llmcmp-distill-llama3-8b-lora-v6h-hard-soft-loss-adapter