Training
- Base revision:
0c351dd01ed87e9c1b53cbc748cba10e6187ff3b.
- 173,553 cached training examples; 2,712 optimizer steps; one epoch.
- 32 GPUs; per-device batch 1; gradient accumulation 2; global batch 64.
- Learning rate 1e-5; cosine scheduler; warmup ratio 0.1; seed 42.
- BF16, FlashAttention 2, DeepSpeed ZeRO-3; sequence limit 11,000 tokens.
- AURA answer selection and silent-token weighting are preserved. Focal
modulation applies to all retained supervised tokens, with the original
weighted-token denominator.
- Visual patch embedding, transformer blocks, deepstack mergers and projector
are frozen.
model.visual.pos_embed.weight remains trainable to match the CE
baseline. The language model and LM head are trainable.
Mean training Focal loss: 0.493920. Final logged step loss: 0.3440.
Focal loss values cannot be directly compared with CE values as model-quality scores.

Loading
Tested with Transformers 4.57.1 and PyTorch 2.6.0.
import torchfrom transformers import AutoModelForCausalLM, AutoProcessor model_id = "leobo09263013/focal"processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto")
The checkpoint retains the custom <|silent|> token (ID 151669). The saved
Fast tokenizer removes it when skip_special_tokens=True; account for that
when evaluating silence decisions.
For continued training, Focal is a runtime loss hook and is not restored by
loading model weights alone. Reattach it from training/aura_loss.py using
attach_aura_loss(model, processor.tokenizer, "focal", focal_gamma=2.0).
The original CE model implementation is retained in the custom model file.
This export does not contain the distributed optimizer state.