Transfer learning & fine-tuning

Pre-training objectives (MLM, CLM, NSP)

Learning rate scheduling

Mixed precision training

Gradient clipping

Early stopping

Model checkpointing

Distributed training concepts