Update scale at the active optimizer step.
A learning-rate schedule is a time-varying update budget. Compare warmup, constant, linear, and cosine schedules while watching cumulative movement and the final low-rate refinement phase.
Draw the scheduleWarmup starts cautiously while moments, activations, and distributed statistics stabilize. The peak rate drives rapid movement. Decay reduces update scale later, often helping the run settle into a useful basin. The schedule must be defined against optimizer steps, not vague wall-clock progress.
Update scale at the active optimizer step.
Optimizer updates before the peak rate.
Approximate sum of rates over the run.
How softly the schedule ends.
Large early updates can be unstable before adaptive moments or normalization statistics become representative. Warmup gradually exposes the model to the peak rate.
lr(t) = peak_lr × t / warmup_stepsAfter warmup, cosine annealing bends smoothly from peak to minimum. Linear decay removes rate at a constant pace; neither is universally best.
lr(t) = min + ½(peak−min)(1 + cos(πp))Gradient accumulation, skipped mixed-precision updates, resumed checkpoints, and changing dataloader lengths can alter the number of parameter updates. The scheduler and optimizer must agree on when a real step occurred.