Paper of the DayAI & Machine LearningPreprint, not yet peer reviewed
Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
Language models are trained on data fed in batches, and how large those batches should be at each stage is usually set by rule of thumb.
Why it matters Language models can be trained on small batches for most of the run and switched to large ones late, saving data without losing quality.
The researchers used a mathematical description of how training error falls over time to find the best batch-size schedule when the amount of data is fixed. For easy tasks the batch should keep growing; for hard ones it should stay small for most of training and jump to large only near the end. After that late switch the model quickly catches up, because the noise built up from small batches fades fast. Tests on models of up to 1.1 billion parameters showed late switching beat constant and early-switch schedules while using less data.
Wang et al., arXiv cs.AI, 17 November 2026.


















































