A recent paper reported by arXiv cs.CL investigates whether Step Law applies to small-scale language models below 59M parameters. Step Law provides power-law formulas to determine the optimal peak learning rate and batch size for pre-training language models. Originally, the law was calibrated on models ranging from 59M to 1B parameters, leaving the smaller regime untested by its authors.
Why the Small-Model Regime Matters
This sub-59M parameter regime is critical for several practical applications. It directly impacts single-GPU training workflows, interpretability research, and educational experiments. Furthermore, it matters for settings where deploying larger models is restricted due to strict memory or cost limitations. The new empirical recalibration tests three specific outcomes: H1, where the original coefficients function directly, and H2, where the power-law form holds with different parameters.
Unknowns and What Remains to Be Seen
The available information does not specify the final results of testing H1 and H2, nor does it detail the exact third outcome considered in the study. Additionally, the announcement did not specify the complete set of coefficient adjustments required if the original formulas fail to transfer directly.
Future work will need to clarify how reliably hyperparameter scaling laws translate downward into resource-constrained environments.
Lead image: Nicolas Foster / Pexels
