A power law is a mathematical relationship between two quantities where one variable changes as a power function of another, typically expressed as y = a × x^p, where a is a constant coefficient and p is the exponent determining the relationship's strength. In this formulation, the output y scales predictably with changes in the input x, creating characteristic nonlinear relationships that appear linear when plotted on logarithmic scales. Power laws describe phenomena across numerous domains, from physics and economics to machine learning and artificial intelligence.

The fundamental power law equation y = a × x^p encodes the core scaling relationship. When p is positive, increases in x produce accelerating growth in y (superlinear scaling). When p is negative, the relationship becomes inverse—larger values of x produce diminishing returns in y, approaching asymptotic limits ¹⁾

In logarithmic space, power laws transform into linear relationships: log(y) = log(a) + p × log(x). This property enables straightforward identification and analysis of power law behavior through log-log plots, where genuine power laws appear as straight lines. The slope of this line directly corresponds to the exponent p, facilitating empirical measurement and validation of scaling behavior ²⁾

Large language model performance exhibits inverse power law scaling characteristics, where model quality improves predictably but with diminishing returns as scale increases. In this context, the exponent p is typically negative, meaning that doubling computational resources, training data, or model parameters produces progressively smaller improvements in downstream task performance. The relationship can be expressed as performance ≈ a × N^(-p), where N represents scale metrics such as parameter count or training tokens.

This scaling behavior creates practical constraints on improvement strategies. Early scaling phases yield substantial quality gains per unit of additional resources, but continued scaling requires exponentially greater investments to achieve linear improvements in performance metrics. The inverse power law relationship means that while models continue improving with scale, the cost-effectiveness of achieving marginal gains decreases substantially ³⁾

The empirical observation of power law scaling in language models has profound implications for model development strategies. Rather than assuming linear relationships between resources and performance, practitioners must account for the exponential difficulty in pushing performance beyond certain thresholds. This reality influences decisions about optimal model size, training duration, data requirements, and overall resource allocation ⁴⁾

Power law relationships enable predictive modeling of language model capabilities before full-scale training. By training smaller models and measuring their performance across multiple scales, researchers can extrapolate expected performance at larger scales using power law functions. This approach reduces computational requirements for capability forecasting while informing decisions about whether proposed scaling efforts justify their resource costs.

The scaling law framework also guides allocation decisions between different scaling dimensions. Research demonstrates that optimal training efficiency typically requires balanced scaling across model size, dataset size, and training compute, rather than maximizing any single dimension ⁵⁾

Understanding power law behavior informs architectural and methodological choices. When improvements from raw scale show diminishing returns, alternative approaches become more cost-effective: improved training procedures, architectural innovations, or specialized techniques may yield greater gains per unit resource than simple scaling.

Power law models assume relatively stable relationships across the scales being considered, but real systems may exhibit different scaling characteristics in different regimes. Extrapolating far beyond observed data ranges risks prediction errors when underlying assumptions change. Additionally, power laws describe average trends rather than capturing variability or anomalous cases, and specific implementation details can significantly affect actual scaling behavior independent of the mathematical relationship.

The measurement and validation of power laws in language models requires careful experimental design, sufficient data points across multiple scales, and appropriate statistical techniques to avoid spurious correlation identification. Real-world confounding factors, data heterogeneity, and training instabilities can obscure or distort the underlying power law relationships.

• Scaling Laws