How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

Imagine you're trying to build a machine learning model that performs well as you increase its size and the amount of data you feed it. Traditionally, researchers have looked at how models generalize based on either the amount of data or the computational resources available, but this can be limiting. For instance, you might find that simply adding more data or more parameters doesn't always lead to better performance, especially if the model isn't structured to take advantage of them. This is where things can go wrong — you might end up with a model that overfits or underfits depending on how you scale it, which is a common issue known as generalization failure.

What this paper does is take a fresh look at the relationship between model size and data by focusing on a specific type of model — a two-layer network with regularization. By analyzing how the generalization error changes as you adjust the number of parameters and the amount of training data, the authors uncover a phase diagram that shows different scaling behaviors. They find that the generalization error can be described by power laws that depend on the data's structure, which helps clarify when a model will perform well or poorly as you scale it up.

This approach is significant because it provides a more nuanced understanding of how to balance model complexity and data size. Instead of relying on broad generalizations, builders can use these insights to make more informed decisions about how to structure their models and what data to use, potentially leading to better performance in real-world applications.

The paper introduces a new perspective on scaling laws by explicitly linking generalization error to model parameters and sample size.

The analysis is grounded in a specific model and provides a clear characterization of generalization error, though it may lack extensive empirical validation across diverse settings.