Random Reshuffling Dominates Stochastic Gradient Descent

Imagine you're trying to optimize a machine learning model using a method called Stochastic Gradient Descent (SGD). It's a classic approach that works well in theory, but when you implement it, you often run into issues that make it less effective. One common tweak is called Shuffling SGD, which involves randomly reshuffling your data before each update. This method has shown great results in practice, but many considered it just a clever trick because it lacked solid theoretical backing. The problem is that the existing theory suggested that Shuffling SGD only works under strict conditions, which limits its use in real-world scenarios. This is what's called a theoretical limitation.

What this paper does is provide a breakthrough by proving that Random Reshuffling actually beats traditional SGD in smooth convex optimization, regardless of the step size, after a finite number of epochs. This means that you can use Random Reshuffling more flexibly than previously thought, making it a more reliable choice for optimization tasks. In practical terms, if you're building models that rely on optimization, this finding could help you achieve better performance without being constrained by the earlier theoretical limitations.

The paper provides a significant theoretical advancement in understanding the convergence of Shuffling SGD with Random Reshuffling.

The claims are supported by theoretical proofs, though empirical validation could be more extensive.