The Machine Learning Advent Calendar Bonus 2: Gradient Descent Variants in Excel

Gradient Descent, along with its variants Momentum, RMSProp, and Adam, share the same optimization goal of reaching the minimum of a loss function, but they differ in their approaches to navigating the parameter space. Each successive method introduces mechanisms to address limitations of the previous algorithmssuch as improving convergence speed, stability, or adaptivenessresulting in more efficient and smarter updates during training. These enhancements do not alter the ultimate target but optimize the path taken to reach it, making the training process more robust and effective. The evolution from basic Gradient Descent to Adam exemplifies how incremental improvements in optimization