The Institute for Statistics and Mathematics is pleased to invite you to the next research seminar, taking place on campus:

Thomas Nagler (Department of Statistics, LMU Munich; Munich Center for Machine Learning (MCML))
The Surprising Effect of Reshuffling the Data During Hyperparameter Tuning
Wednesday, January 22, 2025, 17:30, in Building D4, Room D4.0.127.

Abstract:
Tuning parameter selection is crucial for optimizing predictive power of statistical and machine learning models models. The standard protocol evaluates various parameter configurations using a resampling estimate of the generalization error to guide optimization and select a final parameter configuration. Without much evidence, paired resampling splits, i.e., either a fixed train-validation split or a fixed cross-validation scheme, are often recommended. We show that, surprisingly, reshuffling the splits for every configuration often improves the final model's generalization performance on unseen data. Our theoretical analysis explains how reshuffling affects the asymptotic behavior of the validation loss surface and provides a bound on the expected regret in the limiting regime. This bound connects the potential benefits of reshuffling to the signal and noise characteristics of the underlying optimization problem. We confirm our theoretical results in a controlled simulation study and demonstrate the practical usefulness of reshuffling in a large-scale, realistic hyperparameter optimization experiment.

We aim to stream all on-campus talks via Zoom. A direct link to the stream will be posted on our website.

For further information and the seminar schedule, please see:
www.wu.ac.at/en/statmath/research/resseminar