An Investigative Study on Recent Sharpness- and Flatness-Based Optimizers for Enhanced Self-Supervised Speaker Verification

Flat minima have recently proven effective for improving generalization and robustness against label noise. While Adam and Stochastic gradient descent (SGD) remain the most popular optimizers in deep learning, other optimizers such as Sharpness-Aware Minimization (SAM), simultaneously minimizing loss value and minimizing loss sharpness and/or improving flatness, have proven to be more effective across a variety of tasks. Motivated by this, we present the first comprehensive study of general-purpose optimizers (ADOPT, AdEMAMix) and sharpness-aware variants (SAM, ASAM, GAM, GSAM) for the supervised and self-supervised speaker verification tasks. We also explore different regularization methods such as weight decay, EMA, and SWITCH EMA to enhance robustness and learn flat optima without extra optimization costs. Our experimental results highlight the significant impact of the optimizer choice on generalization, achieving state-of-the-art performance in self-supervised speaker verification.