Towards Early Prediction of Self-Supervised Speech Model Performance

In Self-Supervised Learning (SSL), pre-training and evaluation are resource intensive. In the speech domain, current indicators of the quality of SSL models during pre-training, such as the loss, do not correlate well with downstream performance. Consequently, it is often difficult to gauge the final downstream performance in a cost efficient manner during pre-training. In this work, we propose unsupervised efficient methods that give insights into the pre-training quality of SSL speech models, namely, measuring the cluster quality and rank of the embeddings produced by the SSL model. Results show that measures of cluster quality and rank correlate better with downstream performance than the pre-training loss, reducing the need for GPU hours and labeled data in SSL model evaluation.