Who to Trust? Aggregating Client Predictions in Federated Distillation.

Federated Distillation enables distributed learning for clients with heterogeneous model architectures. In this paradigm, the server and clients exchange predictions on a shared unlabeled public dataset, rather than model parameters or gradients. % However, under data heterogeneity (e.g., class mismatch), clients produce unreliable predictions for instances from unfamiliar classes. % An equally weighted combination of such predictions corrupts the teacher signal used for distillation. % In this paper, we theoretically analyze Federated Distillation and show that aggregating client predictions on a shared public dataset converges to a neighborhood of the optimum, with the neighborhood size controlled by the aggregation quality. % We propose two uncertainty-aware aggregation methods, $\textbf{UWA}$ and $\textbf{sUWA}$, that use density-based estimates to down-weight unreliable client predictions. % Experiments on image and text classification datasets confirm that our methods are most effective under high data heterogeneity, while matching standard averaging when heterogeneity is low.