Evaluating the Role of Great Pre-trained Diffusion Models in Few-shot Phase: Warm-up and Acceleration

Due to the customized requirements, few-shot diffusion models have attracted much attention. However, only a few works analyze few-shot models, and none involve the fast few-shot optimization process, which is important for quickly responding to users. In this work, we evaluate the role of each operation in the optimization process and prove the convergence guarantee for few-shot diffusion models. A standard operation for the few-shot model is only fine-tuning some key parameters to avoid overfitting the limited target dataset. We first show that this operation is insufficient from empirical and theoretical perspectives. Empirically, we conduct real-world few-shot fine-tuning experiments with underfitting and overfitting bad pre-trained models and show that the results are heavily influenced by these bad models. Theoretically, we also prove that the few-shot phase can not learn the ground-truth parameters and suffers from a small gradient when using a bad pre-trained model. Based on these results, we highlight the importance of a great pre-trained model by showing it can warm up few-shot models and lead to a strongly convex landscape for few-shot diffusion models. As a result, the few-shot model fast converges to the ground-truth parameters. In contrast, we show that with a bad initialization, the pretraining phase requires large optimization steps to converge.