RefineSeg: Dual Coarse-to-Fine Learning for Medical Image Segmentation

High-quality pixel-level annotations of medical images are essential for supervised segmentation tasks, but obtaining such annotations is costly and requires medical expertise. To address this challenge, We propose a novel segmentation framework that relies entirely on coarse annotations, encompassing both target and complementary labels, despite their inherent noise. The framework works by introducing transition matrices in order to model the noise in the coarse annotations. By jointly training on multiple sets of annotations, it progressively refines the network’s outputs and infers the true segmentation distribution, achieving a robust approximation of precise annotations through matrix-based modeling. To validate the flexibility and effectiveness of the proposed method, we demonstrate the results on two public cardiac imaging datasets, ACDC and MSCMRseg, and further evaluate its performance on the UK Biobank dataset. Experimental results indicate that our method achieves segmentation performance that surpasses state-of-the-art weakly supervised methods and closely matches fully supervised approach. Moreover, our method offers a promising pathway for making it feasible to train large medical segmentation models, e.g., MedSAM \cite{medsam}, with minimal manual labeling effort while maintaining high performance.