RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Mining

Developing advanced medical imaging retrieval systems is significantly challenged, since the concept of ‘similar images’ varies significantly across different medical contexts and perspectives, resulting in a pressing lack of large-scale, high-quality medical imaging retrieval datasets and benchmarks. In this paper, we propose a novel methodology that leverages dense radiology reports to define image-wise similarity ordering at multiple granularities in a scalable and fully automatic way. We build up two comprehensive medical imaging retrieval datasets for Chest X-ray and CT, MIMIC-IR and CTRATE-IR, with detailed image-image ranking ordering annotations conditioned on diverse anatomical structures. Lastly, we develop two retrieval systems, namely, RadIR-CXR and RadIR-ChestCT, which consistently show superior performance in traditional image-image and image-report retrieval tasks, and further enable flexible and effective image retrieval conditioned on specific anatomical structures in text form, achieving state-of-the-art results on 77 out of 78 metrics.