NeuroConText: Contrastive Text-to-Brain Mapping for Neuroscientific Literature

Neuroscientific literature faces challenges in reliability due to limited statistical power, reproducibility issues, and inconsistent terminology. To address these challenges, we introduce NeuroConText, the first brain meta-analysis model that uses a contrastive approach to enhance the association between textual data and brain activation coordinates reported in 20K neuroscientific articles from PubMed Central. NeuroConText integrates the capabilities of recent advancements in large language models (LLM) such as Mistral-7B instead of traditional bag-of-words methods, to better capture the text semantic, and improve the association with brain activations. Our method is adapted to processing neuroscientific textual data regardless of length and generalizes well across various textual content—titles, abstracts, and full-body—. Our experiments show that NeuroConText significantly outperforms state-of-the-art methods by a threefold increase in linking text to brain activations regarding recall@10. Also, NeuroConText allows decoding brain images from text latent representations, successfully maintaining the quality of brain image reconstruction compared to the state-of-the-art.