TRACS: Telescope Reference Astronomy Categorization Dataset and Shared Task
Abstract
AbstractTo assess the scientific impact of observational facilities, Astronomers analyze publications which have made use of the data by the facility. This requires bibliographies annotating and linking data products with the literature, allowing bibliometrics to measure impact of the data. Creating bibliographies is laborious, involves specialists searching the literature for names, acronyms and identifiers, then determining how/if observations were used in those publications.These bibliographies are useful beyond metrics: from the perspective of a research scientist, the data-literature connections provide a critical path to data discovery and access. Thus, by leveraging the efforts of librarians and archivists, we can use telescope bibliographies to support the scientific inquiry process. To this end, we introduce the TRACS: Telescope Reference and Astronomy Categorization dataset and Shared task, consisting of over 89K public English texts from space telescope bibliographies, categorized using a new, simple taxonomy based on discussion with established bibliographers.We hope TRACS will be used to train modern Machine Learning and Artificial Intelligence models that in turn will help data curators build bibliographies. We evaluate how off-the-shelf Large Language Models perform on automatic bibliography curation as a baseline analysis. Results show the difficulty of the task and the need for custom tools.