Position: Significant impact of numerical precision in scientific machine learning
Abstract
The machine learning community has focused on computational efficiency, often leveraging lower-precision formats such as FP16, rather than the standard FP32. In contrast, little attention has been paid to higher-precision formats, such as FP64, despite their critical role in scientific domains like materials science, where even small numerical differences can lead to significant inaccuracies in physicochemical properties. This need for high precision extends to the emerging field of machine learning for scientific tasks, yet it has not been thoroughly investigated. According to several studies and our experiments, models trained with FP32 and FP64 can yield different scientific conclusions, yet this discrepancy is currently underreported, indicating that numerical precision is also a critical factor in scientific machine learning, as in traditional scientific computing. This precision issue limits the potential of scientific machine learning to serve as a reliable alternative or complement to traditional scientific computing in practical research. Our position paper not only highlights these precision-related issues but also recommends reporting comparisons between FP32 and FP64 results, encouraging the release of FP64 models. We believe that these efforts can enable machine learning to contribute meaningfully to the natural sciences, ensuring both scientific reliability and practical applicability.