BEiTScore++: Reference-free Image Captioning Evaluation with BEiT-3 and Conformalized Quantile Regression
Abstract
AbstractReliable uncertainty estimation is important for deploying evaluation metrics in real-world settings, where both input ambiguity and domain shifts are common. In this work, we propose an uncertainty-aware extension of the BEiTScore image captioning metric, producing calibrated confidence intervals alongside alignment scores. Our approach models predictive distributions over human judgments, enabling both accurate scoring and principled uncertainty quantification. We evaluate our BEiTScore++ metric across multiple benchmarks with human-annotated ratings, demonstrating strong performance in terms of ranking correlation, calibration, and interval sharpness. BEiTScore++ not only produces accurate predictions, but also captures aleatoric uncertainty arising from ambiguous inputs, and epistemic uncertainty due to limited model knowledge, making it a reliable tool for uncertainty-aware evaluation of image-caption alignment.