A Strategy for Ranking Optimization Methods using Multiple Criteria

Many methods for optimizing black-box functions exist, and many metrics exist for judging the performance of a specific optimization method. There is not, however, a generally agreed upon strategy for simultaneously comparing the performance of multiple optimization methods for multiple performance metrics across a range of optimization problems. This paper proposes such a methodology, which uses nonparametric statistical tests to convert the metrics recorded for each problem into a partial ranking of optimization methods; these partial rankings are then amalgamated through a voting mechanism to generate a final score for each optimization method. Mathematical analysis is provided to motivate decisions within this strategy, and numerical results are provided to demonstrate the potential insights afforded thereby.