A fused score for ranking and evaluation using multiple performance metrics

Performance evaluation and ranking based on multiple performance metrics are required in many fields. These metrics have commonality since they evaluate the same thing despite in different aspects. If the performance is deemed good in all metrics, the overall performance is indeed good. In view of this, making full use of this commonality should be good for ranking and evaluation. Voting theory deals with problems that are quite similar to ours if we treat algorithms and metrics, respectively, as candidates and voters. Many criteria have been proposed in voting theory to justify voting methods. In this paper, we propose a new score, named comprehensive score of performance (CSP) considering these criteria. Its main idea is to measure the difference between the cumulative distribution functions of the perfect algorithm and the one to be evaluated. CSP fuses multiple metrics by considering their commonality. Also, it fuses preference information and numerical information. Ranking should rely mainly on preference information and evaluation should rely mostly on numerical information. Thus CSP is suitable for both evaluation and ranking. Moreover, CSP can comprehensively reflect the statistic characteristics of the performance and is also monotonic, consistent, and transitive. Two illustrative examples are presented to demonstrate the effectiveness and appropriateness of the CSP.