Challenges and recent advances in estimation performance evaluation

Existing methods and metrics for estimation performance evaluation have serious limitations. They are narrowly focused, overly pessimistic, hyper sensitive to evaluation scenario, totally reliant on knowing the ground truth, and may ignore crucial coupling between the estimate and estimatee. Three challenges have been articulated: development of a comprehensive metric of estimation performance, evaluation of estimation performance without the need to know the ground truth, and development of alternative methods and measures for estimation performance ranking. A general approach to comprehensive evaluation of estimation performance without the need to know the ground truth is possible. Such an approach has been developed and is outlined here. Various metrics in three classes have been proposed recently: absolute metrics, relative metrics, and frequency counts. Having distinctive pros and cons, they should take over the widely used RMSE in many cases. Three other classes of metrics have been developed: error spectrum, desirability metrics, and concentration measures. They are much more comprehensive than existing ones. The use of a common performance measure is not the only yardstick for performance ranking. Alternatives do exist and may prove superior. A viable one is to rank by outcomes of competitions among estimators. Two measures in this category have been proposed and a third class is under development.