Common fallacies in applying hypothesis testing

Although the theory of hypothesis testing is well developed and has a long history of application, practical application of hypothesis testing is plagued with fallacies, confusions, misconceptions, misuses, and abuses. This paper addresses four of the most widespread ones, particularly in statistical processing of signals, data, and information in uncertainty. They concern the decision on a single hypothesis, the assignment of the null hypothesis in the Neyman-Pearson framework, the confusion between two classes of significance tests, and the interpretation of a hypothesis not rejected. We articulate the principle underlying the tests, explain and analyze where the fallacies and confusions arise from, and present convincing arguments, clear conclusions, and succinct guidelines for practice, along with detailed, representative examples.