Introduction

In the world of hypothesis testing, one often encounters the challenge of multiple testing, where researchers perform multiple hypothesis tests on the same data set. This chapter delves into the intricacies of multiple testing, elucidating how it can lead to biased results and impact the overall reliability of findings. As we explore this phenomenon, we’ll uncover the underlying reasons behind its occurrence and delve into real-world examples to highlight its implications.

Problem of Multiple Testing

Multiple testing refers to the practice of conducting numerous hypothesis tests on the same data set without proper adjustments. While this might seem innocuous, it can result in an increased likelihood of encountering significant results purely by chance. Imagine a scenario where you flip a coin multiple times, and with each flip, you’re trying to prove whether the coin is biased towards heads or tails. With each test, there’s a chance that you might observe a biased outcome due to random chance alone. This principle extends to hypothesis testing as well.

Biased Results and Inflated Type I Error Rate

When multiple hypothesis tests are conducted on the same data set, the risk of encountering a Type I error (false positive) increases. This is because with each test, there’s a probability of incorrectly rejecting the null hypothesis when it’s actually true. As more tests are performed, the likelihood of at least one of them yielding a significant result due to chance alone rises. This can lead to drawing incorrect conclusions and presenting biased findings to stakeholders.

Example: Consider a pharmaceutical company conducting clinical trials for multiple drugs simultaneously. If each drug is tested for effectiveness without proper adjustments, the likelihood of at least one drug showing a positive effect purely by chance increases. This could lead to unwarranted enthusiasm for a drug that might not truly be effective.

Controlling the Family-Wise Error Rate (FWER)

To mitigate the issue of multiple testing and the subsequent bias, researchers use methods to control the Family-Wise Error Rate (FWER). FWER measures the probability of making at least one Type I error among all conducted tests. One such method is the Bonferroni correction, where the significance level (alpha) is divided by the number of tests. This adjustment reduces the risk of drawing erroneous conclusions due to multiple testing.