Multiple comparison

Multiple comparisons occur when a researcher increases the probability of a Type I error by performing several statistical tests on one dataset. This bias is most evident in post-hoc testing after performing relational tests like ANOVA and regression.

Also, many researchers conduct numerous statistical tests on a particular dataset to find some significance. This process is also classified as unethical.

If multiple comparisons are performed on a dataset, the statistical significance level must be duly adjusted based on the number of tests performed using the Bonferroni correction method. The False Discovery Rate (FDR control) correction can also adjust the p-value given the multiple comparisons.

1. Bonferroni Correction: The Bonferroni correction adjusts the significance level (α) to account for the number of comparisons (m). The corrected significance level is calculated as α_Bonferroni​=mα​. Therefore, if 10 comparisons are made and the original significance rate of rejection is 5%, the adjusted significance level becomes 0.005.

Despite its effectiveness in guarding against type I errors, the Bonferroni correction can also reduce sample power.

2. False Discovery Rate (FDR) Correction: The FDR method controls the expected proportion of false positives among the rejected hypotheses. It is less conservative than the Bonferroni method and retains more statistical power. The FDR procedure involves ranking the p-values in ascending order and finding the largest p-value. All p-values less than or equal to the largest rank are considered significant. This method balances the need to identify true effects while controlling the rate of false discoveries.

Test your knowledge by filling in the blanks below!