The Bonferroni correction ensures the confidence level remains high when you're comparing more than two subgroups.
When comparing two groups, a 95% confidence level means that in situations where there are no actual differences in proportions between groups, we accept a false positives 5% of the time. However, false positives at a rate greater than 5% may be accidently accepted because we are making multiple comparisons. This is the case when comparing multiple groups and every group is getting compared using a confidence level of 95%.
For example, you're comparing different age groups response to a survey:
- Age 18-24
- Age 25-34
- Age 35-44
- Age 45-54
When you run the significance tests at 95% confidence, 6 tests are run to compare the different age groups:
- Test 1: 18-24 vs 25-34
- Test 2: 18-24 vs 35-44
- Test 3: 18-24 vs 45-54
- Test 4: 25-34 vs 35-44
- Test 5: 25-34 vs 45-54
- Test 6: 35-44 vs 45-54
If we looked at each comparison in these 6 tests individually, we would have a false positive 5% of the time where there is no difference in proportions. When all comparisons are made, the overall rate of false positives would be greater than 5% when there are no differences. If you apply the Bonferroni correction, it will ensure that the confidence level across all six tests remains at 95%. When applied the Bonferroni correction may lead to fewer significance differences being identified, but it will result in fewer false positives.
- By default, the Bonferroni correction is turned off.
- As a best practice, apply the Bonferroni correction when you are testing more than two subgroups. However, if you're only interested in how two subgroups (i.e., Age 18-24 vs Age 35-44) differ then you do not need to apply the Bonferroni correction.