The
confidence level is a number that acts as a threshold for a
surprising
P-value and as a cut-off for determining when two groups
have a difference in proportions.
The P-value is a numeric probability of seeing the difference in proportions under the assumption there isn't a real difference in the proportions between the two groups. With a confidence level of 95%, any P-value less than or equal to 0.05 is considered surprising. A small P-value means it is unlikely to occur if the variations between group proportions was random. If there is a difference because of random chance it would be surprising enough to believe it is wrong. So, it can be concluded there is a true difference in the proportions between two groups.
Example
In an example where the confidence level is set to 95%, that means any
P-value that is less than 0.05 (1 - 0.95 = 0.05), is considered
surprising
, and we would reject the assumption there is no
difference between proportions and instead say there is an actual difference.
The confidence level sets what is the acceptable probability of getting a false positive, when there is no real difference between groups.
Example
Using the same example where the confidence level is 95%, it indicates we can accepting false positives 5% of the time in a situation where there is no real difference between two group proportion.
- By default, the confidence level is set at 95%, but you can adjust the confidence level based on your needs.
- Using a lower confidence level may result in more differences being identified, but this can lead to more false positives among the differences that are identified.