Multiple t tests and Type I error
As a simple example, you know that there is a 0.50 probability of
obtaining heads in a coin flip. If I flip the coin four times, what is the
probability of obtaining a heads one or more times across all four flips?
For two coin flips, the probability of not obtaining at
least one heads (i.e., getting tails both times) is 0.50 × 0.50 = 0.25.
probability of one or more heads in two coin flips is 1 0.25 = 0.75.
Three-fourths of "two coin flips" will have at least one heads.
So, if I flip the coin four times, the probability of one or more
heads is 1 (0.50 × 0.50 × 0.50 × 0.50) = 1 (0.50)4 = 1 0.625 =
0.9375; you will get one or more heads in about 94% of sets of "four coin flips".
Similarly, for a statistical test (such as a t test) with α= 0.05,
if the null hypothesis
is true then the probability of not obtaining a significant result is 1
0.05 = 0.95.
Multiply 0.95 by the number of tests to calculate the probability
of not obtaining one or more significant results across all tests. For
two tests, the probability of not obtaining one or more significant results is
0.95 × 0.95 = 0.9025.
Subtract that result from 1.00 to calculate the probability of
making at least one type I error with multiple tests: 1 0.9025 = 0.0975.
Example (p. 162): You are comparing 4 groups (A, B, C, D). You
compare these six pairs (α= 0.05 for each): A vs B, B vs C, C vs D, A vs C, A vs
D, and B vs D.
Using the convenient formula (see p. 162), the probability of
not obtaining a significant result is 1 (1 0.05)6 = 0.265,
which means your chances of incorrectly rejecting the null hypothesis (a
type I error) is about 1 in 4 instead of 1 in 20!!
ANOVA compares all means simultaneously and maintains the type I
error probability at the designated level.