Multiple comparison procedure

Home > Statistics Every Writer Should Know > The Stats Board > Discusssion

Multiple comparison procedure
Message posted by Kathy on July 30, 2000 at 12:00 AM (ET)

Did anyone hear about the multiple comparison procedure before? Can anyone explain the meaning to me? As I know, the multiple comparison problem would occur when we make many comparisons involving the same means. Now, i have one example, I am not sure whether there is multiple comparison problem in such case or not. That is, one group using "Learning method 1" to perform test 1, test 2, test 3, and test 4. Then, i want to test whether there is mean difference between test 1 and test 2, mean difference between test 1 and test 3, mean difference between test 1 and test 4 etc. Is the multiple comparison problem occur? How can I overcome this problem? Should i divide the alpha (significance level) by the no. of tests I perform in order to overcome the multiple comparison problem? Thanks for answering my question.

READERS RESPOND:
(In chronological order. Most recent at the bottom.)

Re: Multiple comparison procedure
Message posted by nancy diehl on July 31, 2000 at 12:00 AM (ET)

Multiple comparison tests are applied when the variable under analysis has more
than two levels (treatments). If there are only two levels then you would have
only one comparison to make and you would use the t-test. However, what would
you do if you had 5 levels? You would have 10 t-tests (5 levels comparing 2 at
a time = 10) to perform and with a .05 significance level the chances are that
you are going to find at least one with a significant difference. So you want
to perform some type of multiple comparison test where the "family-wise" error
rate is .05 rather than the "pair-wise" error rate of .05. By family-wise I
mean that no matter how many multiple comparisons you make, the entire group
error rate (chance of drawing an incorrect conclusion) is no more than 5%.

There are many different multiple comparison tests out there: Bonferroni,
Tukey, LSD, Duncan, Scheffe, etc. You would need to read up on which one
appears to be suitable to your problem. But in your example of 4 tests to
compare for differences, if the ANOVA results in significance, then this needs
to be followed with a multiple comparison test to see which pairs of tests are
different from each other.

Your $5 contribution helps cover part the $500 annual cost of keeping this site online.