Reliability of an Instrument
In plain language, reliability is the extent to which your instrument produces the same results time after time, indicating that the majority of your sample interpreted the questions in about the same way. Reliability coefficients range from 0-1, and you are highly unlikely to ever get perfect reliability (1). Acceptable reliability is .70 or better. You might get away with something like .68, but it is better for your research results if you strive to make the instrument more reliable than this during the pilot stage.
Reliability reflects random error, and there is always some amount of random error in an instrument. What is random error? Things like ambiguous instructions, mistakes in coding, mood of the subjects at the time they filled out the survey all contribute to random error. Reliability tells us how much random error is in our survey. The higher our reliability coefficient, for example .90, the lower our random error.
There are four common ways to check for reliability. The test-retest method involves giving your new instrument to the same sample of people twice, usually 2-4 weeks apart. Then, you would correlate the scores from both times and the resulting correlation is your reliability coefficient. This sounds fairly easy to do, but there are problems with assessing reliability in this way. For one thing, most people will remember how they answered the questions the first time, and will tend to answer them the same way from memory. This can falsely inflate your reliability coefficient. Another problem may be due to “mortality”. This is where you can’t get some of your original participants to fill out the questions the second time (maybe they leave the area and you can’t find them again). A third problem is when the instrument itself influences the way people answer the second time. For example, just answering the questions the first time about how satisfied they are with their counselor can easily make them think about their satisfaction too much, which will influence how they answer the second time around. In this case, you aren’t measuring true reliability, but instead the reflection of attitude change.
Another type is the alternate-form method, often used in educational settings. This is where you simply create two forms of the same instrument and pass out form one the first time, and form two the second time. Then you correlate the scores on the two versions and the result is your reliability coefficient. This solves the problems of human memory and influence of attitude that test-retest had, but this method also has its problems. Mainly, you can never be fully certain that the two forms of the instrument are equal to each other. This is similar to the third method, the split-halves method, where half the instrument is given to one group of people and the other half is given to another group. The correlation between the two halves gives you your reliability coefficient. However, since this is really reliability for only half the test, a mathematical adjustment (the Spearman-Brown prophecy formula) must be made to get the full reliability. The main problem with this method is that there are too many ways to split the instrument in two. You can do odd-even, or first half vs. last half, or any other way. Different ways of splitting the test can lead to different reliability coefficients, and how do you know which one is correct? You don’t.
The fourth method is that of internal consistency. With this method, every question is correlated with every other question (known as inter-item correlations) to give a coefficient known as Cronbach’s alpha coefficient, which ranges from 0-1. Though you will often see some of the other methods of reliability estimation reported in research, this one is commonly accepted as preferred because it doesn’t require giving the questionnaire twice, nor splitting up the items, and it isn’t influenced by human memory.
A reliable instrument is important so that you can be assured that you are measuring your construct in a consistent way. Come back next week for information on establishing validity of your instrument. In the mean time, eat a cookie 🙂