Factors Affecting Reliability in Psychological Tests

Psychologists who study psychometrics (the science of psychological test and measurements) identify two categories of factors that affect the reliability of psychological tests. The first category is factors related to the construction of the tests, these are called systemic errors and are built into the test. The second category is factors that arise from random issues related to the how the test is given or how the test is taken. These factors are called unsystematic errors and have nothing to do with the test itself.

A close-up of a black and blue paint Rorscach test. (Image: denamazanikignatencko/iStock/Getty Images)

Question Construction

Researchers construct questions on psychological tests to bring about a response on some mental quality such as depression. If test questions are difficult, confusing or ambiguous, reliability is negatively affected. Some people read the question to mean one thing, whereas others read the same question to mean something else. Errors in question construction are systemic errors and can be corrected only through research and redesign of the test.

Administration Errors

Instructions with the test may contain errors that create another type of systemic error. These errors exist in either the instructions provided to the test-taker or those given to the psychologist who is conducting the test. Instructions that interfere with accurately gathering information (such as a time limit when the measure the test is seeking has nothing to do with speed) reduce the reliability of a test.

Scoring Errors

Reliable tests have an accurate method of scoring and interpreting the results. All tests come with a set of instructions on scoring. Errors in these instructions, such as making unsupported conclusions, reduce the reliability of the test. Test construction begins with research to support the conclusions drawn--but if the research has flaws, again a system error may result.

Environmental Factors

Environmental factors such as uncomfortable room temperature, or distracting sounds are one form of unsystematic errors. The errors made by the psychologist proving a test are another type of environmental factor that can affect reliability. Although psychologists are trained in psychological testing, human error is always a possibility. The administrator's attitude toward the test-taker also influences scoring or interpretation when clinical judgment is called for in the test instructions.

Test-Taker Factors

We often think that factors related to the test-taker have an effect on reliability. However, factors related to the test-taker, such as poor sleep, feeling ill, anxious or “stressed-out” are integrated into the test itself. Solid reliable tests do not claim to give results that are the “real-score” of the test-taker--but rather a score that is a combination of both the “real-score” and the “error-score.” The real score is the test-taker's performance, and the error-score is a margin of error that is built into the test for factors such as poor sleep and anxiety. The factoring-in of the “error-score” makes it all the more important for other factors that negatively affect reliability to be low.