Skip to main content

Understanding the Reliability of Educational and Psychological Tests


Why aren't tests reliable?

The reason tests are not reliable is that reliability is a property of the interpretation of scores not the tests themselves. 

This isn't a matter of semantics.

Think about it this way.

Give all the students in one school an achievement test. The test items don't change so they appear stable, consistent, and reliable. However, when publishers report reliability values, they calculate the reliability statistics based on scores. Scores vary from one administration to another. If you ever took a test twice and got a different score, you know what I mean. Individuals change from day to day. And we change from year to year. Also, even a representative sample of students for a nation can be different each year.


Every time we calculate a reliability statistic, the statistic is slightly different.

Reliability values vary with the sample.

Reliability values vary with the method of calculation.

Reliability values also vary with the method used for calculation. You can get high reliability values using coefficient alpha with scores from a one-time administration. This method is common in research articles. But you will see different values from the same research team in different samples in the same article.


If we use a split-half method, which usually calculates reliability based on a correlation between two halves of one test, then we can get a reliability value based on one administration. But that's only half a test! Researchers use the Spearman-Brown formula to correct for the shortened half-test problem- but that's just an estimate of what the full test could be.


There's also a test-retest reliability method. Give a test one time, wait awhile- maybe a week or several weeks, then retest. That gives you an estimate of stability. But if you have a good memory, you can score higher on the second test on some tests like intelligence and achievement.


By now you get the point. Any one test can be associated with a lot of reliability values. The reliability problem is not just about tests. The problem can be understanding that tests do not have one reliability value. As with many things in science, there are many variables to consider when answering a question.

Reputable test publishers include reliability values in their test manuals. Teachers, Counselors, Psychologists, and other users ought to know about test score reliability.

Learn more assessment and statistical concepts in


Applied Statistics: Concepts for Counselors 

AMAZON BOOKS




Learn more about assessment and statistics at the Applied Statistics website


Learn more about Creating Surveys




Quick Notes on Test Reliability

Reliability is a property of scores not tests.

Reliability may mean stability of scores over time.

Reliability may mean how consistently test questions measure whatever the test measures.

Reliable test scores in one culture do not mean they will be reliable in another culture.

Reliable test scores do not guarantee the score are valid - but reliability places a limit on validity.

Reliability statistical concepts apply to tests, quizzes, polls, surveys...sets of questions yielding numerical scores.



Note: This is a re-posting of a post to this new blog. 

Links to Connections

My Page    www.suttong.com

  

My Books  AMAZON          and             GOOGLE STORE

 

FOLLOW   FACEBOOK   Geoff W. Sutton   TWITTER  @Geoff.W.Sutton

 

PINTEREST  www.pinterest.com/GeoffWSutton

 

Articles: Academia   Geoff W Sutton   ResearchGate   Geoffrey W Sutton 

 

 
Labels:

Comments

Post a Comment

Popular posts from this blog

Student Self-Efficacy

  Assessment name:  STUDENT SELF-EFFICACY SCALE * Note. This post has been updated to provide an available measure of student self-efficacy. ———- Scale overview:  The  student self-efficacy scale i s a 10-item measure of self-efficacy. It was developed using data from university nursing students in the United States. Authors: Melodie Rowbotham and Gerdamarie Schmitz Response Type:  A four-choice rating scale as follows: 1 = not at all true 2 = hardly true 3 = moderately true 4 = exactly true   Self-efficacy is the perception that a person can act in a way to achieve a desired goal.  Scale items There are 10 items. Examples: I am confident in my ability to learn, even if I am having a bad day. If I try hard enough, I can obtain the academic goals I desire.   Psychometric properties The authors reported that their sample scores ranged from 25 to 40 with a scale mean of 34.23 ( SD  = 3.80. Internal consistency was high at alpha = .84. The authors reported the results of a principal compon
Post a Comment
Read more

Personal Self-Concept Questionnaire (PSQ)

  The Personal Self-Concept Questionnaire  ( PSQ )   Overview The Personal Self-Concept Questionnaire (PSQ) measures self-concept based on ratings of 18 items, which are grouped into four categories: Self-fulfilment, autonomy, honesty, and emotional self-concept. Subscales : The PSQ has four subscales 1. Self-fulfilment (6 items) 2. Autonomy (4 items) 3. Honesty (3 items) 4. Emotional self-concept (5 items)  ðŸ‘‰ [ Read more about Self-Concept and Self-Identity] The PSQ is a Likert-type scale with five response options ranging from totally disagree to totally agree. Reliability and Validity In the first study, coefficient alpha = .85 and in study two, alpha = .83. Data analysis supported a four-dimensional model (see the four categories above). Positive correlations with other self-concept measures were statistically significant. Other notes The authors estimated it took about 10 minutes to complete the PSQ. Their first study included people ages 12 to 36 ( n = 506). In the second s
Post a Comment
Read more

Academic Self-Efficacy Scale ASE

  Overview The  Academic Self-Efficacy Scale is an application of Self-Efficacy Theory   to examine the relationship between self-efficacy and academic performance using 8-items rated on a 7-point scale. The work of Chemers et al. (2001) has been widely cited. Format The 8-items are rated on a 7-point Likert-type scale ranging from 1 = Very Untrue to 7 = Very True. Sample Items 2. I know how to take notes. 6. I usually do very well in school and at academic tasks.   Reliability, Validity, and Other Research notes In the article describing the development and use of the ASE, the authors observed: “As predicted, academic self-efficacy was significantly and directly related to academic expectations and academic performance.” (Chemers et al., 2001, p. 61)   Sutton et al. (2011) reported Cronbach's alpha of .83 in their study of academic self-esteem and personal strengths. ASE was highly positively correlated with ACT scores (.24) and GPA (.39). An Arabic version of t
Post a Comment
Read more