Why Counselor's Tests Are Not Reliable

The reason counselor's tests are not reliable is that reliability is a property of scores not tests. This isn't a matter of semantics. Think about it this way.

Give all the students in one school an achievement test. The test items don't change so they appear stable, consistent, and reliable. However, when publishers report reliability values, they calculate the reliability statistics based on scores. Scores vary from one administration to another. If you ever took a test twice and got a different score, you know what I mean. Individuals change from day to day. And we change from year to year. Also, even a representative sample of students for a nation can be different each year.

Everytime we calculate a reliability statistic, the statistic is slightly different.

Reliability values vary with the sample.

Reliability values also vary with the method used for calculation. You can get high reliability values using coefficient alpha with scores from a one-time administration. This method is common in research articles. But you will see different values from the same research team in different samples in the same article.


If we use a split-half method, which usually calculates reliability based on a correlation between two halves of one test, then we can get a reliability value based on one administration. But that's only half a test! Researchers use the Spearman-Brown formula to correct for the shortened half-test problem- but that's just an estimate of what the full test could be.


There's also a test-retest reliability method. Give a test one time, wait awhile- maybe a week or several weeks, then retest. That gives you an estimate of stability. But if you have a good memory, you can score higher on the second test on some tests like intelligence and achievement.


By now you get the point. Any one test can be associated with a lot of reliability values. The problem is not necessarily with counselor tests. The problem can be misunderstanding that tests do not have one reliability value. As with many things in science, there are many variables to consider when answering a question.

Reputable test publishers include reliability values in their test manuals. Counselors, Psychologists, and other test users ought to know about test score reliability.

Learn more assessment and statistical concepts in

Applied Statistics: Concepts for Counselors

AMAZON BOOKS

Connections

My Page    www.suttong.com

My Books   AMAZON     GOOGLE PLAY STORE

FACEBOOK   Geoff W. Sutton

TWITTER  @Geoff.W.Sutton

PINTEREST  www.pinterest.com/GeoffWSutton

Publications (many free downloads)

     

  Academia   Geoff W Sutton   (PhD)

     

  ResearchGate   Geoffrey W Sutton   (PhD)

Take a brief Counseling Test Quiz 101

Can you answer these questions that every counselor ought to know?

Choose the BEST available answer.

I'll post the answers below.

1. If the correlation between a test of intelligence and a test of achievement is usually between .88 and .92, how well can you use the intelligence test results to predict achievement test results?

A. Very well
B. Moderately well
C. Not well at all
D. None of the above

2. A personality test score was high on a scale of Extraversion. The validity of the Extraversion scale was reported as .52 to .57 compared to similar tests. How much confidence should the person have that their score is "valid?"

A. A high degree
B. A moderate degree
C. A low degree
D. None of the above

3. School counselors administered a questionnaire to 1,000 students. They calculated results for answers about four school improvements rated on a scale of 1 to 5. Most of the scores were in the range of 18 to 20. The counselors reported a mean rating of 4.6 for each of the 4 items. Based on these data, what should they have reported?

A. The mean is fine-- an average is all that is needed.
B. They should report the Mean and Standard Deviation.
C. They should report the reliability with the mean.
D. They should report the median and range.

4. An agency director asks a counselor to determine if there was evidence of improvement in well-being for clients in one of three treatment groups. Assuming a normal distribution of the data, which of the following statistical procedures could provide the best answer?

A. An independent samples t test
B. A one-way analysis of variance
C. A two-way analysis of variance
D. A chi-square test





ANSWERS



1. A. Other things being equal, the correlation between the two tests is strong thus, most of the time the intelligence test score will be a good predictor of the achievement test score. See Chapter 12 in Applied Statistics: Concepts for Counselors.

2. C. We do not know much about the validation of the Extraversion scale ; however, we know the validity values in the .50s are low so the best answer, given the limited data, is C. Validity coefficients range from 0.0 to 1.0. Important note: Validity is a product of the interpretation of data based on scores. Although it is common to refer to a test's validity, tests really do not have validity. Instead, there is a history of validity statistics and interpretations associated with validity. See chapter 20 in Applied Statistics: Concepts for Counselors.

3. D. The data appeared skewed given that 4 items on a 5-point scale would yield a maximum of 20. So, based on the limited data, the median would be the most typical value. When reporting the mean, counselors ought to report the standard deviation, but in this case, the median appears to be the best value. See Chapters 7-10 of Applied Statistics: Concepts for Counselors.

4. A one-way analysis of variance can be used to analyze data from two or more groups. If the overall value is statistically significant, t tests or other post hoc tests can be used to compare pairs of means. See Chapters 15-17 of Applied Statistics: Concepts for Counselors.



APPLIED STATISTICS: CONCEPTS FOR COUNSELORS is available as an eBook or paperback from AMAZON.

Book website  
https://sites.google.com/view/counselorstatistics/home

"If you need to review basic statistics and don’t know where to begin, this book is perfect! It makes difficult concepts easy to understand. I would recommend it for my undergraduate students who haven’t had Statistics in a while and need a refresher, or for graduate students facing their first graduate level research class!"

...Heather L. Kelly, Psy.D., Professor of Psychology, Evangel University

Springfield, Missouri, USA

You may also find this book relevant.

CREATING SURVEYS

Measuring Spiritual Outcomes in Psychotherapy

The Theistic Spirituality Outcome Scale (TSOS) has potential as a useful outcome measure.


Recently, a group of us completed a study of clients who saw Christian counselors. We assessed their current well-being using two measures: The Schwartz Outcome Scale (SOS) and the Theistic Outcome Scale (TSOS). (See references below.)


The TSOS was designed by Richards (2005) as a measure of well-being for people associated with a theistic religion like Christianity, Judaism or Islam. We used the 17-item version, which uses a 5-point response format from 1 = never to 5 = almost always to rate each item (e.g., “I felt spiritually alive.”).


Reliability

We only calculated coefficient alpha, which was strong at .95.

Validity

The TSOS was significantly correlated with ratings of satisfaction with Christian counseling (.65) and likelihood of returning to Christian counseling (.62).

It was significantly correlated with the SOS measure of general well-being (.84).

Other significant correlations were:

TIPI (a Big 5 measure; Gosling et al., 2003)

Extraversion .34

Agreeableness .50

Neuroticism .51

Conscientiousness .39

Snyder's Hope Scale .72 (Snyder et al., 2010)

Attachment to God Inventory (Beck & McDonald, 2004)

  Avoidant  -.55

  Anxious   -.40

Religious Practices Index  .41 (See Sutton et al., 2016)

Intratextual Fundamentalism Scale .56 (See Williamson et al., 2010)

Counselors, read more about reliability and validity of test scores in APPLIED STATISTICS: CONCEPTS FOR COUNSELORS

Resource Link:  A – Z Test Index

References

Beck, R., & McDonald, A. (2004). Attachment to God: The Attachment to God Inventory, tests of working model correspondence, and an exploration of faith group differences. Journal of Psychology and Theology, 32, 92-103. doi:10.1037/t46035-000

Gosling, S. D., Rentfrow, P. J., & Swann, Jr., W. B. (2003). A very brief measure of the big-five personality domains. Journal of Research in Personality, 37, 504-528. doi:10.1016/s0092-6566(03)00046-1

Richards, P. S., Smith, T. B., Schowalter, M., Richard, M., Berrett, M. E., & Hardman, R. K. (2005). Development and validation of the Theistic Spiritual Outcome Survey. Psychotherapy Research, 15, 457-469. doi:10.1080/10503300500091405

Snyder, C. R., Harris, C., Anderson, J. R., Holleran, S. A., Irving, L. M., Sigmon, S. T., Yoshinoba, L., Gibb, J., Langelle, C., & Harney, P. (1991). The will and the ways: Development and validation of an individual-differences measure of hope. Journal of Personality and Social Psychology, 60, 570-585. doi:10.1037/0022-3514.60.4.570

Sutton, G. W., Arnzen, C. A., & Kelly, H. L. (2016). Christian counseling and psychotherapy: Components of clinician spirituality that predict type of Christian intervention. Journal of Psychology and Christianity, 35, 204-214.

Sutton, G. W., Kelly, H., Worthington, E. L. Jr., Griffin, B. J., & Dinwiddie, C. (in press) Satisfaction with Christian Psychotherapy and Well-being: Contributions of Hope, Personality, and Spirituality. Spirituality in Clinical Practice.

Williamson, W. P., Hood, R. W. Jr., Ahmad, A., Sadiq, M., Hill, P. C. (2010). The Intratextual Fundamentalism Scale: Cross-cultural application, validity evidence, and relationship with religious orientation and the big 5 factor markers. Mental Health, Religion & Culture, 13, 721-747. doi:10.1080/13674670802643047

Read more about validity of surveys and tests in CREATING SURVEYS

Please check out my website   www.suttong.com

   and see my books on   AMAZON       or  GOOGLE STORE

Also, consider connecting with me on    FACEBOOK   Geoff W. Sutton    

   TWITTER  @Geoff.W.Sutton    

You can read many published articles at no charge:

  Academia   Geoff W Sutton     ResearchGate   Geoffrey W Sutton 

 

 

Understanding the Reliability of Educational and Psychological Tests

Why aren't tests reliable?

The reason tests are not reliable is that reliability is a property of the interpretation of scores not the tests themselves. 

This isn't a matter of semantics.

Think about it this way.

Give all the students in one school an achievement test. The test items don't change so they appear stable, consistent, and reliable. However, when publishers report reliability values, they calculate the reliability statistics based on scores. Scores vary from one administration to another. If you ever took a test twice and got a different score, you know what I mean. Individuals change from day to day. And we change from year to year. Also, even a representative sample of students for a nation can be different each year.


Every time we calculate a reliability statistic, the statistic is slightly different.

Reliability values vary with the sample.

Reliability values vary with the method of calculation.

Reliability values also vary with the method used for calculation. You can get high reliability values using coefficient alpha with scores from a one-time administration. This method is common in research articles. But you will see different values from the same research team in different samples in the same article.


If we use a split-half method, which usually calculates reliability based on a correlation between two halves of one test, then we can get a reliability value based on one administration. But that's only half a test! Researchers use the Spearman-Brown formula to correct for the shortened half-test problem- but that's just an estimate of what the full test could be.


There's also a test-retest reliability method. Give a test one time, wait awhile- maybe a week or several weeks, then retest. That gives you an estimate of stability. But if you have a good memory, you can score higher on the second test on some tests like intelligence and achievement.


By now you get the point. Any one test can be associated with a lot of reliability values. The reliability problem is not just about tests. The problem can be understanding that tests do not have one reliability value. As with many things in science, there are many variables to consider when answering a question.

Reputable test publishers include reliability values in their test manuals. Teachers, Counselors, Psychologists, and other users ought to know about test score reliability.

Learn more assessment and statistical concepts in


Applied Statistics: Concepts for Counselors 

AMAZON BOOKS




Learn more about assessment and statistics at the Applied Statistics website


Learn more about Creating Surveys

Quick Notes on Test Reliability

Reliability is a property of scores not tests.

Reliability may mean stability of scores over time.

Reliability may mean how consistently test questions measure whatever the test measures.

Reliable test scores in one culture do not mean they will be reliable in another culture.

Reliable test scores do not guarantee the score are valid - but reliability places a limit on validity.

Reliability statistical concepts apply to tests, quizzes, polls, surveys...sets of questions yielding numerical scores.



Note: This is a re-posting of a post to this new blog. 

Links to Connections

My Page    www.suttong.com

  

My Books  AMAZON          and             GOOGLE STORE

 

FOLLOW   FACEBOOK   Geoff W. Sutton   TWITTER  @Geoff.W.Sutton

 

PINTEREST  www.pinterest.com/GeoffWSutton

 

Articles: Academia   Geoff W Sutton   ResearchGate   Geoffrey W Sutton