Wednesday, August 30, 2017

Measuring outcomes in counseling

The Schwartz Outcome Scale (SOS) has the potential to be a useful measure of counseling outcomes.

I and my colleagues used the 10-item version in two recent studies about Christians attending Christian counseling.

The scale measures several aspects of well-being: physical, relational, and psychological functioning and the person’s capacity to be peaceful, interested, excited, and satisfied with life.

In previous research, the SOS was linked to hope, self-esteem, affect, mental health, and life satisfaction  (Young, Waehler, Laux, McDaniel, & Hilsenroth, 2003).

In our two studies, coefficient alpha reliability values were .93 and .96.

Some validity findings based on positive correlations with other measures may be of interest to clinicans and researchers.

Satisfaction with counseling .63

Likely to return to counseling .56

Spiritual well-being .84

Big 5 measures of Extraversion (.47) and Neuroticism (emotional stability) .40.

Read more about validity of surveys and tests in CREATING SURVEYS- Chapter 18.

Counselors, read more about validity of test scores in APPLIED STATISTICS: CONCEPTS FOR COUNSELORS- Chapter 20.


Sutton, G. W., Kelly, H., Worthington, E. L. Jr., Griffin, B. J., & Dinwiddie, C. (in press) Satisfaction with Christian Psychotherapy and Well-being: Contributions of Hope, Personality, and Spirituality. Spirituality in Clinical Practice.

Young, J.L., Waehler, C. A., Laux, J. M., McDaniel, P.S., & Hilsenroth, M. J. (2003). Four studies extending the utility of the Schwartz Outcome Scale (SOS-10). Journal of Personality Assessment, 80, 130-138. doi:10.1207/s15327752jpa8002-02

Tuesday, August 29, 2017

What makes a test valid?

What makes a test valid? is a tricky question. 

The short, and rather obnoxious response is, “nothing.” 

Like reliability, validity is a property of test scores
 rather than tests but more accurately, an interpretation
of the scores.

But it is important to take the question seriously when test-takers and users are wondering how much confidence to place in a test score. As with many aspects of science, the answers can be simply stated but there is a complicated backstory.

Validity Traditions

For many, the traditional views of test score validity will be sufficient. Tests measure constructs. Scientific constructs are ideas that have features that can be measured like reading comprehension, dominance, short-term memory, and verbal intelligence.

Construct validity is not a single entity but rather the current state of knowledge about how a test instrument’s scores have functioned in many settings and in relation to criteria. Construct validity primarily includes findings from studies of content validity, convergent validity and discriminant validity.

Content validity is based on judgment analysis from experts who mostly agree that test items measure the construct (e.g., marital satisfaction).

The other types of validity are based on the concept of correlations with a criterion. Researchers ask participants to take a specific test X along with other tests Y and Z. Test X is the test of interest such as a new math achievement test. Test Y represents other similar tests such as other math tests. When test X and test Y yield similar scores we have evidence of convergent validity.

When test X and test Z yield dissimilar results such as a relationship between our test X math achievement and test Z vocabulary, we have evidence of discriminant validity—a math test ought not to measure vocabulary aside from the minimal vocabulary used in the instructions and word problems. The relationship between the tests is based on a statistic called the validity coefficient, which will vary anytime you have a group of people taking two tests—even the very same people will get different scores on two different testing dates.

Criterion validity compares test scores to some criterion. The relationship between depression test scores measuring depression today is called concurrent validity. The relationship between test scores today and some future measurable performance is predictive validity—for example, a pre-employment test may be correlated with supervisor ratings after six months on the job.
Aside from content validity, most traditional studies are looking at the strength of the relationship between one set of test scores and another.

Factor analysis is a complex correlational procedure that examines the underlying relationship among test items and how they relate to other test items. For example, a set of vocabulary items may be correlated with answers to questions about general knowledge and be considered a “verbal factor” when the two sets of items may be grouped as representing an underlying verbal factor. These abstract underlying factors are sometimes called latent variables or latent traits.

Read more about validity of surveys and tests in CREATING SURVEYS- Chapter 18.

Counselors, read more about validity of test scores in APPLIED STATISTICS: CONCEPTS FOR COUNSELORS- Chapter 20.

Related Post

Monday, August 28, 2017

Measuring Religious Fundamentalism

Photo by Geoff W. Sutton, 2017
Researchers define religious fundamentalism in different ways. One recent model focuses on the way religious people view their sacred text. I have written about the Intratextual Fundamentalism model in a previous post (October 2013). In this post, I provide some data related to the 5-item version of the Intratextual Fundamentalism Scale (IFS), which I have found useful in research projects.

The revised version of the scale (IFS) has five items--each measuring a dimension of intratextuality (Williamson, Hood, Ahmad, Sadiq, & Hill, 2010). Here are the five dimensions (from my previous blog):

  1. Divine: The sacred text is a revelation from God (or of divine origin) to humans. Regardless of the involvement of people in the writing of the text, God (or a deity) is the author.
  2. Inerrant: The sacred text does not contain errors, inconsistencies, or contradictions. The text is objectively true.
  3. Privileged: The sacred text of the fundamentalist group is not just another sacred writing. It is the truth. Fundamentalists may show respect to people from other religions and their sacred writings but they do not consider other texts to be on the same level as their own text.
  4. Authoritative:  The sacred text is the final authority. If a conflict in belief arises, the sacred text wins.
  5. Unchanging: The sacred text is unchangeable and true for eternity. The truths are absolutes. The truths can be depended on to understand the world and as a guide for life.
You can find the scale items in the article by Williamson et al. (2010)--See reference below.

Reliability and Validity Data

In a recent study (Sutton, Kelly, Griffin, Worthington & Dinwiddie, 2016), coefficient alpha = .92). It was highly correlated with a measure of religious practices (r = .51).

Selected items (alpha = .83), but not the full five, were also correlated with Christian Beliefs Index (.56), Christian Social Values (.64), Christian Service Scale (.23), and Christian Practices (.36) in Sutton, Arnzen, and Kelly (2016).

More recently, Heather Kelly and others (2017) used the IFS in two studies of Christians' views of sin. In study one, alpha = .85. Christian Beliefs (.68) and Practices (.30) were significantly correlated with the IFS. Big Five traits also supported the validity: Conscientiousness (.23) and Openness (-.19). And in study two alpha = .93 following are the significant correlations with the IFS:  Beliefs .64; Practices .44; Openness -.19; Conscientiousness was not significant at -.10.

Although I have only used the IFS in Christian samples, it has been used with people of other religions.


Kelly, H.L., Sutton, G. W, Hicks, L., Godfrey, A. & Gillihan, C. (2018). Factors predicting the moral appraisal of sexual behavior in Christians. Journal of Psychology and Christianity, 37, (2), 162-177.

Sutton, G. W., Arnzen, C., & Kelly, H. (2016). Christian counseling and psychotherapy: Components of clinician spirituality that predict type of Christian intervention. Journal of Psychology and Christianity, 35, 204-214. Academia Link    ResearchGate Link

Williamson, W. P., Hood, R. W. Jr., Ahmad, A., Sadiq, M., Hill, P. C. (2010). The Intratextual Fundamentalism Scale: Cross-cultural application, validity evidence, and relationship with religious orientation and the big 5 factor markers. Mental Health, Religion & Culture13, 721-747. doi:10.1080/13674670802643047

Read more about Applied Statistics available on AMAZON


My Page

My Books  

 Geoff W. Sutton

TWITTER  @Geoff.W.Sutton

Publications (many free downloads)
  Academia   Geoff W Sutton   (PhD)
  ResearchGate   Geoffrey W Sutton   (PhD)

Monday, August 7, 2017

Christian Beliefs Index Measuring Christian Spirituality

One way to think about the components of religion is three-dimensional, which includes beliefs, practices, and experiences. A few years ago, a group of us studied Christian counseling to discover what Christian counselors actually did that was different from other counselors (Sutton, Arnzen, & Kelly, 2016). We wanted to get more specific about the identity of Christian counselors--beyond a simple checklist of their affiliation with a large group such as Presbyterian or a movement such as Pentecostal. As part of our plan to be more specific about spirituality, we created a few measures. 

Previously, I reported on a scale for assessing spiritual practices. This time I present a measure of beliefs, the Christian Beliefs Index.
The wording of the items clearly applies to the Christian faith, but the point of our measure was to be more precise about the diversity of beliefs within Christian cultures (i.e., groups or denominations). I’ll comment on the items below.
The full index used in the published article follows. It is presented by asking respondents to rate the items using a 5-point rating scale from 1 = strongly disagree to 5 = strongly agree.  

Please tell us a little about your Christian beliefs.

1.      I have had a born-again experience.
2.      God heals some people without human intervention.
3.      All Christians are called to share their faith with others.
4.      People who do not accept Jesus as their personal savior will spend eternity in hell.

Score the index by adding the values. Scores can range from 4 to 20 unless you use a different metric. In our study, the mean was 18.10 and the standard deviation was 2.19. The skew was -1.30, which was acceptable (+/- 1.50). Kurtosis was also acceptable (1.61).  Coefficient alpha for our study (Sutton et al., 2016) was adequate (.76).

We have used the scale in other studies. Here are the alpha levels: .72 (Kelly et al., 2018) and .80 (Sutton, Kelly, & Huver, 2018).

Validity data indicate significant positive correlations with other aspects of spirituality, which supports its use as measuring another dimension of the construct, Christian spirituality. Following is a table showing the Pearson Correlation Coefficients for the relationship between the Beliefs Index and four other measures of Christian spirituality.

Index or Measure
Personal Christian Practices
Intratextual Fundamentalism Items
Christian Social Values
Christian Service Scale

p < .01

Brief Discussion

We designed the Christian Beliefs Index to obtain a more diverse view of Christian beliefs than would be possible from reporting an affiliation with a denomination or group. The born-again item (1) is a marker of evangelicalism so, we would expect high scores reflecting this segment of Christianity. Although many Christians believe in God’s healing power, the belief in divine healing (2) without human intervention is more typical of Pentecostals and charismatics. Of course, belief in miracles, including healing, is also a part of Catholic teaching. The sharing of faith (3) is a Christian mandate and part of what it means to be an evangelical Christian. Finally, the belief in Jesus as a personal savior and the consequences of an eternity in hell is closer to fundamentalism (4). Overall, high scores reflect a conservative type of evangelical beliefs close to fundamentalism, which is supported by the shared variance with the five-item Intratextual Fundamentalism Scale (Williamson, Hood, Ahmad, Sadiq & Hill, 2010).

Note also that these beliefs are significantly related to practices as we might expect but the correlation is lower reflecting a difference between what people believe and how they practice their faith.

The Christian Beliefs Index may be used by teachers and researchers without requesting permission. We just ask you cite either the Sutton, Arnzen, and Kelly (2016) or Sutton (2017b) reference below, which are sources that provide the text of the scales. Of course, more recent studies will include the reliability and validity data.


Kelly, H.L., Sutton, G. W, Hicks, L., Godfrey, A. & Gillihan, C. (2018). Factors predicting the moral appraisal of sexual behavior in Christians. Journal of Psychology and Christianity, 37, (2), 162-177.

Sutton, G. W. (2017a). Applied statistics: Concepts for counselors. Springfield, MO: Sunflower. Amazon  Paperback ISBN-10: 1521783926, ISBN-13: 978-1521783924

Sutton, G. W. (2017b). Creating surveys: Evaluating programs and reading research. Springfield, MO: Sunflower. Amazon  Paperback ISBN-10: 1522012729  ISBN-13: 978-1522012726

Sutton, G. W., Arnzen, C., & Kelly, H. (2016). Christian counseling and psychotherapy: Components of clinician spirituality that predict type of Christian intervention. Journal of Psychology and Christianity35, 204-214.

Sutton, G.W. & Kelly, H. (2018, April 14). Christian Spirituality and Moral Judgments. Paper presented at the annual meeting of the Christian Association for Psychological Studies, International Conference, Roanoke, VA. [Manuscript in progress]

Williamson, W.P., Hood, R. W. Jr., Ahmad, A., Sadiq, M., Y Hill, P.C. (2010). The intratextual fundamentalism scale: cross-cultural application, validity evidence, and relationship with religious orientation and the Big 5 factor markers. Mental Health, Religion & Culture, 13, 721-747.

 Read more about test and other statistics in Applied Statistics.

AMAZON   Paperback or Kindle

Read more about surveys, including assessment of spirituality in Creating Surveys.

AMAZON  Paperback or Kindle

Religious Coping: The Brief RCOPE scale

The Brief RCOPE scale is a 14-item measure of religious coping developed and studied by Kenneth Pargament (e.g., 1997) and his colleagues...