Restricted accessResearch articleFirst published online 2019-12
Digit Span Subscale Scores May Be Insufficiently Reliable for Clinical Interpretation: Distinguishing Between Stratified Coefficient Alpha and Omega Hierarchical
The Digit Span subscale (Digit Span Forward, Backward, and Sequencing combined composite) internal inconsistency reliability has been reported at .93, based on a coefficient known as stratified coefficient alpha. With accessible examples, we demonstrate that stratified coefficient alpha can deviate substantially from a model-based internal consistency reliability that represents an underlying dimension, that is, omega hierarchical. Next, we simulated item-level Digit Span subscale data to correspond very closely to the Wechsler Adult Intelligence Scale–fourth edition normative sample. Based on omega hierarchical, we estimated the internal consistency reliability associated with the Digit Span subscale scores at .74. In light of the results, clinicians are cautioned against interpreting Digit Span subscale scores. Instead, interpretations should probably be restricted to the Digit Span Forward, Digit Span Backward, and Digit Span Sequencing test scores. Finally, we offer suggestions for improvement to achieve higher levels of Digit Span subscale score reliability.
ArbuckleJ. L. (2012). AMOS (Version 21.0) [Computer Program]. Chicago, IL: IBM SPSS.
2.
BankenJ. A. (1985). Clinical utility of considering Digits Forward and Digits Backward as separate components of the Wechsler Adult Intelligence Scale-Revised. Journal of Clinical Psychology, 41, 686-691.
3.
BlackburnH. L.BentonA. L. (1957). Revised administration and scoring of the digit span test. Journal of Consulting Psychology, 21, 139-143.
4.
BlankenshipA. B. (1938). Memory span: A review of the literature. Psychological Bulletin, 35, 1-25.
5.
BrunnerM.NagyG.WilhelmO. (2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80, 796-846.
6.
ColomR.JungR. E.HaierR. J. (2007). General intelligence and memory span: Evidence for a common neuroanatomic framework. Cognitive Neuropsychology, 24, 867-878.
7.
CortinaJ. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98-104.
8.
CronbachL. J.SchönemannP.McKieD. (1965). Alpha coefficients for stratified-parallel tests. Educational and Psychological Measurement, 25, 291-312.
9.
DempsterF. N. (1981). Memory span: Sources of individual and developmental differences. Psychological Bulletin, 89, 63-100.
10.
DobbsA. R.RuleB. G. (1989). Adult age differences in working memory. Psychology and Aging, 4, 500-503.
11.
FeldtL. S.QuallsA. L. (1996). Bias in coefficient alpha arising from heterogeneity. Applied Measurement in Education, 9, 277-286.
12.
GignacG. E. (2007). Multi-factor modeling in individual differences research: Some recommendations and suggestions. Personality and Individual Differences, 42, 37-48.
13.
GignacG. E. (2009). Psychometrics and the measurement of emotional intelligence. In StoughC.SaklofskeD. H.ParkerJ. D. A. (Eds.), Assessing emotional intelligence: Theory, research, and applications (pp. 9-40). London, England: Springer.
14.
GignacG. E. (2014). On the inappropriateness of using items to calculate total scale score reliability via coefficient alpha for multidimensional scales. European Journal of Psychological Assessment, 30, 130-139.
15.
GignacG. E. (2015). The magical numbers 7 and 4 are resistant to the Flynn effect: No evidence for increases in forward or backward recall across 85 years of data. Intelligence, 48, 85-95.
16.
GignacG. E.WatkinsM. W. (2013). Bifactor modeling and the estimation of model-based reliability in the WAIS-IV. Multivariate Behavioral Research, 48, 639-662.
17.
GignacG. E.WeissL. G. (2015). Digit Span is (mostly) related linearly to general intelligence: Every extra bit of span counts. Psychological Assessment, 27, 1312-1323.
18.
GlassmireD. M.RossP. T.KinneyD. I.NitchS. R. (2016). Derivation and cross-validation of cutoff scores for patients with schizophrenia spectrum disorders on WAIS-IV Digit Span–based performance validity measures. Assessment, 23, 292-306.
19.
GreenS. B.LissitzR. W.MulaikS. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality1. Educational and Psychological Measurement, 37, 827-838.
20.
KanaiC.HashimotoR.ItahashiT.TaniM.YamadaT.OtaH.. . . KatoN. (2017). Cognitive profiles of adults with high-functioning autism spectrum disorder and those with attention-deficit/hyperactivity disorder based on the WAIS-III. Research in Developmental Disabilities, 61, 108-115.
21.
KaplanR. M.SaccuzzoD. P. (1997). Psychological testing: Principles, applications, and issues (4th ed.). Pacific Grove, CA: Brooks/Cole.
22.
KaufmanA. S.LichtenbergerE. O. (1999). Essentials of WAIS-III assessment. New York, NY: Wiley.
23.
LichtenbergerE. O.KaufmanA. S. (2009). Essentials of WAIS-IV assessment. New York, NY: Wiley.
24.
MacDonaldM. C.AlmorA.HendersonV. W.KemplerD.AndersenE. S. (2001). Assessing working memory and language comprehension in Alzheimer’s disease. Brain and Language, 78, 17-42.
25.
MatarazzoJ. D. (1972). Wechsler’s measurement and appraisal of adult intelligence (5th ed.). New York, NY: Oxford University Press.
26.
McDermottP. A.FantuzzoJ. W.GluttingJ. J. (1990). Just say no to subtest analysis: A critique on Wechsler theory and practice. Journal of Psychoeducational Assessment, 8, 290-302.
27.
MeyerE. M.ReynoldsM. R. (2018). Scores in space: Multidimensional scaling of the WISC-V. Journal of Psychoeducational Assessment, 36(6), 562-75. doi:10.1177/0734282917696935
28.
MillerM. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.
29.
MosierC. I. (1943). On the reliability of a weighted composite. Psychometrika, 8, 161-168.
30.
NiilekselaC. R.ReynoldsM. R.KaufmanA. S. (2013). An alternative Cattell–Horn–Carroll (CHC) factor structure of the WAIS-IV: Age invariance of an alternative model for ages 70–90. Psychological Assessment, 25, 391-404.
31.
NunnallyJ. (1978). Psychometric methods (2nd ed.). New York, NY: McGraw-Hill.
32.
OberauerK.SüßH.-M.SchulzeR.WilhelmO.WittmannW. W. (2000). Working memory capacity: Facets of a cognitive ability construct. Personality and Individual Differences, 29, 1017-1045.
33.
OlsonK.HewitJ.SlaterL. G.ChambersT.HicksD.FarmerA.. . . KolbB. (2016). Assessing cognitive function in adults during or following chemotherapy: A scoping review. Supportive Care in Cancer, 24, 3223-3234.
34.
PutkaD. M.SackettP. R. (2010). Reliability and validity. In FarrJ. L.TippinsN. T. (Eds.), Handbook of employee selection (pp. 9-49). New York, NY: Routledge.
35.
RaeG. (2008). A note on using alpha and stratified alpha to estimate the reliability of a test composed of item parcels. British Journal of Mathematical and Statistical Psychology, 61, 515-525.
36.
ReynoldsC. R. (1997). Forward and backward memory span should not be combined for clinical analysis. Archives of Clinical Neuropsychology, 12, 29-40.
37.
ReynoldsC. R.BiglerE. D. (1994). Test of Memory and Learning (TOMAL). Austin, TX: Pro-Ed.
38.
RichardsonJ. T. (2007). Measures of short-term memory: A historical review. Cortex, 43, 635-650.
39.
SchmittN. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.
40.
SchofieldN. J.AshmanA. F. (1986). The relationship between digit span and cognitive processing across ability groups. Intelligence, 10, 59-73.
41.
WatkinsM. W. (2000). Cognitive profile analysis. School Psychology Quarterly, 15, 465-479.
42.
WatkinsM. W. (2013). Omega [Computer software]. Phoenix, AZ: Ed & Psych.
43.
WebbN. M.ShavelsonR. J.HaertelE. H. (2006). Reliability coefficients and generalizability theory. In RaoC. R.SinharayS. (Eds.), Handbook of statistics: Psychometrics (Vol. 26, pp. 81-124). Amsterdam, Netherlands: Elsevier.
44.
WechslerD. (1939). The measurement and appraisal of adult intelligence. Baltimore, MD: Lippincott Williams & Wilkins.
45.
WechslerD. (1955). Manual for the Wechsler Adult Intelligence Scale. New York, NY: Psychological Corporation.
46.
WechslerD. (1974). Wechsler Intelligence Scale for Children–Revised. New York, NY: Psychological Corporation.
47.
WechslerD. (1981). Wechsler Adult Intelligence Scale–Revised. New York, NY: Psychological Corporation.
48.
WechslerD. (1997). Wechsler Adult Intelligence Scale–Third edition: Administration and scoring manual. San Antonio, TX: Psychological Corporation.
49.
WechslerD. (2008). Wechsler Adult Intelligence Scale–Fourth edition: Technical and interpretive manual. San Antonio, TX: Pearson Assessment.
50.
WechslerD. (2014). Wechsler Intelligence Scale for Children–Fifth edition. San Antonio, TX: Pearson Assessment.
51.
WerheidK.HoppeC.ThoneA.MullerU.MungersdorfM.Yves von CramonD. (2002). The Adaptive Ordering Test: Clinical application, reliability, and validity of a verbal working memory test. Archives of Clinical Neuropsychology, 17, 547-565.
52.
WisdomN. M.MignognaJ.CollinsR. L. (2012). Variability in Wechsler Adult Intelligence Scale-IV subtest performance across age. Archives of Clinical Neuropsychology, 27, 389-397.
53.
WoodsD. L.KishiyamaM. M.YundE. W.HerronT. J.EdwardsB.PolivaO.. . . ReedB. (2011). Improving digit span assessment of short-term verbal memory. Journal of Clinical and Experimental Neuropsychology, 33, 101-111.
54.
ZinbargR. E.RevelleW.YovelI.LiW. (2005). Cronbach’s α, Revelle’s β, and McDonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123-133.
