To the detriment of human resource development (HRD) theory building and research, many scholars may think that research data with a low coefficient alpha is destined for the file drawer; this does not have to be the case. Contemporary literature suggests that many scholars do not know how to move forward with data that yields α < .70. In addition, an investigation revealed that many scholars practice the method of item deletion to increase alpha. Besides supporting the case that discarding research simply because of low coefficient alphas may be unnecessary, a guide is presented to demonstrate how scholars and scholar–practitioners may be able to analyze data when an initial estimate of internal reliability is low. We caution that deleting items may increase reliability at the cost of validity. As an alternative, this study demonstrates that eliminating subjects can increase alpha and maintain the integrity of the scale. This guide presents generalizability theory as a means to identify the source of error variance in data as well as a step-by-step process to correct for low coefficient alpha. The guide is illustrated with data and R syntax.
BakerR.TurnerJ. R.YoonH. J.KwonK. (Eds.). (2019). Examination of theoretical foundation: Implications for HRD research. Advances in Developing Human Resources, 21, 283–400.
BernardiR. A. (1994). Validating research results when Cronbach’s alpha is below 0.70: A methodological procedure. Educational and Psychological Measurement, 54, 766–775. https://doi.org/10.1177/0013164494054003023
4.
BonettD. G.WrightT. A. (2015). Cronbach’s alpha reliability: Interval estimation, hypothesis testing, and sample size planning. Journal of Organizational Behavior, 36, 3–15. https://doi.org/10.1002/job.1960
5.
BoscoF. (2018). File drawer problem. In FreyB. B. (Ed.), The Sage encyclopedia of educational research, measurement, and evaluation (Vols. 1–4, pp. 677–678). SAGE. https://doi.org/10.4135/9781506326139
6.
BrownJ. D. (2018). Spearman-Brown prophecy formula. In FreyB. B. (Ed.), The Sage encyclopedia of educational research, measurement, and evaluation. SAGE. https://doi.org/10.4135/9781506326139
7.
CantyA.RipleyB. (2019). boot: Bootstrap R (S-PLUS) functions (R package version 1.3-22).
8.
ChalofskyN. (2007). The seminal foundation of the discipline of HRD: People, learning, and organizations. Human Resource Development Quarterly, 18, 431–442. https://doi.org/10.1002/hrdq.1212
9.
CookC.ThompsonB. (2001). Psychometric properties of scores from the web-based LibQUAL+ study of perceptions of library service quality. Library Trends, 49, 585–603.
10.
CortinaJ. M. (1993). What is coefficient alpha an examination of theory and applications. Journal of Applied Psychology, 78, 98–104.
11.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. https://doi.org/10.1007/BF02310555
12.
CronbachL. J.GleserG. C.NandaH.RajaratnamN. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley & Sons.
13.
CronbachL. J.RajaratnamN.GleserG. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x
14.
CronbachL. J.ShavelsonR. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391–418. https://doi.org/10.1177/0013164404266386
15.
EdwardsJ. M.FinchW. H. (2018). Missing data analysis. In FreyB. B. (Ed.), The Sage encyclopedia of educational research, measurement, and evaluation. SAGE. https://doi.org/10.4135/9781506326139
16.
EndersC. K. (2004). The impact of missing data on sample reliability estimates: Implications for reliability reporting practices. Educational and Psychological Measurement, 64, 419–436. https://doi.org/10.1177/0013164403261050
17.
EndersC. K.BandalosD. L. (1999). The effects of heterogeneous item distributions on reliability. Applied Measurement in Education, 12, 133–150. https://doi.org/10.1207/s15324818ame1202_2
18.
FanX.ThompsonB. (2001). Confidence intervals for effect sizes: Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 61, 517–531. https://doi.org/10.1177/0013164401614001
19.
FieldA. P.MilesJ.FieldZ. (2012). Discovering statistics using R. SAGE.
GillM. J. (2017). Outlier analysis. In AllenM. (Ed.), The Sage encyclopedia of communication research methods. SAGE. https://doi.org/10.4135/9781483381411
22.
GrecoL. M.O’BoyleE. H.CockburnB. S.YuanZ. (2018). Meta-analysis of coefficient alpha: A reliability generalization study. Journal of Management Studies, 55, 583–618. https://doi.org/10.1111/joms.12328
23.
GronlundN. E.LinnR. L. (1990). Measurement and evaluation in teaching (6th ed.). Palgrave Macmillan.
24.
HeiderF. (1958). The psychology of interpersonal relations. John Wiley & Sons.
25.
HelmsJ. E.HenzeK. T.SassT. L.MifsudV. A. (2006). Treating Cronbach’s alpha reliability coefficients as data in counseling research. The Counseling Psychologist, 34, 630–660. https://doi.org/10.1177/0011000006288308
26.
HensonR. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177–189. https://doi.org/10.1080/07481756.2002.12069034
27.
KaplanR. W.SaccuzzoD. P. (1982). Psycological testing: Principles, applications, and issues. Brooks/Cole.
28.
KellyH. H. (1992). Common-sense psychology and scientific psychology. Annual Review of Psychology, 43, 1–23.
29.
KopalleP. K.LehmannD. R. (1997). Alpha inflation? The impact of eliminating scale items on Cronbach’s alpha. Organizational Behavior and Human Decision Processes, 70, 189–197. https://doi.org/10.1006/obhd.1997.2702
30.
KuderG. F.RichardsonM. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151-160. https://doi.org/10.1007/BF02288391
NimonK.ReioT. G.Jr (2011). Measurement invariance: A foundational principle for quantitative theory building. Human Resource Development Review, 10, 198–214. https://doi.org/10.1177/1534484311399731
37.
NimonK.ReioT. G.ShuckB. (Eds.). (2015). Quantitative data-analytic techniques to advance HRD theory and practice. Advances in Developing Human Resources, 17(1), 1–134.
38.
NunnallyJ. C. (1967). Psychometric theory. McGraw-Hill.
39.
NunnallyJ. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
40.
PetersonR. A. (1994). A meta-analysis of Cronbach’s coefficient alpha. Journal of Consumer Research, 21, 381-391. doi:10.1086/209405
41.
RaykovT. (2007). Reliability if deleted, not ‘alpha if deleted’: Evaluation of scale reliability following component deletion. British Journal of Mathematical and Statistical Psychology, 60, 201–216. https://doi.org/10.1348/000711006X115954
42.
R Development Core Team. (2019). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. http://www.R-project.org/
SongJ. H.LimD. H. (2015). Mediating analysis approaches: Trends and implications for advanced applications in HRD research. Advances in Developing Human Resources, 17, 57–71. https://doi.org/10.1177/1523422314559807
50.
TavakolM.DennickR. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
51.
ThompsonB. (2003). Score reliability: Contemporary thinking on reliability issues. SAGE.
52.
ThompsonB. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association.
53.
ThompsonB.CookC.HeathF. M. (2001). How many dimensions does it take to measure users’ perceptions of libraries? A LibQUAL+ study. Portal: Libraries and the Academy, 1, 129–138. https://doi.org/10.1353/pla.2001.0030
54.
ThompsonB.CookC.ThompsonR. L. (2002). Reliability and structure of LibQUAL+ scores: Measuring perceived library service quality. Portal: Libraries and the Academy, 2, 3–12. https://doi.org/10.1353/pla.2002.0022
55.
ThompsonB.DanielL. (1996). Seminal readings on reliability and validity: A “hit parade” bibliography. Educational and Psychological Measurement, 56, 741–745.
56.
ThompsonB.Vacha-HaaseT. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174–195. https://doi.org/10.1177/0013164400602002
57.
Vacha-HaaseT. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6–20.
58.
Van GinkelJ. R.Van der ArkL. A.SijtsmaK. (2007). Multiple imputation of item scores in test and questionnaire data, and influence on psychometric results. Multivariate Behavioral Research, 42, 387–414. https://doi.org/10.1080/00273170701360803
59.
WebbN. M.ShavelsonR. J. (2005). Generalizability theory: Overview. Encyclopedia of Statistics in Behavioral Science, 2, 717–719.
60.
WebbN. M.ShavelsonR. J.HaertelE. H. (2006). 4 Reliability coefficients and generalizability theory. Handbook of Statistics, 26, 81–124. https://doi.org/10.1016/S0169-7161(06)26004-8
61.
WilkinsonL., Task Force on Statistical Inference, American Psychological Association, & Board of Scientific Affairs. (1999). Statistical methods in psychology journals. American Psychologist, 54, 594–604. https://doi.org/10.1037/0003-066x.54.8.594
62.
WorthingtonR. L.WhittakerT. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806–838. https://doi.org/10.1177/0011000006288127
