Temporal consistency (otherwise known as test-retest reliability) is often calculated by correlating test scores for participants who have taken a test on two separate occasions. Unfortunately, analysis of the relationship between total scores across test administrations potentially ignores information related to the consistency of individual items, item interrelationships, and individual respondents. The purpose of this article is to introduce a set of new statistics intended to address these issues. Illustrative examples are provided to demonstrate the potential utility of each proposed statistic and provide proof of concept. Each of the proposed statistics works as intended, providing evidence that this set of analyses can be used to enhance the assessment of temporal consistency.
BassB. M.CascioW. F.O’ConnorE. J. (1974). Magnitude estimations of expressions of frequency and amount. Journal of Applied Psychology, 59, 313–320.
2.
BentlerP. M. (1995). EQS structural equations program manual. Encino, CA: Multivariate Software.
3.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
4.
BrownW. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322.
5.
CliffN. (1959). Adverbs as multipliers. Psychological Review, 66, 27–44.
6.
CronbachL. J.GleserG. C.NandaH.RajaratnamN. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
7.
GorsuchR. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
HoughL. M.EatonN. K.DunnetteM. D.KampJ. D.McCloyR. A. (1990). Criterion-related validities of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology, 75, 581–595.
9.
HuL.BentlerP. M. (1999). Cutoff criterion for fit indices in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.
10.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business Psychology, 27, 99–114.
11.
JacksonD. N. (1976). The appraisal of personal reliability. Paper presented at the meeting of the Society of Multivariate Experimental Psychology, University Park, PA.
12.
LeBretonJ. M.SenterJ. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.
13.
LiuM.BowlingN. A.HuangJ. L.KentT. A. (2013). Insufficient effort responding to surveys as a threat to validity: The perceptions and practices of SIOP members. The Industrial-Organizational Psychologist, 51, 32–38.
14.
MahalanobisP. C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, 2, 49–55.
15.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
16.
NewmanD. A. (2009). Missing data techniques and low response rates: The role of systematic nonresponse parameters. In LanceC. E.VandenbergR. J. (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity, and fable in the organizational and social sciences (pp. 7–36). New York, NY: Routledge.
17.
NicholsD. S.GreeneR. L.SchmolckP. (1989). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: Rationale, development, and empirical trials. Journal of Clinical Psychology, 45, 239–250.
18.
NunnallyJ. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw Hill.
19.
PearsonK. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2, 559–572.
20.
SchaferJ. L.GrahamJ. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177.
21.
SchmittN.StultsD. M. (1985). Factors defined by negatively keyed items: The result of careless respondents?Applied Psychological Measurement, 9, 367–373.
22.
SmithG. T.McCarthyD. M.AndersonK. G. (2000). On the sins of short-form development, Psychological Assessment, 12, 102–111.
23.
SpearmanC. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271–295.
24.
VandenbergR. J.LanceC. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70.
25.
ViswesvaranC.OnesD. S.SchmidtF. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557–574.
26.
WoodsC. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 189–194.
27.
YuC. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Unpublished dissertation.
