Previous studies have shown that method effects associated with item wording produce artifactual factors and threaten scale validity. This study examines item wording effects on a scale of attitudes toward learning mathematics for Taiwanese and U.S. samples. Analyses from a series of CFA (confirmatory factor analysis) models support the presence of method effects for both samples. In addition, findings show that U.S. students tended to report higher means on not only the substantive factors but also the method factor, compared to Taiwanese students. The effect sizes on the mean differences are medium to large.
American Educational Research Association, American Psychological Association, & the National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2.
BagozziR. P.YiY. (1991). Multitrait-multimethod matrices in consumer research. Journal of Consumer Research, 17, 426-439.
3.
BensonJ.HocevarD. (1985). The impact of item phrasing on the validity of attitude scales for elementary school children. Journal of Educational Measurement, 22(3), 231-240.
4.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
5.
ChenF. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464-504.
6.
CoteJ. A.BuckleyR. (1987). Estimating trait, method, and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 24, 315-318.
7.
DiStefanoC.MotlR. W. (2009). Self-esteem and method effects associated with negatively worded items: Investigating factorial invariance by sex. Structural Equation Modeling: A Multidisciplinary Journal, 16, 134-146.
8.
EidM. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241-261.
9.
HarveyR. J.BillingsR. S.NilanK. J. (1985). Confirmatory factor analysis of the job diagnostic survey: Good news and bad news. Journal of Applied Psychology, 70, 461-468.
10.
HinkinT. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967-988.
11.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
12.
KennyD. A.KashyD. A. (1992). Analysis of the multitrait–multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
13.
LeungF. K. S. (2002). Behind the high achievement of East Asian students. Educational Research and Evaluation, 8, 87-108.
14.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
15.
MarshH. W. (1986). The bias of negatively worded items in rating scales for young children: A cognitive-developmental phenomenon. Developmental Psychology, 22, 37-49.
16.
MarshH. W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.
17.
MartinM. O.PreuschoffC. (2008). Creating the TIMSS 2007 background indices. In OlsonJ. F.MartinM. O.MullisI. V. S. (Eds.), TIMSS 2007 Technical Report (pp. 281-338). Boston, MA: TIMSS & PIRLS International Study Center.
18.
McDonaldR. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
19.
MillsapR. E.KwokO.-M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93-115.
20.
MotlR. W.ConroyD. E. (2000). Validity and invariance of the Social Physique Anxiety Scale. Medicine and Science in Sports and Exercise, 25, 1049-1053.
21.
MullisI. V. S.MartinM. O.FoyP. (2008). TIMSS 2007 International Mathematics Report: Findings from IEA’s trends in international mathematics and science study at the fourth and eighth grades. Chestnut Hill, MA: TIMSS & PIRLS International Study Center.
22.
MuthénL. K.MuthénB. O. (1998-2009). Mplus user’s guide (5th ed.). Los Angeles, CA: Muthén & Muthén.
23.
NunnallyJ. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
24.
PodsakoffP. M.MacKenzieS. B.LeeJ. Y.PodsakoffN. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879-903.
25.
ThompsonM. S.GreenS. B. (2006). Evaluating between-group differences in latent variable means. In HancockG. R.MuellerR. O. (Eds.), A second course in structural equation modeling (pp. 119-169). Greenwich, CT: Information Age Publishing.
26.
TomásJ. M.OliverA. (1999). Rosenberg’s self-esteem scale: Two factors or method effects. Structural Equation Modeling, 6, 84-98.
27.
WidamanK. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9(1), 1-26.
28.
YangY.GreenS. B. (2011). Coefficient alpha: A reliability coefficient for the 21st century?Journal of Psychoeducational Assessment, 29(4), 377-392.
