Item response theory (IRT) is not widely used in counseling psychology research, despite its considerable advantages for instrument development. Focus groups help create a broad and representative item pool that is more likely to tap the full range of the latent dimension, thereby capitalizing on IRT strengths. We provide suggestions for using these tools, with an empirical example, the Everyday Multicultural Competencies/Revised Scale of Ethnocultural Empathy (EMC/RSEE). Rasch IRT methods were used to (a) analyze response format performance and collapse poorly performing categories, (b) evaluate differential item functioning sex bias, and (c) select items to maximize subscale sensitivity and bandwidth. Traditional classical test theory (CTT) subscales composed of items with the highest factor loadings were compared with subscales based on IRT criteria. Compared with CTT subscales, IRT subscales used in the EMC/RSEE demonstrated generally higher correlations with other variables of interest, and superior sensitivity to change over time and to group differences.
AndrichD. (2002). Understanding Rasch measurement: Understanding resistance to the data-model relationship in Rasch’s paradigm: A reflection for the next generation. Journal of Applied Measurement, 3, 325-359.
2.
BandalosD. J.FinneyS. J. (2001). Item parceling issues in structural equation modeling. In MarcoulidesG. A.SchumackerR. E. (Eds.), New developments and techniques in structural equation modeling (pp. 269-296). Mahwah, NJ: Lawrence Erlbaum.
3.
BocklischF.BocklischS. F.KremsJ. (2012). Sometimes, often, and always: Exploring the vague meanings of frequency expressions. Behavior Research Methods, 44, 155-157. doi:10.3758/s13428-011-0130-8
4.
BondT. G.FoxC. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
5.
BordinE. S. (1976). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research and Practice, 16, 252-260. doi:10.1037/h0085885
CareyM. A.AsburyJ. (2012). Focus group research. Walnut Creek, CA: Left Coast Press.
8.
CastroF. G.KellisonJ. G.BoydS. J.KopakA. (2010). A methodology for conducting integrative mixed methods research and data analysis. Journal of Mixed Methods Research, 4, 342-360. doi:10.1177/1558689810382916
9.
CattellR. B. (1974). Radial item parcel factoring vs. item factoring in defining personality structure in questionnaires: Theory and experimental checks. Australian Journal of Psychology, 26, 103-119. doi:10.1080/00049537408255223
10.
CronbachL. J.MeehlP. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. doi:10.1037/h0040957
11.
CundiffN. L.KomarrajuM. (2008). Gender differences in ethnocultural empathy and attitudes toward men and women in authority. Journal of Leadership & Organizational Studies, 15, 5-15. doi:10.1177/1548051808318000
12.
de AyalaR. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
13.
del PradoA. M.ChurchA. T. (2010). Development and validation of the enculturation scale for Filipino Americans. Journal of Counseling Psychology, 57, 469-483. doi:10.1037/a0020940
14.
DeVellisR. F. (2003). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: SAGE.
15.
DoucetteA.WolfA. W. (2009). Questioning the measurement precision of psychotherapy research. Psychotherapy Research, 19, 374-389. doi:10.1080/10503300902894422
16.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
17.
FabrigarL. R.WegenerD. T.MacCallumR. C.StrahanE. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. doi:10.1037/1082-989X.4.3.272
18.
FlickW. (1998). An introduction to qualitative research. London, England: SAGE.
19.
FloydF. J.WidamanK. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286-299. doi:10.1037/1040-3590.7.3.286
20.
ForsythR. A.SaisangjanU.GilmerJ. (1981). Some empirical results related to the robustness of the Rasch Model. Applied Psychological Measurement, 5,175-186. doi:10.1177/014662168100500203
21.
FoxC. M.JonesJ. A. (1998). Uses of Rasch modeling in counseling psychology research. Journal of Counseling Psychology, 45, 30-45. doi:10.1037/0022-0167.45.1.30
22.
FuertesJ. N.MivilleM. L.MohrJ. J.SedlacekW. E.GretchenD. (2000). Factor structure and short form of the Miville-Guzman Universality-Diversity Scale. Measurement and Evaluation in Counseling and Development, 33, 157-169.
23.
GastonL.MarmarC. R.ThompsonL. W.GallagherD. (1988). Relation of patient pretreatment characteristics to the therapeutic alliance in diverse psychotherapies. Journal of Consulting and Clinical Psychology, 56, 483-489. doi:10.1037/0022-006X.56.4.483
24.
GorsuchR. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
25.
GreensonR. R. (1967). Technique and practice of psychoanalysis. New York, NY: International Universities Press.
HansonW. E.CreswellJ. W.Plano ClarkV. L.PetskaK. S.CreswellJ. D. (2005). Mixed methods research designs in counseling psychology. Journal of Counseling Psychology, 52, 224-235. doi:10.1037/0022-0167.52.2.224
28.
HarringtonD. (2008). Confirmatory factor analysis. New York, NY: Oxford University Press.
29.
HarveyR. J.HammerA. L. (1999). Item response theory. The Counseling Psychologist, 27, 353-383. doi:10.1177/0011000099273004
30.
HorvathA. O.GreenbergL. S. (1986). Development of the Working Alliance Inventory. In GreenbergL. S.PinsoffW. M. (Eds.), The psychotherapeutic process: A research handbook (pp. 529-556). New York, NY: Guilford Press.
31.
HorvathA. O.GreenbergL. S. (1989). Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36, 223-233. doi:10.1037/0022-0167.36.2.223
32.
HoytW. T.MallinckrodtB. (2012). Improving the quality of research in counseling psychology: Conceptual and methodological issues. In FouadN. A.CarterJ. A.SubichL. M. (Eds.), APA handbook of counseling psychology, vol. 1: Theories, research, and methods (pp. 31-57). Washington, DC: American Psychological Association. doi:10.1037/13754-003
33.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:90/10.1080/10705519909540118
34.
KhuriM. L. (2004). Working with emotion in educational intergroup dialogue. International Journal of Intercultural Relations, 28, 595-612. doi:10.1016/j.ijintrel.2005.01.012
35.
KlassenA. C.CreswellJ.Plano ClarkV. L.Clegg SmithK.MeissnerH. L. (2012). Best practices in mixed methods for quality of life research. Quality of Life Research, 21, 377-380. doi:10.1007/s11136-012-0122-x
36.
KlineR. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.
37.
KruegerR. A.CaseyM. A. (2009). Focus groups: A practical guide for applied research (4th ed.). Thousand Oaks, CA: SAGE.
38.
LigtvoetR.Van der ArkL. A.BergsmaW. P.SijtsmaK. (2011). Polytomous latent scales for the investigation of the ordering of items. Psychometrika, 76, 200-216. doi:10.1007/s11336-010-9199-8
39.
LinacreJ. M. (1998). Detecting multidimensionality: Which residual data-type works best?Journal of Outcome Measurement, 2, 266-283.
40.
LinacreJ. M. (1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122.
41.
LinacreJ. M. (2004). Item discrimination, guessing and carelessness: Estimating IRT parameters with Rasch. Rasch Measurement Transactions, 18, 959-960.
42.
LinacreJ. M. (2013). A user’s guide to WINSTEPS MINISTEP Rasch-model computer programs: Program manual 3.80. Retrieved from http://www.winsteps.com/winsteps.htm
43.
LopezF. G.GormleyB. (2002). Stability and change in adult attachment style over the first-year college transition: Relations to self-confidence, coping and distress patterns. Journal of Counseling Psychology, 49, 355-364. doi:10.1037/0022-0167.49.3.355
44.
LozanoL. M.Garcia-CuetoE.MunizJ. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 4, 73-79. doi:10.1027/1614-2241.4.2.73
45.
MallinckrodtB.ArmerJ. M.HeppnerP. P. (2012). A threshold model of social support, adjustment, and distress after breast cancer treatment. Journal of Counseling Psychology, 58, 150-160. doi:10.1037/a0026549
46.
MallinckrodtB.MilesJ. R.BhaskarT.CheryN.ChoiG.SungM.-R. (2014). Development and psychometric evaluation of a comprehensive scale to assess college diversity programming. Journal of Counseling Psychology, 61, 133-145. doi:10.1037/a0035214
47.
MallinckrodtB.TekieY. T. (in press). Revision of the Working Alliance Inventory and development of a brief revised version guided by item response theory. Psychotherapy Research.
48.
ManiaciM. R.RoggeR. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83.
49.
McAleaveyA. A.NordbergS. S.HayesJ. A.CastonguayL. G.LockeB. D.LockardA. J. (2012). Clinical validity of the Counseling Center Assessment of Psychological Symptoms-62 (CCAPS-62): Further evaluation and clinical applications. Journal of Counseling Psychology, 59, 575-590. doi:10.1037/a0029855
50.
McNamaraT.KnochU. (2012). The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing, 29, 555-576. doi:10.1177/0265532211430367
51.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455. doi:10.1037/a0028085
PanayidesP.RobinsonC.TymmsP. (2010). The assessment revolution that has passed England by: Rasch measurement. British Educational Research Journal, 36, 611-626. doi:10.1080/01411920903018182
54.
PascarellaE. T.EdisonM.NoraA.HagedonL. S.TerenziniP. T. (1996). Influences on students’ openness to diversity and challenge in the first year of college. Journal of Higher Education, 67, 174-195.
55.
PaulhusD. L. (1991). Measurement and control of response bias. In RobinsonJ. P.ShaverJ. P.WrightsmanL. S. (Eds.), Measures of personality and social psychological attitudes (pp. 17-59). San Diego, CA: Academic Press.
56.
PrestonC. C.ColmanA. M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1-15. doi:10.1016/S0001-6918(99)00050-5
57.
ProchaskaJ. O.NorcrossJ. C. (2001). Stages of change. Psychotherapy, 38, 443-448. doi:10.1037/0033-3204.38.4.443
58.
ReiseS. P.WallerN. G. (2003). How many IRT parameters does it take to model psychopathology items?Psychological Methods, 8, 164-184. doi:10.1037/1082-989X.8.2.164
SchlomerG. L.BaumanS.CardN. A. (2010). Best practices for missing data management in counseling psychology. Journal of Counseling Psychology, 57, 1-10. doi:10.1037/a0018082
61.
SteinbergL.ThissenD. (1996). Uses of item response theory and the testlet concept in the measurement of psychopathology. Psychological Methods, 1, 81-97. doi:10.1037/1082-989X.1.1.81
62.
ThissenD.WainerH. (1982). Some standard errors in item response theory. Psychometrika, 47, 397-412.
63.
vandeVijverF. J. R. (1986). The robustness of Rasch estimates. Applied Psychological Measurement, 10, 45-57.
64.
WampoldB. E. (1998). Necessary (but not sufficient) innovation: Comment on Fox and Jones (1998), Koehly and Shivy (1998), and Russell, Kahn, Spoth, and Altmaier (1998). Journal of Counseling Psychology, 45, 46-49.
65.
WangW.ChenH.JinK. (2015). Item response theory models for wording effects in mixed-format scales. Educational and Psychological Measurement, 75, 157-178. doi:10.1177/0013164414528209
66.
WangY.-W.DavidsonM. M.YakushkoO. F.Savoy BielsteinH.TanJ. A.BleierJ. K. (2003). The Scale of Ethnocultural Empathy: Development, validation, and reliability. Journal of Counseling Psychology, 50, 221-234. doi:10.1037/0022-0167.50.2.221
67.
WorthingtonR. L.WhittakerT. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806-838. doi:10.1177/0011000006288127
68.
WrightB. D. (1997). A history of social science measurement. Educational Measurement: Issues and Practice, 16, 33-52. doi:10.1111/j.1745-3992.1997.tb00606.x
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
