Despite the utility of cognitive interviews, very few studies report on the practical application of this survey design procedure. Likewise, very few studies querying perceptions of honors education incorporate valid survey items that reflect the views of students of color who are eligible yet opt out of participation. We respond to these areas of inquiry by describing the development of survey items that explore why students would opt out of honors education in collegiate settings and may identify unintentional institutional barriers that prevent the participation of marginalized groups. We report on the process and outcomes of cognitive interviews used to refine our survey tool and the categorical principal components analysis (CATPCA) that not only provided additional refinements but also revealed the underlying two-component structure of the instrument.
AbesE. S.JonesS. R.McEwenM. K. (2007). Reconceptualizing the model of multiple dimensions of identity: The role of meaning-making capacity in the construction of multiple identities. Journal of College Student Development, 48(1), 1–22. https://doi.org/10.1353/csd.2007.0000
2.
BastedoM. N.CumportP. J. (2003). Access to what? Mission differentiation and academic statrification in U.S. Public higher education. Higher Education, 46 (3), 341–359. https://doi.org/10.1023/a:1025374011204
3.
BernardiR. A.NashJ. (2023). The importance and efficacy of controlling for social desirability response bias. Ethics & Behavior, 33(5), 413–429. https://doi.org/10.1080/10508422.2022.2093201
4.
BoatengG. O.NeilandsT. B.FrongilloE. A.Melgar-QuiñonezH. R.YoungS. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 1-18. https://doi.org/10.3389/fpubh.2018.00149
5.
BowlegL. (2008). When black + lesbian + woman ≠ black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research. Sex Roles, 59 (5-6), 312–325. https://doi.org/10.1007/s11199-008-9400-z
6.
Bradbury-JonesC. (2007). Enhancing rigour in qualitative health research: Exploring subjectivity through Peshkin’s I’s. Journal of Advanced Nursing, 59(3), 290–298. https://doi.org/10.1111/j.1365-2648.2007.04306.x
7.
BrimeyerT.M.SchuethsA.M.SmithW.L. (2014). Who benefits from honors: An empirical analysis of honors and non-honors students’ backgrounds, academic attitudes, and behaviors. Journal of the National Collegiate Honors Council, 15(1) 69–83. https://digitalcommons.unl.edu/nchcjournal/420/
ChooH. Y.FerreeM. M. (2010). Practicing intersectionality in sociological research: A critical analysis of inclusions, interactions, and institutions in the study of inequalities. Sociological Theory, 28(2), 129–149. https://doi.org/10.1111/j.1467-9558.2010.01370.x
10.
Cognard-BlackA.J.SpisakA.L. (2019). Creating a profile of an honors student: A comparison of honors and non-honors students at public research universities in the United States. Journal of the National Collegiate Honors Council, 20(1), 123–157. https://digitalcommons.unl.edu/nchcjournal/623/
11.
Cognard-BlackA.J.SpisakA.L. (2021). Forging a more equitable path for honors education: Advancing racial, ethnic, and socioeconomic diversity. Journal of National Collegiate Honors Council, 22(1), 81–114. https://digitalcommons.unl.edu/nchcjournal/670/
12.
CollinsP. H. (1993). Toward a new vision: Race, class, and gender as categories of analysis and connection. Race, Sex & Class, 1(1), 25–45. https://www.jstor.org/stable/41680038
13.
CollinsD. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12 (3), 229–238. https://doi.org/10.1023/a:1023254226592
14.
CrenshawK. W. (1994). Mapping the margins: Intersectionality, identity politics, and violence against women of color. In FinemenM. A.MykitiukR. (Eds.), The public nature of private violence (pp. 93–118). Routledge. https://doi.org/10.4324/9780203060902-12
15.
CuádrazG.H.UttalL. (1999). Intersectionality and in-depth interviews: Methodological strategies for analyzing race, class, and gender. Race, Gender, Class, 6(3), 156–186. https://www.jstor.org/stable/41674900
16.
DiazD.FarraggiaS. P.WellmanM. E.BottomsB. L. (2019). Honors education has a positive effect on college student success. In Cognard-BlackA. J.HerronJ.SmithP. J. (Eds.), The demonstrable value of honors education: New research evidence (pp. 59–91). National Collegiate Honors Council. https://digitalcommons.unl.edu/nchcmonochap/55/
17.
DietrichH.EhrlenspielF. (2010). Cognitive interviewing: A qualitative tool for improving questionnaires in sport science. Measurement in Physical Education and Exercise Science, 14(1), 61–60. https://doi.org/10.1080/10913670903455025
18.
DillmanD. A.SmythJ. D.ChristianL. H. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley. https://doi.org/10.1002/9781394260645
19.
FieldsA. (2017). Discovering statistics using IBM SPSS Statistics: North American edition (5th ed.). Sage.
20.
FordD.WrightB. L.Trotman ScottM. (2020). A matter of equity: Desegregating and integrating gifted and talented education for under-represented students of color. Multicultural Perspectives, 22(1), 28–36. https://doi.org/10.1080/15210960.2020.1728275
21.
FrasierM. M. (1991). Disadvantaged and culturally diverse gifted students. Journal for the Education of the Gifted, 14(3), 234–245. https://doi.org/10.1177/016235329101400305
22.
GriffinK. A.MuseusS. D. (2011). Application of mixed-methods approaches to higher education and intersectional analyses. New Directions for Institutional Research, 2011(151), 15–26. https://doi.org/10.1002/ir.396
23.
HartC. M.RitchieT. D.HepperE. G.GebauerJ. E. (2015). The balanced inventory of desireable responding short form (BIDR-16). Sage Open, 5(4), 1-9. https://doi.org/10.1177/2158244015621113
24.
HébertT. P.McBeeM. T. (2007). The impact of an undergraduate honors program on gifted university students. Gifted Child Quarterly, 51(2), 136–151. https://doi.org/10.1177/001698620729947
25.
HillJ. P. (2005). What honors students want (and expect): The views of top Michigan high school and college students. Journal of the National Collegiate Honors Council-Online Archive, 6(2), 95–107. https://digitalcommons.unl.edu/nchcjournal/177
26.
HiltonC. E. (2017). The importance of pretesting questionnaires: A field research example of cognitive pretesting the exercise referral quality of life scale (ER-QLS). International Journal of Social Research Methodology, 20(1), 21–34. https://doi.org/10.1080/13645579.2015.1091640
27.
HiltonJ.T.JordanJ. (2021). The recruitment and retention of diverse students in honors: What the last twenty years of scholarship say. Journal of the National Collegiate Honors Council-Online Archive, 22(1), 115–133. https://digitalcommons.unl.edu/nchcjournal/673/
28.
HofmeyerA.SheingoldB. H.TaylorR. (2015). Do you understand what I mean? How cognitive interviewing can strengthen valid, reliable study instruments and dissemination products. Journal of International Education Research (JIER), 11(4), 261–268. https://doi.org/10.19030/jier.v11i4.9460
KoolA.MainhardT.JaarsmaD.van BeukelenP.BrekelmansM. (2017). Effects of honours programme participation in higher education: A propensity score matching approach. Higher Education Research & Development, 36(6), 1222–1236. https://doi.org/10.1080/07294360.2017.1304362
31.
LakinJ. (2016). Universal screening and the representation of historically underrepresented minority students in gifted education. Journal for Advanced Academics, 27(2), 139–149. https://doi.org/10.1177/1932202X16630348
LintingM.van der KooijA. (2012). Nonlinear principal components analysis with CATPCA: A tutorial. Journal of Personality Assessment, 94(1), 12–25. https://doi.org/10.1080/00223891.2011.627965
34.
LongD. A.McCoachD. B.SiegleD.CallahanC. M.GubbinsE. J. (2023). Inequality at the starting line: Underrepresentation in gifted identification and disparities in early achievement. AERA Open, 9 (1), 1-25. https://doi.org/10.1177/23328584231171535
McBeeM. (2006). A descriptive analysis of referral sources for gifted identification screening by race and socioeconomic status. Journal of Secondary Gifted Education, 17(2), 103–111. https://doi.org/10.4219/jsge-2006-686
37.
McCallL. (2005). The complexity of intersectionality. Journal of Women in Culture and Society, 30(3), 1771–1800. https://doi.org/10.1086/426800
McMasterN. C.VookR. (2019). The contribution of intersectionality to quantitative research into educational inequalities. Review of Education, 7(2), 271–292. https://doi.org/10.1002/rev3.3116
40.
MeadA.D. (2018). Socioeconomic equity in honors education: Increasing numbers of first- generation and low-income students. Journal of the National Collegiate Honors Council, 19(1), 25–31. https://digitalcommons.unl.edu/nchcjournal/576
41.
MillerA.L. (2018). Ways we can do better: Bridging the gap between gifted education and honors colleges. Journal of the National Collegiate Honors Council -Online Archive, 19(2), 39–44. https://digitalcommons.unl.edu/nchcjournal/588/
42.
MillerK.CheppV.WillsonS.PadillaJ. L.PadillaJ. L. (2014). Cognitive interviewing methodology: A sociological approach for survey question evaluation. John Wiley & Sons. https://doi.org/10.1002/9781118838860
43.
MillerA. L.DumfordA. D. (2018). Do high-achieving students benefit from honors college participation? A look at student engagement for first-year students and seniors. Journal for the Education of the Gifted, 41(3), 217–241. https://doi.org/10.1177/0162353218781753
44.
MillerA. L.Speirs NeumeisterK. L. (2017). The influence of personality, parenting styles, and perfectionism on performance goal orientation in high-ability students. Journal for Advanced Academics, 28(4), 313–344. https://doi.org/10.1177/1932202X17730567
NicholsT.AiltsJ.ChangK.-L. (2016). Why not honors: Understanding students’ decisions not to enroll and persist in honors programs. Honors in Practice, 12, 33–58. https://digitalcommons.unl.edu/nchchip/251/
48.
NicholsT. J.ChangK.-L. M. (2013). Factors influencing honors college recruitment, persistence, and satisfaction at an upper-midwest land grant university. Journal of the National Collegiate Honors Council, 14(2), 105–127. https://digitalcommons.unl.edu/nchcjournal/399
PaulhusD. L. (1991). Measurement and control of response bias. In RobinsonJ. P.ShaverP. R.WrightsmanL. S. (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Academic Press. https://doi.org/10.1016/b978-0-12-590241-0.50006-x
PittmanA.A. (2001). Diversity issues & honors education. Innovations in Undergraduate Research and Honors Education: Proceedings of the Second Schreyer National Conference 2001. https://digitalcommons.unl.edu/nchcschreyer2
53.
RheaD. M.GoodwinK. (2014). High-impact recruiting: A focus group of prospective honors students. Honors in Practice-Online Archive, 10, 111–118. https://digitalcommons.unl.edu/nchchip/200
54.
RinnA. N.PluckerJ. A. (2004). We recruit them, but then what? The educational and psychological experiences of academically talented undergraduates. Gifted Child Quarterly, 48(1), 54–67. https://doi.org/10.1177/001698620404800106
55.
RinnA. N.PluckerJ. A. (2019). High-ability college students and undergraduate honors programs: A systematic review. Journal for the Education of the Gifted, 42(3), 187–215. https://doi.org/10.1177/0162353219855678
56.
RoulstonK.SheltonS. A. (2015). Reconceptualizing bias in teaching qualitative research methods. Qualitative Inquiry, 21(4), 332–342. https://doi.org/10.1177/1077800414563803
57.
RyanK.Gannon-SlaterN.CulbertsonM. J. (2012). Improving survey methods with cognitive interviews in small- and medium-scale evaluations. American Journal of Evaluation, 33(3), 414–430. https://doi.org/10.1177/1098214012441499
58.
SantosC. E.ToomeyR. B. (2018). Integrating an intersectionality lens in theory and research in developmental science. Envisioning the Integration of an Intersectional Lens in Developmental Science: New Directions for Child and Adolescent Development, 2018(161), 7–15. https://doi.org/10.1002/cad.20245
59.
SaukaniN.IsmailN. A. (2019). Identifying the components of social capital by categorical principal component analysis (CATPCA). Social Indicators Research, 141(2), 631–655. https://doi.org/10.1007/s11205-018-1842-2
60.
ScottR.I.SmithP.J. (2016). Demography of honors: The national landscape of honors education. Journal of National Collegiate Honors Council, 17(1), 73–91. https://digitalcommons.unl.edu/nchcjournal/512/
61.
ScottR.L.SmithP.J.Cognard-BlackA.J. (2017). Demography of honors: The census of U.S. Honors programs and colleges. Journal of the National Collegiate Honors Council – Online Archive, 548(1), 189–224. https://digitalcommons.unl.edu/nchcjournal/548
SpurrierR. (2008). The newest “basic characteristic” of a fully developed honors program. Honors in Practice-Online Archive, 4, 191-197. https://digitalcommons.unl.edu/nchchip/66
64.
StrayhornT. L. (2017). Using intersectionality in student affairs research. New Directions for Student Services, 2017 (157), 57–67. https://doi.org/10.1002/ss.2020
65.
ThompsonB. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association. https://doi.org/10.1037/10694-000
66.
TicknorC.S.FrazierA.D.WilliamsJ.ThompsonM. (2020). Using possible selves and intersectional theory to understand why students of color opt out of honors. Journal of the National Collegiate Honors Council, Online Archive, 21(2), 67–93. https://digitalcommons.unl.edu/nchcjournal/663/
VietzeJ.van HerpenS. G. A.Dias-BroensA.SeveriensS. E.MeeuwisseM. (2022). Self-selection from higher education: A meta-review of resources for academic decision-making of mainstream and underrepresented students. Research in Post-Compulsory Education, 27(3), 454–477. https://doi.org/10.1080/13596748.2022.2076057
69.
WarnerL. R.ShieldsS. A. (2013). The intersections of sexuality, gender, and race: Identity research at the crossroads. Sex Roles, 68 (11-12), 803–810. https://doi.org/10.1007/s11199-013-0281-4
70.
WellerB. E.BowenN. K.FaubertS. J. (2020). Latent class analysis: A guide to best practice. Journal of Black Psychology, 46(4), 287–311. https://doi.org/10.1177/0095798420930932
71.
WillisG. B. (2015). Research synthesis: The practice of cross-cultural cognitive interviewing. Public Opinion Quarterly, 79(S1), 359–395. https://doi.org/10.1093/poq/nfu092
72.
WitenkoV.Mireles-RiosR.RiosV. M. (2017). Networks of encouragement: Who’s encouraging Latina/o students and White students to enroll in honors and advanced-placement courses?Journal of Latinos and Education, 16(3), 176–191. https://doi.org/10.1080/15348431.2016.1229612
73.
WolfensbergerM. V. C.OffringaG. J. (2012). Qualities honours students look for in faculty and courses. Journal of the National Collegiate Honors Council-Online Archive, 13(2), 171–182. https://digitalcommons.unl.edu/nchcjournal/361
74.
ZissetteS.AtujunaM.TolleyE. E.OkumuE.AuerbackJ. D.HodderS. L.AralS. O.AdimoraA. A. (2020). Cognitive testing of an instrument to evaluate acceptability and use of pre-exposure prophylaxis products among women. Applied Cognitive Psychology, 34(1), 78–84. https://doi.org/10.1002/acp.3590
