Assessment development and validation is a complex endeavor but studies that are intentionally guided by specific theories and approaches are still needed. As an effective integration method, ways to expand the use of joint displays to visualize and make transparent integrations across the research process need to be explored in different fields. Using an example in education, this article contributes to mixed methods validation research in four ways: (a) linking validation theory and practice through mixed methods methodology, (b) extending the use of joint displays to assessment development and validation, (c) leveraging theories of validation to constructing joint displays, and (d) providing guidelines for mixed methods validation practice.
AddeyC.MaddoxB.ZumboB. D. (2020). Assembled validity: Rethinking Kane’s argument-based approach in the context of International Large-Scale Assessments (ILSAs). Assessment in Education: Principles, Policy & Practice, 27(6), 588–606. https://doi.org/10.1080/0969594X.2020.1843136
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (AERA/APA/NCME). (2014). Standards for educational and psychological testing. American Educational Research Association.
3.
BorsboomD.CramerA. O. J.KeivitR. A.ScholtenA. Z.FranicS. (2009). The end of construct validity. In LissitzR. W. (Ed.), The concept of validity: Revisions, new directions and applications (pp. 135–170). Information Age Publishing.
BraunV.ClarkeV. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. https://doi.org/10.1080/2159676X.2019.1628806
6.
ChapelleC. A.EnrightM. K.JamiesonJ. (2010). Does an argument-based approach to validity make a difference?Educational Measurement: Issues and Practice, 29(1), 3–13. https://doi.org/10.1111/j.1745-3992.2009.00165.x
7.
CooperA. L.BrennanM. C.LeslieG. D.BrownJ. A. (2023). Integrating literature as a data source in mixed methods research. Journal of Mixed Methods Research, 0(0), 447–461. https://doi.org/10.1177/15586898231188500
8.
DawadiS.ShresthaS.GiriR. A. (2021). Mixed-methods research: A discussion on its types, challenges, and criticisms. Journal of Practical Studies in Education, 2(2), 25–36. https://doi.org/10.46809/jpse.v2i2.20
9.
EvansB. C.CoonD. W.UmeE. (2011). Use of theoretical frameworks as a pragmatic guide for mixed methods studies: A methodological necessity?Journal of Mixed Methods Research, 5(4), 276–292. https://doi.org/10.1177/1558689811412972
10.
FerraraS.LaiE. (2015). Documentation to support test score interpretation and use. In Handbook of test development (pp. 619–639). Routledge.
11.
FettersM. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage Publications.
12.
FettersM. D.CurryL. A.CreswellJ. W. (2013). Achieving integration in mixed methods designs— principles and practices. Health Services Research, 48(6.2), 2134–2156. https://doi.org/10.1111/1475-6773.12117
13.
FettersM. D.FreshwaterD. (2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9(2), 115–117. https://doi.org/10.1177/1558689815581222
14.
FettersM. D.GuettermanT. C. (2021). Development of a joint display as a mixed analysis. In OnwuegbuzieA. J.JohnsonR. B. (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 259–275). Routledge. https://doi.org/10.4324/9780203729434-23
15.
FirestoneA. R.CruzR. A.MasseyD. (2024). Theorizing with joint displays: Salient contributors to improving preservice teachers’ practice. Journal of Mixed Methods Research, 18(2), 115–136. https://doi.org/10.1177/15586898231179508
16.
GreeneJ. C.CaracelliV. J.GrahamW. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/01623737011003255
17.
GuestG. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141–151. https://doi.org/10.1177/1558689812461179
18.
GuettermanT. C.FàbreguesS.SakakibaraR. (2021). Visuals in joint displays to represent integration in mixed methods research: A methodological review. Methods in Psychology, 5, Article 100080. https://doi.org/10.1016/j.metip.2021.100080
19.
GuettermanT. C.FettersM. D.CreswellJ. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. The Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865
20.
Haynes-BrownT. K. (2023). Using theoretical models in mixed methods research: An example from an explanatory sequential mixed methods study exploring teachers’ beliefs and use of technology. Journal of Mixed Methods Research, 17(3), 243–263. https://doi.org/10.1177/15586898221094970
21.
Haynes-BrownT. K.FettersM. D. (2021). Using joint display as an analytic process: An illustration using bar graphs joint displays from a mixed methods study of how beliefs shape secondary school teachers’ use of technology. International Journal of Qualitative Methods, 20, 1–14. https://doi.org/10.1177/1609406921993286
22.
HiroseM.CreswellJ. W. (2023). Applying core quality criteria of mixed methods research to an empirical study. Journal of Mixed Methods Research, 17(1), 12–28. https://doi.org/10.1177/15586898221086346
23.
HubleyA. M.ZumboB. D. (2017). Response processes in the context of validity: Setting the stage. In Understanding and investigating response processes in validation research (pp. 1–12). Springer.
24.
Jackson-GordonR.Plano ClarkV. L. (2023). Using a joint display for building integration in a sequential study: Informing data collection for a participatory second phase. Journal of Mixed Methods Research, 0(0), 137–146. https://doi.org/10.1177/15586898231179848
25.
JamesT. G.DeJonckheereM.GuettermanT. C. (2024). Integrating transformative considerations and quantitative results through a participant selection joint display in explanatory sequential mixed methods studies. Journal of Mixed Methods Research, 18(1), 14–30. https://doi.org/10.1177/15586898221149470
JohnsonR. B.OnwuegbuzieA. J.TurnerL. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689806298224
28.
JohnsonR. E.GroveA. L.ClarkeA. (2019). Pillar integration process: A joint display technique to integrate data in mixed methods research. Journal of Mixed Methods Research, 13(3), 301–320. https://doi.org/10.1177/1558689817743108
KaneM. T. (2021). Articulating a validity argument. In The Routledge handbook of language testing (pp. 32–47). Routledge.
32.
KoskeyK. L. K.SondergeldT. A.StewartV. C.PughK. J. (2018). Applying the mixed methods instrument development and construct validation process: The transformative experience questionnaire. Journal of Mixed Methods Research, 12(1), 95–122. https://doi.org/10.1177/1558689816633310
33.
KutscherE. L.HowardL. C. (2022). Integration as a process: Applying iterative multiple correspondence analysis to surface dynamic findings. Journal of Mixed Methods Research, 16(3), 328–349. https://doi.org/10.1177/15586898211021669
34.
LaneS.RaymondM. R.HaladynaT. M. (Eds.), (2016). Handbook of test development (pp. 3–18). Routledge.
35.
LauneanuM.HubleyA. M. (2017). Some observations on response processes research and its future theoretical and methodological directions. In Understanding and investigating response processes in validation research (pp. 93–113). Springer.
36.
LaveryM. R.BosticJ. D.KruseL.KrupaE. E.CarneyM. B. (2020). Argumentation surrounding argument-based validation: A systematic review of validation methodology in peer-reviewed articles. Educational Measurement: Issues and Practice, 39(4), 116–130. https://doi.org/10.1111/emip.12378
37.
LissitzR. W.SamuelsenK. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36(8), 437–448. https://doi.org/10.3102/0013189X07311286
38.
LoomisD.MaxwellJ. (2003). Mixed methods design: An alternative approach. In Handbook of mixed methods in social & behavioral research. Sage Publications.
39.
LynamT.DamayantiR.Rialine TitaleyC.SuharnoN.BradleyM.KrentelA. (2020). Reframing integration for mixed methods research. Journal of Mixed Methods Research, 14(3), 336–357. https://doi.org/10.1177/1558689819879352
40.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education and Macmillan.
41.
NewmanI.LimJ.PinedaF. (2013). Content validity using a mixed methods approach: Its application and development through the use of a table of specifications methodology. Journal of Mixed Methods Research, 7(3), 243–260. https://doi.org/10.1177/1558689813476922
NewtonP. E. (2019). What is response process validation evidence and how important is it? An essay reviewing Ercikan and Pellegrino (2017) and Zumbo and Hubley (2017). Assessment in Education: Principles, Policy & Practice, 26(2), 245–253. https://doi.org/10.1080/0969594X.2018.1447909
45.
Ng Yee PingD. (2019). Assessment of argumentation in chemistry: A model for designing items. Argumentation in Chemistry Education, 106–141. https://doi.org/10.1039/9781788012645-00106
46.
PareyB.KutscherE. (2023). Operationalizing and expanding the rigorous mixed methods framework for systematic reviews: The case of inclusive education. Journal of Mixed Methods Research, 0(0), 147–164. https://doi.org/10.1177/15586898231175192
47.
Plano ClarkV. L. (2019). Meaningful integration within mixed methods studies: Identifying why, what, when, and how. Contemporary Educational Psychology, 57, 106–111. https://doi.org/10.1016/j.cedpsych.2019.01.007
48.
SchoonenboomJ.JohnsonR. B. (2017). How to construct a mixed methods research design. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(Suppl 2), 107–131. https://doi.org/10.1007/s11577-017-0454-1
49.
ShawS.CrispV. (2012). An approach to validation: Developingand applying an approach for the validation of general qualifications. https://doi.org/10.17863/CAM.100449
50.
ShearB. R.ZumboB. D. (2014). What counts as evidence: A review of validity studies in educational and psychological measurement. In Validity and validation in social, behavioral, and health sciences (pp. 91–111). https://doi.org/10.1007/978-3-319-07794-9_6
51.
SireciS. G. (2013). Agreeing on validity arguments. Journal of Educational Measurement, 50(1), 99–104. https://doi.org/10.2307/23353800
TeddlieC.TashakkoriA. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage.
54.
WilsonM. (2004). Constructing measures: An item response modeling approach. Routledge.
55.
YangW. T.LinY. R.SheH. C.HuangK. Y. (2015). The effects of prior-knowledge and online learning approaches on students’ inquiry and argumentation abilities. International Journal of Science Education, 37(10), 1564–1589. https://doi.org/10.1080/09500693.2015.1045957
56.
ZhangJ.BrowneW. J. (2023a). An approach to generating guidelines for designing scientific argumentation competence assessments. In Fostering scientific citizenship in an uncertain world: Selected papers from the ESERA 2021 conference (pp. 201–218). Springer International Publishing. https://doi.org/10.1007/978-3-031-32225-9_13
57.
ZhangJ.BrowneW. J. (2023b). Exploring Chinese high school students' performance and perceptions of scientific argumentation by understanding it as a three-component progression of competencies. Journal of Research in Science Teaching, 60(4), 847–884. https://doi.org/10.1002/tea.21819
58.
ZhangJ.YuH.LvY. (2024). Scientific argumentation: Multi-dimensional capability model and educational value. Curriculum, Teaching Material and Method, 44(05), 124–130. https://doi.org/10.19877/j.cnki.kcjcjf.2024.05.011
59.
ZhouY. (2019). A mixed-methods model of scale development and validation analysis. Measurement: Interdisciplinary Research and Perspectives, 17(1), 38–47. https://doi.org/10.1080/15366367.2018.1479088
60.
ZumboB. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In The concept of validity: Revisions, new directions and applications, Oct, 2008. IAP Information Age Publishing.
61.
ZumboB. D.ChanE. K. (2014). Reflections on validation practices in the social, behavioral, and health sciences. In Validity and validation in social, behavioral, and health sciences (pp. 321–327). Springer International Publishing. https://doi.org/10.1007/978-3-319-07794-9_19
62.
ZumboB. D.PadillaJ. L. (2020). The interplay between survey research and psychometrics, with a focus on validity theory. In Advances in questionnaire design, development, evaluation and testing (pp. 593–612). https://doi.org/10.1002/9781119263685.ch24
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
