This article reflects on the challenges of establishing trustworthiness in the evaluation of a curricular intervention funded by the National Science Foundation’s Innovations in Graduate Education (IGE) program. Relying primarily on qualitative data, the evaluation aimed to assess how STEM (Science, Technology, Engineering, and Mathematics) graduate students developed creative and collaborative capacities through their engagement in a range of arts-infused activities. Data were supplemented by pre- and post-tests of cognitive flexibility. The authors discuss how contextual features influenced both the program’s implementation and outcomes and complicated efforts to report transferable findings. Authors argue that instead of prescriptive results, qualitative research provides insights that inspire adaptation and innovation in other settings. Trustworthy findings involve reporting what happened during design and implementation phases and acknowledging limitations. The study highlights that admitting to uncertainties and limitations contributes to the generation of further research questions and supports continued inquiry into effective educational interventions.
BehrensJ. P.OlteţeanuA. (2020). Are all Remote Associates Tests equal? An overview of the Remote Associates Test in different languages. Frontiers in Psychology, 11, Article 1125. https://doi.org/10.3389/fpsyg.2020.01125
5.
DazaS. L. (2012). Complicity as infiltration: The (Im)possibilities of research with/in NSF engineering grants in the age of neoliberal scientism. Qualitative Inquiry, 18(9), 773–786. https://doi.org/10.1177/1077800412453021
6.
FioreS. M.SchoolerJ. W. (2004). Process mapping and shared cognition: Teamwork and the development of shared problem models. In SalasE.FioreS.M. (Eds.), Team cognition: Understanding the factors that drive process and performance (pp. 133–152). American Psychological Association.
7.
GlaserB. G.StraussA. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine Publishing.
8.
Hesse-BiberS. (2016). Doing interdisciplinary mixed methods health care research: Working the boundaries, tensions, and synergistic potential of team-based research. Qualitative Health Research, 26(5), 649–658. https://doi.org/10.1177/1049732316634304
9.
HollandP. W. (1986). Statistics and causal inference. Journal of the American Statistics Association, 81, 945–970.
LincolnY. S.GubaE. G. (1985). Naturalistic inquiry. Sage.
13.
LynchM. (1991). Method: Measurement—ordinary and scientific measurement as ethnomethodological phenomena. In ButtonG. (Ed.), Ethnomethodology and the human sciences (pp. 77–108). Cambridge University Press.
14.
National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Committee on Science, Engineering, Medicine, and Public Policy; Board on Research Data and Information; Division on Engineering and Physical Sciences; Committee on Applied and Theoretical Statistics; Board on Mathematical Sciences and Analytics; Division on Earth and Life Studies; Nuclear and Radiation Studies Board; Division of Behavioral and Social Sciences and Education; Committee on National Statistics; Board on Behavioral, Cognitive, and Sensory Sciences; Committee on Reproducibility and Replicability in Science. (2019). Reproducibility and replicability in science. National Academies Press. https://www.ncbi.nlm.nih.gov/books/NBK547537/
RosaH. (2020). The uncontrollability of the world. Polity Press.
17.
SalazarM. R.LantT. K.FioreS. M.SalasE. (2012). Facilitating innovation in diverse science teams through integrative capacity. Small Group Research, 43(5), 527–558.
18.
SchmidtS. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108
19.
SchneiderB.CarnoyM.KilpatrickJ.SchmidtW. H.ShavelsonR. J. (2007). Estimating causal effects: Using experimental and observational designs [report from the Governing Board of the American Educational Research Association Grants Program]. American Educational Research Association.
20.
ServickK. (2018). Cornell nutrition scientist resigns after retractions and research misconduct finding. Science Adviser. https://doi.org/10.1126/science.aav4979
21.
ShamooA. E.ResnikD. B. (2022). Responsible conduct of research (3rd ed.). Oxford University Press.
22.
ShavelsonR. J.TowneL. (Eds.). (2002). Scientific research in education. National Academy Press.
23.
StarS. L.GriesemerJ. R. (1989). Institutional ecology, translations, and boundary objects. Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-1939. Social Studies of Science, 19(3), 387–420.
24.
St. PierreE. (2011). Post qualitative research: The critique and the coming after. In DenzinN. K.LincolnY. S. (Eds.), The SAGE handbook of qualitative research (4th ed.; pp. 611–625). Sage.
25.
TracyS. J. (2010). Qualitative quality: Eight “Big-Tent” criteria for excellent qualitative research. Qualitative Inquiry, 16(10), 837–851. https://doi.org/10.1177/1077800410383121
WebbE. J.CampbellD. T.SchwartzR. D.SechrestL. (1966). Unobtrusive measures: Nonreactive research in the social sciences. Rand McNally College Publishing Company.
28.
WolgemuthJ. R.JordanL. S. (2023). “Will you be our qualitative methodologist?” Reflections on grant work responsibilities. International Journal of Qualitative Methods, 22, 160940692311524. https://doi.org/10.1177/16094069231152452
