Tucked away towards the end of survey instruments, open-ended questions are often only examined post hoc as researchers search for strategically-placed quotations in support of their otherwise quantitatively-derived conclusions. Yet program evaluations can benefit from participants’ spontaneous insights on how their program might be improved, because these insights commonly reveal previously unforeseen aspects needing support or discouragement. This paper introduces a semantic text analysis technique for the analysis of survey data generated from open-ended questions of the form, “How can our program improve?” The technique is illustrated using data from faculty-participants in a National Science Foundation (NSF) program for the development of greater cooperation between industrial and academic researchers.
AgrestiA (1990) Categorical Data Analysis. New York: Wiley.
2.
BiksonTK (1996) Groupware at the World Bank. In: CiborraCU (ed.) Groupware and Teamwork - Invisible Aid or Technical Hindrance? Chichester, UK: Wiley, 145–83.
3.
BosN (2000) High School Students’ Critical Evaluation of Scientific Resources on the World Wide Web. Journal of Science Education and Technology9(2): 161–73.
4.
ChristieCAAzzamT (2004) What’s All the Talk about? Examining EVALTALK, An Evaluation Listserv. American Journal of Evaluation25(2): 219–34.
5.
DeweyJDMontrosseBESchröterDSullinsCDMattoxJR (2008) Evaluator Competencies - What’s Taught versus What’s Sought. American Journal of Evaluation, 29(3): 268–87.
6.
FarquharsonBJohnstonMBuggeC (2011) How People Present Symptoms to Health Services - A Theory-based Content Analysis. British Journal of General Practice61: 267–73. DOI:10.3399/bjgp11X567090.
7.
FierroLAChristieCA (2011) Understanding Evaluation training in Schools and Programs in Public Health. American Journal of Evaluation32(3): 448–468.
8.
FjermestadJHiltzSR (2001) Group Support Systems - A Descriptive Evaluation of Case and Field Studies. Journal of Management Information Systems17(3): 115–159.
9.
FreyKWidmerT (2011) Revising Swiss Policies - The Influence of Efficiency Analyses. American Journal of Evaluation32(4): 494–517.
10.
GenoveseJEC (2002) Cognitive Skills Valued by Educators - Historical Content Analysis of Testing in Ohio. Journal of Educational Research96(2): 101–114.
11.
GrannerMLSharpePABurroughsELFieldsRHallenbeckJ (2010) Newspaper Content Analysis in Evaluation of Community-based Participatory Project to Increase Physical Activity. Health Education Research25(4): 656–67. DOI:10.1093/her/cyp049.
HarnarMAPreskillH (2007) Evaluators’ Description of Process Use - An Exploratory Study. New Directions for Evaluation (2007) 116: 27–44.
14.
HesselberthJF (1991) Consortia - Making Them Work. ChemTech21(5): 271–273.
15.
IllmanDL (1994) NSF Celebrates 20 years of Industry-university Cooperative Research. Chemistry and Engineering News72(4): 25–30.
16.
JacobsonMRAzzamTBaezJG (2013) The Nature and Frequency of Inclusion of People with Disabilities in Program Evaluation. American Journal of Evaluation34(1): 23–44.
17.
Joseph-WilliamsNEdwardsAElwynG (2010) The Importance and Complexity of Regret in the Measurement of “Good” Decisions - A Systematic Review and a Content Analysis of Existing Assessment Instruments. Health Expectations14(1): 59–83.
18.
KittoRJBarnettJ (2007) Analysis of Thin Online Interview Data - Toward a Sequential Hierarchical Language-based Approach. American Journal of Evaluation28(3): 356–368.
19.
MahaffyKACaffreyEM (2003) Are Requests for Teaching Credentials Customary? A Content Analysis of the 1999 Employment Bulletin. Teaching Sociology31(2): 203–11.
20.
MarraRMMooreJLKlimczakAK (2004) Content Analysis of Online Discussion Forums - A Comparative Analysis of Protocols. Educational Technology Research and Development52(2): 23–40.
21.
NewmanDRWebbBCochraneC (1995) A Content Analysis Method to Measure Critical Thinking in Face-to-face and Computer Supported Group Learning. Interpersonal Computing and Technology3(2): 56–77.
22.
NúñezMSanchezANúñezECasalsTAlegreCMuñoz-GomezJ (2006) Patients’ Perceptions of Health Related Quality of Life in Rheumatoid Arthritis and Chronic Low Back Pain. Quality of Life Research15(1): 93–102.
23.
PeckLRKimYLucioJ (2012) An Empirical Examination of Validity in Evaluation. American Journal of Evaluation33(3): 350–365.
PoppingR (2015) Analyzing Open-ended Questions by Means of Text Analysis Procedures. Bulletin of Sociological Methodology28: 28–39.
26.
RobertsCW (2015) Content Analysis. In: WrightJD (ed.) International Encyclopedia of the Social & Behavioral Sciences, 2nd Ed. Oxford: Elsevier, vol. 4, 769–73.
27.
RobertsCWZuellCLandmannJWangY (2010) Modality Analysis - A Semantic Grammar for Imputations of Intentionality in Texts. Quality & Quantity44(2): 239–257. DOI:10.1007/s11135-008-9194-7.
28.
TrochimWMMarcusSEMâsseLCMoserRPWeldPC (2008) The Evaluation of Large Research Initiatives - A Participatory Integrative Mixed-methods Approach. American Journal of Evaluation29(1): 8–28.
29.
Van BenthemJ (2010) Modal Logic for Open Minds. Stanford: CSLI Publications.
