Abstract
Objective
Asking students to answer 30 + questions in an optional student evaluation of teaching (SET) increases the likelihood that instructors may not receive critical feedback due to survey fatigue. A considerable body of educational research on how to incentivize student completion of SETs offers a variety of suggestions to combat fatigue, perceptions of value, and other issues.
Methods
This retrospective cohort study from January 2021 to March 2024 investigates the impact of a shortened, 4-question mandatory SET on response rates and qualitative feedback within a Clinical and Translational Science (CTS) program. Unadjusted linear mixed-effects regression models assessed the relationships of SET type (optional SET vs experimental SET) and SET response outcomes.
Results
Over the course of the study, 844 students completed optional (N = 473) and experimental (N = 371) SETs. Requiring SET completion to access assessments increased both participation (Estimate 62.43, 95% CI: 52.41-72.45; P < .001) and qualitative responses (Est. 43.65, 95% CI: 32.24-55.05; P < .001). Relative to the optional SET, the experimental SET was associated with more blank qualitative responses (Est. 7.28, 95% CI: 4.70-9.86; P < .001) and a higher total word count (Est. 989.59, 95% CI: 513.78-1465.40; P < .001). Course satisfaction scores did not differ significantly between survey types (Est. 0.05, 95% CI: −0.13-0.22; P = .60).
Conclusion
This study's findings demonstrate that short, mandatory, learning management system-integrated SETs effectively increase qualitative feedback and can offer a feasible model for institutions aiming to gather actionable items to inform course design and delivery.
Introduction
End-of-course surveys, often called student evaluation of teaching (SET), are widely used to provide instructors with feedback intended to improve course content and delivery across higher education, including Clinical and Translational Science (CTS programs). 1 The digitalization of SETs in the early 2000s reduced administrative burden but coincided with a decline in response rates relative to paper-based surveys.2,3 In examining this trend, researchers have identified multiple barriers to participation, including survey length, frequency, question structure, and students’ perception of the usefulness of SETs.4,5 These challenges have prompted institutions to adopt a variety of strategies aimed at increasing student participation in SETs.
Incentives have been the most extensively studied approach to improving SET response rates. Extra credit for survey completion has been shown to increase participation across multiple studies.2,5–10 However, the use of extra credit raises concerns regarding grading policy compliance 6 or ethical practice. 2 In addition, evidence indicates that extra credit incentives may influence evaluation scores, potentially inflating course ratings. 11 To reduce these risks, some studies have examined micro-extra credit incentives, which constitute a minimal portion of the overall course grade and have demonstrated positive effects on response rates.2,9,10,12,13 Comparative analyses have further shown micro-extra credit to be more effective than other approaches such as instructor motivation alone or instructor motivation coupled with feedback. 12 Other incentive-based strategies, such as note cards for the final exam or setting a threshold for overall class-wide participation thresholds have also shown promise. 7
Non-incentive strategies have also been explored as mechanisms to increase participation. Instructor engagement during class time, including the discussion of the purpose and value of SETs, have been associated with improved response rates.6,7 The use of instructional time for students to complete SETs similarly increase participation. 5 Survey structure and length further influence completion; a systematic review found that shorter surveys with simplified rating scales were more likely to be completed. 9 Despite these findings, increases in response rates do not necessarily yield more substantive qualitative feedback, limiting the utility of SETs for instructional improvement. 12
The limitations of SETs have been widely discussed. Researchers have raised concerns about the reliability of SETs, particularly when used in high-stakes decision-making contexts such as tenure. 14 Prior research examined both the quality of SET responses 15 and the motivations underlying student participation. 8 Complicating the debate of reliability and low response rates is the correlation between grade inflation and higher evaluation scores. 16 Accounting for all the challenges of using SETs, these findings highlight the challenges of using SETs for evaluative purposes and turn the focus to the importance of aligning survey design, administration, and goals.
To address these challenges, we designed and evaluated a revised approach to SETs that prioritize participation and qualitative feedback while maintaining formative focus. At our institution, the CTS education program uses SETs solely as formative assessments to guide improvements in course design, delivery, and instructional methods. Therefore, this research study sought to improve response rates and qualitative feedback by shortening the SET from 35 questions to 4 and modifying its delivery. The 4-question experimental SET emphasized qualitative responses and required anonymous submission through the learning management system (LMS) prior to accessing a course assessment. By evaluating this approach, the study addresses a gap in the literature regarding the combined impact of survey length reduction and mandatory completing on qualitative feedback. Research for this study was approved by our institution's Education Research Committee and did not require further IRB review. Data was anonymous prior to analysis, and all data was stored on a secure internal institutional server with access limited to the study team directly using the data.
Justification of Study
Within this study, we sought to discover whether reducing the number of questions and restricting access to a course assessment (typically the final assessment) in a shorter experimental SET would produce more qualitative feedback than a longer optional SET (Supplemental File 1) that was used within all courses of our institution. Within the institution-wide optional SET (henceforth referred to as optional SET), we observed a lower ratio of students who completed the survey compared with how many students were enrolled in the course. In addition to a low response rate, our team believed the optional SET length of 35 questions was also problematic as cited in previous publications. 4 Furthermore, intermittent errors in survey question phrasing occurred due to the length of the optional SET. An example of one of these errors was a question asking students to evaluate an LMS that was no longer in use by the institution. Additionally, as reported by McAuley et al, 5 the timing of this optional SET was not tailored to the course end and often was delivered weeks after the course had ended, leading to a delay in results. Course faculty and the education support team expressed that the delay led to an impersonal and disconnected feel when CTS courses were evaluated using the optional SET. Additionally, the optional SET was delivered from a shared inbox email address, which can have a reduced impact when compared with surveys delivered from a known source.6,7 Lastly, faculty and the education support team had no control over the optional SET and how it was administered other than reporting errors after survey dissemination. Data collection was completed by the institutional survey research center, which then provided a report. This led to the ideation of an experimental SET (Supplemental File 1) to remedy low response rates, lack of qualitative feedback, and delivery challenges. The SETs, both optional and experimental, are utilized for the review and improvement of each individual course and do not determine tenure or other promotional metrics for the faculty as teaching is a secondary or tertiary role for all faculty within our institution.
Materials and Methods
Study Design
In response to these identified problems, we developed an experimental SET. We reduced the survey length, aligned the survey window with course completions, and embedded the survey within the LMS to streamline access to the SET. In developing the experimental SET, we reduced the number of questions from 35 (32 qualitative and 3 quantitative) to 4 (3 qualitative questions and 1 quantitative). We maintained 1 quantitative question (Question 4 below) from the optional SET (Supplemental File 1) to compare historical data when discussing results with instructors. We reduced the original 35 questions to the following for the experimental SET (Supplemental File 2):
What did you find valuable about this course? What would you like to learn more about? What aspects would you change about this course? What grade would you give this course? (0 = F, 4 = A)
These questions encompassed the themes analyzed within the optional SET. We piloted the experimental SET within 2 established courses as replacement for the optional SET. The experimental SET pilot required students to complete the SET to access the final assessment, as noted in a previous experiment on SET revision. 17 We controlled all aspects of data collection by utilizing the embedded anonymous survey function, release condition settings, and automatic reminders within the LMS to ensure students understood SET release and completion parameters. Additionally, instructors used in-class time to explain the new process and provided a 1-week window for students to complete the survey. All questions within the experimental SET were optional. Students could open the experimental SET, answer no, some, or all questions, and submit it to access their assessment submission during the 1-week window prior to the final assessment submission deadline. This maintained flexibility in student completion of the experimental SET and final assessment. Upon reviewing results showing increased completion and an increase in optional qualitative feedback from students within the pilot, we implemented the experimental SET process in 4 additional CTS courses to produce the results of this study. To assess the differences in responses by SET type, we conducted a retrospective cohort study.
Setting
This study, from January 2021 through March 2024, included 6 courses required as part of a graduate level CTS education program within a medical research institution. Students enrolled in these courses were in predoctoral, dual degree, master's and certificate programs. Demographics were not collected or analyzed as part of this study. All courses were conducted through an LMS virtual classroom and were selected and included in this study based on consistency in the following: student enrollment numbers, content, faculty, and calendar offering.
Participants
Inclusion criteria for this study was all responses from all learners who completed the surveys in the 6 courses, both optional and experimental, during the time frame of the study. Course subjects focused on foundational concepts within Clinical and Translational Science: study design/methodology, biostatistics, and epidemiology. All courses in the study were required for program completion. There was no exclusion criteria for this study. Both the optional SET and the experimental SET were fully anonymized within the data collection methods, thereby removing any ethical issues related to identifying survey respondents. Across the 6 courses studied, 35 total course iterations with 844 SET responses were included to compare the optional SET with the experimental SET.
Variables
The study outcomes are as follows: percent enrolled and SET completed, number of qualitative statements, number of blank statements (ie “-” and “n/a”), total word count, median word count per statement or question in the SET, median word count per SET response overall, and course score (0 through 4 representing F-A). The exposure of interest was SET type. Given the exploratory nature of this research and the lack of demographics data, confounding variables were not considered.
Data Sources/Measurement
Data from the optional SET was collected by the internal Survey Research Center at our institution. Data from the experimental SET was collected using the LMS and exported by CTS Education program staff.
Bias
Using a convenience sample, we limited selection bias for this study. Data was collected anonymously, limiting researcher bias. Due to the anonymous nature of the data, we were unable to collect any potential confounders to adjust for in the analysis.
Study Size
The study size was based on a convenience sample during the study period of interest.
Statistical Analysis
Unadjusted linear mixed-effects regression models assessed the relationships of SET type (optional SET vs experimental SET) and SET response outcomes. SET response outcomes included: percent enrolled and survey completed, qualitative statements, blank statements, total word count, median word count per statement, median word count per response, and course score. To account for clustering within individual courses, all models were fitted with a random intercept at the course level.
There was no missing data. Two-tailed P-values of .05 or less were considered statistically significant. Data management and statistical analysis were performed in SAS version 9.4 (SAS Institute Inc, Cary, North Carolina). The reporting of this study conforms to the STROBE statement (Supplemental File 3) for cohort studies. 18
Results
Table 1 provides descriptive summaries of outcomes by SET type for all 844 students included in the study. Additionally, Table 1 includes regression model estimates, confidence intervals (CIs), and P-values. A higher percentage of enrolled students completed the experimental SET (E. SET) versus the optional SET (O. SET) (Estimate = 62.43, 95% CI: 52.41-72.45; P < .001) (Table 1).
Analysis Results Comparing Experimental and Optional SET's.
Overall n = 844 students. All values are presented as median (IQR). The reference group for all models was optional SET.
The E. SET had a mean of 43.65 more qualitative statements compared to O. SET (95% CI: 32.24-55.05; P < .001) (Figure 1). The number of blank statements among E. SET respondents was higher than among O. SET respondents (Est. 7.28, 95% CI: 4.70-9.86; P < .001). The mean total word count was 989.59 words more with the E. SET survey versus O. SET (95% CI: 513.78-1465.40; P < .001). The median word count per statement was slightly lower in the E. SET responses compared to O. SET responses, though not statistically significant (Est. −4.58, 95% CI: −9.25-0.09; P = .054). The median word count per response (P = .61) and course score (P = .60) were not significantly different between SETs.
Boxplots display distribution of number of qualitative statements by course (n = 6) by SET type (experimental or optional). Circles within each boxplot represent the mean, lines within represents the median, bottoms and tops represent quartiles 1 and 3, respectively, and whiskers represent the minimum and maximum.
Discussion
The implementation of a shorter, experimental SET within the LMS showed a significant increase in participation and qualitative feedback across all 6 of the courses studied. These results echo previous studies as they demonstrate implementation of a shortened SET approach reduces barriers to completion.4,5 The reduction of questions as well as the inclusion of the experimental SET within the LMS played a role in increasing qualitative feedback as well as offered a consistency in survey availability and access across multiple courses and quarters. These findings offer valuable insights into improved SET delivery and completion rates.
The data demonstrated that the percentage of students who completed the experimental SET was significantly higher than the optional SET, an expected finding given the mandatory completion requirement. However, the meaning of this result is evident in the characteristics of the completed surveys. While mandatory submission drove higher participation, students submitted more qualitative statements across all courses, and total word count was higher in 5 of the 6 courses studied, suggesting that reduced survey lengths may have supported greater willingness to provide written feedback. These findings align with those found by Gupta et al, 19 who similarly reported increased responses with shortened SETs, supporting the study hypothesis. However, the number of blank statements within the surveys increased when comparing the experimental SET to the optional SET. This is explained by the mandatory condition that students submit the experimental SET to access a course assessment. The optional SET had no completion requirement and therefore could be deleted or ignored. The experimental SET required the student to open the survey but did not require a minimum word count to submit. Responses where the students could have written “N/A,” “None,” “- “or a similar non-response instead of qualitative statements were counted as blank statements in this study. One explanation for these non-responses may have been timing as SETs were released in the last week of the course term where students could be very busy. As noted in Sullivan et al, 9 timing SET dissemination with busy times for students negatively impacts response rates.
Total word count doubled in the experimental SET as a result of the mandatory conditions programmed into the LMS. Median word count per statement was not significantly lower in the experimental SET compared to the optional SET. Median word count per response was slightly higher for the experimental SET, though not statistically different when compared to the optional SET. However, the variability of the CIs decreased for the experimental SET, potentially reflecting more uniformity of overall survey responses. This confirms, at least peripherally, the findings of Jaquett et al 8 that students are more likely to submit surveys when they associate the course with positive characteristics such as easy tests, high standards, and good grades.
The increased response rates and the higher number of qualitative statements, particularly those with higher median word count per response from the experimental SET, provide faculty with more feedback overall. Increased feedback can lead to targeted improvements, potentially improving course design and student satisfaction in future course offerings. Faculty aligned with the courses studied have actively implemented formative qualitative feedback that was actionable from the experimental SET, suggesting that increased feedback fosters faculty action.
Limitations
The experimental SET completion was required to access an assessment, ranging in significance from the last quiz in a course to a final exam or project. Although forced participation may have influenced results, the minor inconvenience of a few extra clicks provided more overall feedback. In circumstances where the additional statements may have been neutral in tone (ie, “course was ok”), this information still holds more value than distraction. In general, course faculty would rather receive any comment versus none at all. By ignoring the experimental SET, students may have missed completing these assessments and therefore their final grade may have been impacted.
Although the experimental SET was anonymous, it was programmed within the LMS. Therefore, students could have felt pressure to respond more positively than they would within the optional SET which was administered via a generic email inbox and gathered by a third-party survey research center. The study team recognizes the concerns suggested by Medina et al 1 and Aoun Bahous et al 17 that mandatory SET completion could impact reliability and validity of results. To mitigate these concerns there was no word count requirement for the experimental SET as students were allowed to submit SETs with non-responses (eg, “N/A,” “none,” or “-,” etc.). Due to the study design, the independent effects of survey length and delivery mechanism could not be examined separately. Due to the exploratory nature of this study, with no established meaningful difference threshold for SET feedback, a sample size calculation was not performed.
The experimental SET questions were intentionally broad and concise to allow students minimal guardrails in their responses. This structure did not guarantee feedback related to any particular aspect, such as instructor quality, course material, or assessment. However, the focus was to encourage freedom to respond to topics the student found most salient. Although recall bias is possible given the 12-week term, the experimental SET likely mitigated this risk because survey access and submission were standardized at the end of the quarter, unlike the optional SET, which had variable access and submission windows. The mandatory submission requirement of the experimental SET resulted in a higher number of blank qualitative statements compared to the optional SET. The increase observed in the experimental SET highlights an important trade-off between maximizing participation and optimizing completeness response. While mandatory submission increased overall feedback volume, some students chose not to provide written comments when no minimum response was required. Although an increase in blank responses may contribute to information bias, it also captures question nonresponse that remains unobserved when participation is optional. In the optional SET, students may choose not to complete the survey at all, and this form of nonresponse cannot be quantified.
Future Directions
The courses included in this study are required for many programs outside of CTS within our academic institution. A future study could be expanded to include elective courses or to partner with another institution with similar structures, demographics, and programming to replicate the study. Additional research could include collecting the same data points within this study and comparing courses where the experimental SET is not required to open a final assessment to where the experimental SET is required to open a final assessment. This would add an additional experimental group to compare with the optional SET and could yield results to compare completion percentage and qualitative statements between the 2 groups. Institutions seeking to implement this strategy may mitigate blank responses while preserving higher completion rated by timing the SET to avoid peak academic workload, provide more reflective prompting, or further communicating how qualitative feedback is used in the course. Such strategies may improve response depth without reintroducing barriers to SET completion. And lastly, future iterations could compare the amount of constructive student feedback implemented by faculty when comparing optional SET to the experimental SET, although value in each item would be difficult to quantify.
Conclusion
Targeted refinements of shorter, mandatory SETs integrated within the LMS and timed consistently at course conclusion enabled the collection of formative feedback from a greater proportion of enrolled students, supporting more information for course improvements. Although this study was conducted within a CTS graduate program at a single institution, the results likely reflect broadly applicable mechanisms to other graduate programs. Reduced SET burden, standardized timing, and LMS-embedded delivery are transferable and scalable to any graduate level program focused on improving formative course feedback.
Supplemental Material
sj-pdf-1-mde-10.1177_23821205261431559 - Supplemental material for Maximizing Feedback and Reducing Survey Length: A Retrospective Cohort Study of Student Evaluation of Teaching
Supplemental material, sj-pdf-1-mde-10.1177_23821205261431559 for Maximizing Feedback and Reducing Survey Length: A Retrospective Cohort Study of Student Evaluation of Teaching by Kristina B. Nelson, Ryan C. Jimison and Allison M. LeMahieu in Journal of Medical Education and Curricular Development
Supplemental Material
sj-pdf-2-mde-10.1177_23821205261431559 - Supplemental material for Maximizing Feedback and Reducing Survey Length: A Retrospective Cohort Study of Student Evaluation of Teaching
Supplemental material, sj-pdf-2-mde-10.1177_23821205261431559 for Maximizing Feedback and Reducing Survey Length: A Retrospective Cohort Study of Student Evaluation of Teaching by Kristina B. Nelson, Ryan C. Jimison and Allison M. LeMahieu in Journal of Medical Education and Curricular Development
Supplemental Material
sj-pdf-3-mde-10.1177_23821205261431559 - Supplemental material for Maximizing Feedback and Reducing Survey Length: A Retrospective Cohort Study of Student Evaluation of Teaching
Supplemental material, sj-pdf-3-mde-10.1177_23821205261431559 for Maximizing Feedback and Reducing Survey Length: A Retrospective Cohort Study of Student Evaluation of Teaching by Kristina B. Nelson, Ryan C. Jimison and Allison M. LeMahieu in Journal of Medical Education and Curricular Development
Footnotes
Ethics Approval and Consent to Participate
Research for this study was approved by our institution's Education Research Committee. Research for this study was also submitted to our institution's IRBe Human Subjects Research Wizard tool, and in accordance with the Code of Federal Regulations, 45 CFR 46.102, the research activity does not require IRB Review.
Author Contributions
Conceptualization: RJ, KN. Data curation: KN, RJ. Methodology/formal analysis/validation: KN, RJ, AL. Project administration: KN, RJ. Funding acquisition: RJ, KN. Writing—original draft: KN, RJ, AL. Writing—review & editing: KN, RJ, AL.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This publication [or project] was supported by Grant Number UL1 TR002377 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
