A Case Study of the Implementation of Non-linear Assessment Protocols for Pre-service Music Teachers

Abstract

Grounded in previous research regarding admission and graduation standards of music education undergraduate candidates, we created a framework to assess student readiness to enter the music education profession across multiple points of inquiry. To determine the efficacy of the protocols, we conducted a case study to pilot implementation of the protocols with two university music education programs in the United States. Two of the authors–music education faculty at these universities–served as site coordinators implementing the protocols with multiple student cohorts (comprised of first-year, midpoint, and graduates) within their institution. The authors reviewed the reports employing a three-step process. In the first step, a third independent author reviewed both reports while each site coordinator reviewed the other institution’s report. In the second step, the two site reviewers compared their reviews with the other author’s report leading to a “composite review.” In the third step, all three reviewers discussed the two composite reviews to determine recommendations for refining the protocols without disrupting collection of longitudinal data of their pre-service cohorts. Following final analysis, we revised the framework to better fit a full spectrum of universities and align with current best practices in music teacher education.

Keywords

assessment curriculum fairness music teacher education program evaluation reliability validity

Introduction

To obtain teacher certification, university music programs demonstrate how students meet the licensure standards of their respective U.S. state. While assessment protocols of music education programs exist, most literature focuses on student learning outcomes, tasks, or points within a given curriculum rather than a holistic picture (Hollingsworth et al., 2024). For many researchers, identifying points of alignment remains difficult due to variations in curriculum (Ward & Payne, 2018) or nomenclature (Payne et al. 2023). Ward & Payne (2018) found that while universities varied widely on required tasks, they reported three primary points of assessment: admission/auditions, mid-point, and licensure/graduation. Subsequent studies (Ward & Payne, 2020, 2024) revealed that assessment from admission to graduation suggested three primary areas of consideration: (a) musical skills and knowledge, (b) pedagogical skills and knowledge, and (c) professional dispositions.

The first area of consideration, musical skills and knowledge, is initially measured through an admission audition (Brand, 1987; Colwell, 2006; Kelly, 1988; Motycka, 1969; Royston & Springer, 2015; Shellahamer, 1984). In addition to performance skills, institutions include music theory and aural skills assessments, letters of recommendation, and interviews that address musical skills of students as a part of their admission and audition process (Payne & Ward, 2020). These additional assessments were often comprised of teacher-generated questions with little reporting of reliability or connection to the program outside of course placement. While program representatives found this sufficient for admission criteria, little evidence existed on how that data informed an institution’s assessment of a student’s achievement as the student continued in the program. As a result, Ward & Payne (2024) developed a framework to assess musical learning outcomes throughout a student’s undergraduate experience.

Few programs measure the second area of consideration, pedagogical skills and knowledge, at the point of admission (Ward & Payne, 2020). Only six out of 158 responding institutions evaluated pedagogy by asking auditionees to evaluate or respond to a musical performance or recording (Ward & Payne, 2020). Introduction to and experience with pedagogy appears to increase throughout a program as students enroll in methods and techniques courses (Ward & Payne, 2024). Music teacher education candidates are expected to develop the knowledge and skills necessary to listen, evaluate, and prescribe actions based on their experiences in the classroom.

Payne et al. (2023) found that the lack of a shared language interferes with the clarity by which external assessors can determine the extent to which these pedagogical skills exist within course work settings. For example, “methods” and “techniques” courses are defined differently by institutions but often used interchangeably within discourse. Timothy Brophy and Martin Fautley address the lack of shared language as it relates to assessment among university music programs in the “International Principles of Assessment in Music Education” that were posited at the Sixth International Symposium on Assessment in Music Education in 2017 (Booth et al., 2018). They point to an example that highlights this issue is the difference between student learning assessment and program assessment. While the term and practice of assessment are used in both instances, how the entity collects, analyzes, and makes data-informed decisions will often vary, making the shared definitions critical in the dissemination of findings. Given the lack of shared language, researchers offer recommendations for how institutions introduce and assess knowledge, skills, and approaches to pedagogy within a program. The variance in their discussions necessitates programs to identify commonalities across institutions in defining terms or establishing shared language.

The third area of consideration focuses on professional dispositions—personal traits or characteristics necessary for professional success. Desirable dispositions for music educators include: confidence, communication skills, commitment to music education, cooperation, enthusiasm, flexibility, leadership, organization, personality, previous teaching experience, professionalism, punctuality, interest in teaching, self-control, sense of humor, reliability, responsibility for own learning, and contributions to the learning of others (Brand, 1987; Henry, et al, 2020; Hime et al., 2014). Music teacher educators considered professional dispositions a better indicator of success in student teaching than musical or pedagogical skills (Teachout, 1997; Wayman, 2006). U.S. federal and state agencies have identified the importance of educator dispositions, and the role played by educator preparation programs in developing professional dispositions (Parkes et al., 2019). Fisher et al. (2021) recommended “music teacher education programs consider implementing a type of professional disposition measurement throughout music education students' undergraduate education” (p. 394). Researchers explored the role of assessing professional dispositions for admission into a teacher education program (Alexander et al., 2022; Henry et al., 2020; Hollingsworth et al., 2024). Faculty of various expertise—academic, applied, and ensemble—rated teacher education candidates on a variety of professional dispositions (Henry et al., 2020). The faculty ratings were used as a part of the assessment for admission into the teacher education program, with a high correlation with eventual success in the program. Alexander et al. (2022) found that candidates consistently rated themselves lower on professional dispositions than faculty evaluators, particularly in areas related to teaching. They also found overall consistency in ratings across various types of faculty evaluators. Hollingsworth et al. (2024) found that preservice teacher self-evaluations revealed significant growth in dispositions between candidate admission into the teacher education program and post student-teaching/graduation. Given the robust exploration of professional dispositions and the importance placed on them by education agencies, professional dispositions merit a place in program-spanning assessment protocols for music education programs.

Creating a Framework

As a result of previous findings, Ward & Payne (2024) developed a framework through which programs could identify student achievement of learning outcomes in the areas of musical skills and knowledge, pedagogical skills and knowledge, and professional dispositions. The framework was developed using principles of Backward Design (Wiggins & McTighe, 2005, 2007), an instructional design paradigm that identifies learning objectives of anticipated instruction and works backward through assessment and instruction toward where students begin their academic journey. As achieving professional music teaching licensure is the primary objective for U.S. music teacher education programs, the authors identified primary points of assessment backward from licensure to admission to a university music program (Ward & Payne, 2018). The result was a framework for assessing the progress of student readiness entering the music education profession across multiple points of inquiry that included musical skills and knowledge, pedagogical skills and knowledge, and professional dispositions (Ward & Payne, 2024) that remained grounded in previous research regarding admission and graduation standards of music education undergraduate candidates (Ward & Payne, 2018, 2020).

In its entirety, the framework consisted of non-linear assessment protocols, aligned with outcomes of two major areas: university music education curriculum and professional teaching standards as defined by state teacher licensing agencies in the United States (Ward & Payne, 2024). As music education programs and various teacher licensing agencies may differ, the goal was to create protocols that are adaptable to maintain their content validity, relevancy, and alignment with program processes and expectations. The protocols should, however, maintain high degrees of reliability and fairness so that university music faculty and stakeholders can effectively and realistically analyze student progress across a variety of tasks. Once established, the researchers sought feedback on the structure and volunteers to pilot the framework. Once two pilot locations were identified, the authors focused on implementing the framework to allow for more intensive interrogation of the content and structure of the framework. An overview of the pilot framework can be found in Figure 1 and Appendix B.

Figure 1.

Pilot framework.

The framework presented here is designed to accommodate multiple options to monitor a student’s development throughout the undergraduate curriculum, including baseline data collected during the admissions process, and concluding with graduation and teaching licensure. Grounded in previous research (Ward & Payne, 2020, 2024), the protocols are divided into three distinct points: admissions, midpoint, and graduation and three distinct areas: musical skills and knowledge, pedagogical skills and knowledge, and professional dispositions. The intent of these protocols is to provide formative and holistic feedback for programs to support data-informed actions including adjusting courses, instruction, and data collection in meeting the needs of students as they progress through their undergraduate experience.

The primary purpose of this paper is to pilot this research-based framework while developing a set of protocols that can provide informative data to programs as it relates to music teacher education programs’ efficacy. We framed the following research questions to guide this current pilot study of the framework to aid us in revealing information that will lead us to refine this tool even further for increased effectiveness and utility.

(1) What commonalities exist between the two institutions who piloted this framework?

(2) What were the responses of the site coordinators regarding the content validity, potential reliability, and fairness of the current framework structure?

(3) What suggestions might the site coordinators have to continue refining this framework to allow for maximum applicability across a variety of institutional environments?

Method

Instrument

The admission protocols are designed to guide music education faculty in their mentorship of music teacher candidates as they navigate their selected curriculum and work to meet professional teaching standards upon graduation. Based on the protocols established by Ward and Payne (2024), we¹ piloted this framework to determine both its usefulness and its applicability. The protocols are divided into three primary areas (musical, pedagogical, and professional) assessed across three points of inquiry (admissions, midpoint, and graduation/licensure). The protocols employ broad descriptions to guide development and implementation of assessments leading to more informed, data-driven decisions within the curriculum and assessment process. An overview of the protocols from Ward and Payne (2024) piloted in the current study are provided in Appendix B.

Using admission protocols specific to each university, music faculty collect baseline data for comparison to future assessments administered at varying points during the curriculum up to and including graduation/licensure to analyze student progress. Once students have matriculated and begun completing music education curriculum, this program assessment framework allows for programs to implement their own student learning assessment protocols to measure individual learning, while aggregating these data to determine whether music education students are making adequate progress toward graduation outcomes. Payne and Ward (2020) found that most universities rarely use admission data beyond course placement and then often only report student learning outcome (SLO) data to accrediting agencies.

The mid-point assessment framework, typically during or at the end of a student’s second year, provides an opportunity to aggregate institutions’ student learning assessments into a mechanism for data-informed changes in policies and procedures. Additional benefits include identifying any points of adjustment with current assessments to ensure a comprehensive and accurate data set. Similar to admission protocols, these intermittent protocols will align with points in the teacher licensure process that signify satisfactory progress toward learning outcomes associated with the music education curriculum and demonstration of professional teaching standards.

Finally, graduation protocols serve as both summative assessments allowing students to demonstrate the stated outcomes defined by the music education faculty and the state licensing agency. These protocols also identified desired results (student skills, knowledge and dispositions) for students who successfully complete the music teacher education program.

Longitudinal Case Study

To determine the efficacy of the newly refined protocols (Ward & Payne, 2024), the authors conducted a pilot case study of the implementation of the protocols with two university music education programs in the United States. Two universities (one public university in the midwestern US, one private university in the southern US) served as sites for this investigation. Two of the authors, music education faculty at these universities, served as site coordinators implementing the protocols with multiple student cohorts (comprised of first-year, midpoint, and graduates) within their institution. The third author served as an independent reviewer of each site coordinator’s reports. Over the course of 5 years, site coordinators will complete author-designed annual reports (see Appendix A) to provide feedback on the efficacy of the protocols, specifically:

content validity of the protocols regarding their relevance to music education curriculum and state professional teaching standards;

reliability of the protocols and the use of them to collect data to predict success at the admissions point, progress toward graduation, and success in the teaching profession;

fairness of the framework to meet diverse student needs and the development appropriateness of individual protocols; and

data collected through the protocols and actions/treatments taken by music faculty in response to an analysis of the data.

Using a consensus moderation approach, the authors reviewed the initial/first-year reports through a three-step process. According to Sadler (2013), consensus moderation, an assessment method among multiple reviewers, is used to assess complex responses. Dr. Sadler writes:

When several assessors agree on a judgement but differ in their explanations, it may be that they have attended to different aspects of the work. Alternatively, they may have attended to the same aspects but come up with explanations that are structured or expressed differently. This is why one assessor’s “coverage” of an explanation of a judgement may have much the same “coverage” as another judge’s explanation which seemingly invokes different criteria. The main point is to share the essence of the reasons, which can subsequently be achieved through open discussion about particular cases (Sadler, 2013, p. 10).

Through the review and subsequent discussion of the site coordinator reports, we created agreed upon “criteria” for each of the reporting areas: content validity, reliability, fairness, and use of assessment data.

In the first step, the independent reviewer examined both reports and each site coordinator reviewed the other institution’s report. In the second step, each site reviewer examined their review with the independent reviewer leading toward a “composite review.” In the third step, all three reviewers discussed the two composite reviews to determine fit, efficacy, and recommendations to refine the protocols without disrupting collection of longitudinal data of their pre-service cohorts. Following the final review in this first year, we developed a set of critical adjustments and improvements to the protocol. Final publication of the protocols will be made at the end of the multi-year case study.

Results

We identified the alignment of the timing, fit, and appropriateness of framework with each of the institution’s current assessment practices. Additionally, the site reviewers responded to the usefulness, fairness, action items, and current assessment practices at each assessment point (admission, mid-point, and graduation). The following sections include the responses from the participating institutions and provide general thematic information for the subsequent discussion.

Content Validity

Examining the three stages of the protocol revealed that the admissions audition was a consistent element and aligned with current practice. Theory, aural skills, and keyboard diagnostic exams also seemed to be a consistent theme. We also noted that most of the data is rarely used once the audition process is complete. This finding led to a lengthier discussion of what might be advantageous to music education programs at this point in the student’s university experience.

The discussion about pedagogical and professional protocols during the admission point revealed some differences between the two university programs. Among the most noticeable differences was the delivery of the “Introduction to Music Education” course. While content and philosophical grounding remained similar, the implementation and delivery varied. Considerable discussion occurred to reconcile how these variations would impact the protocols. Discussion then moved to the midpoint assessment, focusing on the intent of the “barrier exams” or “proficiencies” at the various universities, including implementation and assessment. Finally, we centered discussions around pathways to licensure. Given that each U.S. state government varies widely on pathways and requirements, this discussion was extensive. In the end, we identified similarities and found that the content included in the framework aligned with the learning expectations of the two music education programs. As a result of the consensus between programs for the three stages of assessment, we confirmed the validity of content. The largest variances occurred within the delivery and sequence of course content. Given that the current protocol did not allow for this flexibility, we tabled these differences until our final discussion to aid in the development of our implications. Table 1 provides a synopsis of themes that arose from our discussions based on the site coordinator reports as they relate to content validity at the three curricular levels (admission, mid-point, and licensure/graduation).

Table 1.

Site Coordinator Responses to Content Validity.

Points	Private University	Public University
Admission	- Musical protocols align.	- Musical protocols align.
	- Audition is the primary decision point for admission.	- Audition is the primary decision point for -admission.
	- Theory, aural skills, and keyboard assessments are diagnostic and used for placement.	- Theory, aural skills, and keyboard assessments are diagnostic and used for placement.
	- Pedagogical and professional protocols are exploratory and embedded in coursework.	- Protocol language mirrors state licensure standards.
Mid-point	- Musical protocols align.	- Interview is closely aligned with teacher education program admission requirements.
Mid-point	- Keyboard proficiency requirement that is embedded in course work
Licensure/graduation	- Musical protocol aligns	- Aligns well with institution protocols and state licensing standards.
	- Do not have an instructional unit protocol but teaching opportunities in upper-level methods courses
	- Our institution has a digital portfolio
	- Cooperating teachers and field supervisors document proficiencies during student teaching internship.

Reliability

Evaluating reliability revealed some interesting points of discussion. Site coordinator reports focused on the reliability of the protocols in predicting student success in the curriculum and achievement of professional standards. Simultaneously, the site coordinators considered the reliability of student learning assessments as they mapped into this protocol. We then identified alignment issues or potential misunderstandings of terminology that might impact consistent application of the protocols over a variety of environments.

In this initial year, our discussions centered on the reliability and effectiveness of embedded assessments within the two university music programs while debating the merits of how they fit/aligned within the current protocols. While we found the internal student assessment practices of each university to be reliable, the variance in their application created a robust discussion for us. Therefore, we focused on how our processes aligned with the protocols rather than interrogating specific procedures. This led to the eventual emergence of points and phases as a better reflection of our collective processes, which we will discuss later in the paper. Table 2 provides a synopsis of themes that arose from our discussions based on the site coordinator reports as they relate to reliability at the three curricular levels (admission, mid-point, and licensure/graduation).

Table 2.

Site Coordinator Responses to Reliability.

Points	Private University	Public University
Admission	- Musical protocols are reliable for predicting ability for majoring in music	- Pedagogical and professional protocols occur in course over two music education courses (first year and second year)
	- Institution does not have a formal admission process for music education	- Presentation where they self-assess their own level of pedagogical and professional knowledge, skills, and dispositions
	- Discernment process for music education students occurs during first three semesters before allowing continuation in the program	- In pedagogical protocols, a third level of achievement should be added between “Continuation in Teacher Education” and “Retake Intro to Music Ed,” such as “Developing Freshman Level” to identify students who pass the course but have deficiencies
		- Protocol language mirrors state licensure standards
Mid-point	- Receives feedback of protocols from non-music education faculty	- Challenging to align coursework grades with professional learning outcomes
Mid-point		- Need to address variances in achievement, particularly in professional protocols and clarify the process of how students remediate deficiencies
Licensure/graduation	- Portfolio has been useful in assessing “hireability.”	- Need to re-label the “developing educator” level of achievement to “licensed educator” to recognize that minimum requirements for licensing have been met but there are areas for growth. This is where most licensure candidates would fall

Fairness

In considering fairness, we examined how each protocol fit within the two university programs. Our prompts solicited information regarding whether the protocols were “. . .fair in meeting the needs of our student populations from a diversity and developmentally appropriate standpoint (please provide examples, as applicable).” We found several items of interest in our initial reports: (a) protocols remained flexible, (b) they were easily applied, and (c) they were able to fit the two university music programs effectively. The curricula and assessments of the two university music programs met high standards of reliability and content validity as evidenced by accreditation of state agencies. The accreditation is due in part to the programs meeting the needs and educational goals across diverse populations. Because of the close alignment of the protocols with these two university music programs, we determined that the protocols allowed for a level of fairness as defined in Payne et al. (2019). Table 3 provides a synopsis of themes that arose from our discussions based on the site coordinator reports as they relate to fairness at the three curricular levels (admission, mid-point, and licensure/graduation).

Table 3.

Site Coordinator Responses to Fairness.

Points	Private University	Public University
Admission	- Protocols are individualized.	- Protocols are fair.
	- Options from remediation to acceleration are offered based on assessment data.	- Since admission is solely driven by the musical audition, there is the risk of denying potentially excellent music educators.
	- Protocols allow students to form their own priorities and philosophical tenets.
Mid-point	- Our institution has adjusted our language from “probation” to “re-evaluation, recognizing that dispositions develop at different rates for different students.	- A need to define and clearly communicate what “probation” means.
Mid-point		- Creating a “how to” manual would clearly communicate terminology and help faculty see parallels between the framework’s protocols and those used by the institution.
Licensure/graduation	- Highly individualized.	- Re-labeling the “developing educator” level of achievement to “licensed educator” would be a more accurate representation of most graduates.
	- Apprenticeship model of student teaching allows for students to meet their professional aspirations and is fair.
	- Student teaching evaluators adapt the focus of observation reports to match the student population being served.

Use of Assessment Data

Finally, the site coordinators reported “any actions/treatments taken by music faculty in response to assessment data collected based on the protocols.” The largest amount of discussion centered around this final inspection of the framework. The findings largely aligned with Ward and Payne (2020) in that assessment data at the audition phase is rarely used in data-informed decisions beyond admission. Besides course placement, diagnostic data collected at the admission point is rarely systematically used to inform faculty. This conclusion led us to explore how data might better inform scope and sequence of curriculum.

The conversation then shifted to the midpoint where the discussion focused primarily on the awareness of the students of their own learning throughout the process. We discussed the necessity for students to shift their identities from student to teacher (Raiber & Teachout, 2022). Rather than compliance with course work, were students aware of class engagement and activities as embedded assessments? Essentially, clarifying expectations and increasing transparency for all involved could create meaningful pathways for students moving through the midpoint assessments. As it relates to the final stage, graduation/licensure, we discussed the impact of changing state requirements, increased faculty loads, and shifting teaching assignments as factors in the adaptability of the framework. Table 4 provides a synopsis of themes that arose from our discussions based on the site coordinator reports as they relate to the use of assessment data at the three curricular levels (admission, mid-point, and licensure/graduation).

Table 4.

Site Coordinator Responses to Use of Assessment Data.

Points	Private University	Public University
Admission	- An addition of a “coachability” disposition.	- A need to determine if there is a correlation between the entrance music theory exam and success in the first two semesters of music theory.
Mid-point	- Continue to assess “coachability.”	- A need to define and clearly communicate expectations.
Licensure/graduation	- Create collaborative music-making opportunities by combining two student half-recitals in one evening that could lead to future collaborative work and peer teaching when the student is in the field.	- Music education faculty is not observing student teachers so there is a need to collect assessment data at this point.
	- Peer review of portfolios.
	- Public presentation on exceptionalities in response to a new state licensing standard.

Discussion

Assessment of music education program effectiveness benefits both music education programs and the students who matriculate through these programs. Given that most university music education programs in the U.S. lead to state-level teacher certification, we concluded that many commonalities existed in the assessment protocols across programs. Building on previous research on audition procedures, intermittent program assessments, and assessing professional dispositions, Ward and Payne (2024) created a framework of assessment protocols to track student progress from pre-admission to graduation/licensure. The present pilot study sought to apply those protocols in two different music education programs in two different U.S. states to determine the applicability of the protocols and potential generalizability of the framework.

Reports soliciting information about program alignment with the protocols were created by site coordinators at each program. Using a consensus moderation model where criteria emerge during the moderation of a work assessed, the three authors discussed the reports, focusing on several items including: (a) similarities and differences, (b) successes and challenges, and (c) potential improvements. As a result of the discussion, several common struggles or misunderstandings arose in the assessment of the protocols including: (a) alignment, (b) communications, and (c) recommendations for enhancement. This conversation led to recommendations for protocol revisions and for future research threads to continue refining this framework.

Alignment

Between Institutions

The site coordinators concluded that there was an overall alignment between the assessment protocol framework and their respective institutional approaches to assessment of music education students from all three perspectives (musical skills and knowledge, pedagogical skills and knowledge, and professional dispositions). Additionally, the site coordinators reported that the protocol framework for program assessment had enough flexibility to adapt to their curricular and state licensure requirements. It is notable that, while one institution is public and is generally impacted to a greater extent by state legislature and university systems, both institutions must align their curriculum and assessment protocols to state teacher licensing standards. Readers should note that teacher licensure standards, policies, and procedures are established at the state level in each U.S. state and vary widely from state to state. Given that these institutions are in two different states, the site coordinators indicated the applicability of these protocols as they related to their respective licensing requirements and all procedures required therein. This highlights a strength of the framework in its adaptability to multiple academic environments.

Role of the Certification Unit

A difference between the two institutions was the role of the certification unit (e.g. College of Education) in the music education curriculum and the decision to admit students into the teacher education program. As in many large U.S. institutions, the site coordinators are in institutions where music education faculty are in a school of music, which is a different academic unit than the college of education. In the private institution, all but one professional and pedagogical course was taught by School of Music faculty. In comparison, all professional and some pedagogical courses in the public institution were taught outside of the School of Music. Furthermore, the decision of admittance into teacher education was entirely at the discretion of the private institution music education faculty, while the certification unit determined student admission in the public institution with little input from music faculty.

While the authors are not suggesting that this approach to curriculum and admission to teacher education is indicative of private versus public institutions, it represents two models that affect the approach to curriculum sequence and remediation. Furthermore, it suggests that the framework can adapt for a variety of environments, policies, and procedures. Because the private institution had sole discretion as to whether a student was admitted to teacher education, their music education curriculum in the first 2 years allowed students to focus on music performance, theory, and aural and keyboard skills, while allowing students to determine whether they want to pursue a music education degree. Students continued enrolling in applied music courses in the last 2 years but shifted to a more pedagogical and professional focus. Because the music education faculty had implemented effective formative assessments throughout the curriculum, they could address deficiencies in their pedagogical and professional courses taken in the last years of the curriculum.

Likewise, the public institution’s approach to admission to teacher education influenced its curriculum and approach to remediation. In this case, music education faculty, while not making direct recommendations, have designed a course sequence that allows the students who demonstrate the knowledge, skills, and dispositions of successful music teachers to seek admittance into the teacher education program. This allows the music education program to retain a voice while communicating deficiencies of a candidate and remediate any deficiencies prior to admission. Of course, music faculty also have an opportunity to address these issues through their pedagogy courses in the School of Music; but since many of these second phase courses prior to student teaching occur outside the School of Music, it is vital that the two faculty communicate and collaborate.

Communications

Terminology

Based on our discussions, we recommend revising terminology in the framework to better reflect a greater spectrum of achievement. For example, at the mid-point, the site coordinators suggested the addition of “developing freshman level” as a middle level of achievement to recognize students that have passed an “Introduction to Music Education” course but still demonstrated pedagogical deficiencies. This allows a program to continue refining the knowledge, skills, and dispositions of future teachers while maintaining the passion and career aspirations of the students. Furthermore, this reflects the critical nature of embedded assessments within programs and how these align with the framework to ensure success throughout a program.

The site coordinators also suggested that the graduation/licensure “developing educator” level of achievement be re-labeled as “licensed educator” to recognize that minimum requirements for licensure have been met² but that there are areas for growth, noting that this is the best description of most licensure candidates at the end of their Educator Preparation Program.³ The students who complete a degree are licensed to teach. As with all learning, there is a range of achievement within those who have been identified as successful. This is critical to keep in mind throughout the implementation of this framework.

The review also revealed that some institutions use varying terms for the assessment protocols and levels of achievement in the framework. These variations may conflict with one another and lead to confusion. The site coordinators recommended the creations of a “how-to-manual” to crosswalk varying terminology used by faculty and to provide music faculty procedures for using the assessment protocol framework that can guide faculty curricular and assessment development.

Accessibility

We agreed that the revised protocol framework could serve as a useful communication tool for 4-year institutions to collaborate with 2-year community colleges and students transferring from other 4-year institutions. If a 4-year institution can demonstrate program expectations and how these expectations are assessed at the program level to a community college, better curricular and assessment alignment between institutions may be possible. Additionally, students who are considering transferring can compare their educational experiences to those in the protocol to determine their standing in the new institution.

Course-Embedded Protocols

The site coordinators identified some protocols that might be embedded in music education courses, while other protocols that existed outside coursework at the program level. Between the two university music programs, there were differences as to the location of individual protocols. Regardless of where these protocols are implemented, it is critical that faculty collaborate and clearly communicate to ensure consistency of assessment protocols and the use of this data to successfully employ the protocol framework. Regardless of when or where the assessments occur, the framework ensures that they are implemented in a manner consistent with program expectations. Faculty and administrators at local institutions should be sensitive to concerns of academic freedom and be flexible with varying approaches by instructors. Since assessment data is important to determine program effectiveness, as well as student readiness, faculty must work together in constructing, implementing, and analyzing the assessment process. This framework could provide the structure through which to organize these efforts. Expanded pilot studies and a wider implementation could provide further evidence of generalized applicability of the framework.

Recommendations for Enhancement

Similarities and Differences

We found that having three points of assessment [admission, mid-point, and graduation/licensure] (Ward & Payne, 2024) was not completely descriptive of current assessment paradigms in the two university music programs. While both institutions used the musical audition as the primary driver in making admission decisions to their music program and used diagnostic exams in theory, aural skills, and keyboard skills for course-level placement, pedagogical and professional learning outcomes were not assessed at the point of admission. Rather, faculty assessed these at various points throughout the early semesters to set a baseline. Given potential differences of when concepts are introduced and assessed, we noted that some concepts existed within a phase that encompassed the time between a student’s admission and the mid-point of their time in the program. Although some variance in approach existed between the two university music programs, this phase consisted of student self-reflection and discovery of their beliefs about teaching, initial teaching experiences, and development of professional skills, such as time management, use of technology, collaboration, and written and verbal communication. All of which led to a clear delineation at the mid-point assessment.

Both university music programs used the mid-point as a gateway for entrance into the institution’s teacher education program. This point of assessment allowed the programs to observe whether students maintained adequate progress in all three areas while also ensuring readiness for professional and musical development throughout the remainder of their undergraduate experience. In addition, both programs had procedures at this point wherein students with cited deficiencies were placed on “growth plans” to improve musical, pedagogical, or (most often) professional learning outcomes.

As with the audition/admission point, we uncovered both content and sequence differences between programs, as well as additional internal assessment procedures that added nuance to the application of this framework. While progress since audition/admission is charted at the mid-point assessment, substantial instruction in educational theory, pedagogy and practice, and dispositional habits occurs after the mid-point. We also agreed that the musical assessment protocols at admission and mid-point produced data that allowed faculty to make decisions related to course placement and were predictive of a student’s potential to successfully complete a degree recital at the end of their degree plan.

Specific Recommendations

Although we noticed several strengths throughout the initial pilot, we realized that these protocols are not specifically aligned with formative and self-assessment at the student level. Furthermore, the framework does not specifically address student learning at all but are a vehicle by which programs can organize and implement current assessment practices/protocols to improve analysis and reporting of curricular- and program-level data. Thus, we suggested the inclusion of phases to allow for more adaptability as it related to content, scope, and sequence. This allows programs to tailor their assessments to the environment in which they exist. In other words, the phases allow for programs to embed assessments within their respective experiences to ensure future success of the students through tracking student learning and adjusting of instruction within their given programs. Notably, the protocols, as currently designed, did not include the ability for a program to provide treatments for students needing to address deficiencies in student learning outcomes along the way. Therefore, we identified that by reorienting the protocols from three points to four points with three corresponding phases, music faculty can ground decisions on instruction and degree progress using multiple types of assessment data [formative, self, and summative] (Scott, 2012) at various points of matriculation.

We agreed that the second point in the protocol should be revised to “Mid-point/Admission to Teacher Education” as both programs used this point to decide whether students should continue in the music education program. Additionally, we agreed that a phase prior to the mid-point could be designed to allow students to self-reflect and discover their beliefs about teaching, engage in initial teaching experiences, and develop professional skills such as time management, use of technology, collaboration, and written and verbal communication. The phase after the midpoint would then largely focus on pedagogical and professional learning outcomes and demonstrating growth in musical, pedagogical, and professional learning outcomes as the music teaching candidates prepare for their professional semester.

Through this “professional development” phase, students prepared for approval as a candidate for student teaching. Approval for the student teaching internship is a rigorous process and requires students to meet external academic, professional, and pedagogical standards. While these vary by state, there is a protocol by which each student is selected, placed, and evaluated to ensure readiness for the field. Given this set of criteria, we recommended adding “Admission to the Student Teaching Semester” as an assessment point that would align well with the framework while increasing the adaptability. Because the bulk of pedagogical instruction and development occurs between the mid-point evaluation and student teaching, we agreed that an assessment between mid-point and graduation should be added to the protocols coinciding with approval for the student to begin the student teaching internship.

We recognize that a pre-professional phase follows this third point, wherein university music education faculty collaborate with cooperating teachers in the student teaching internship placement to evaluate the applicable graduation/licensure protocols. The formative assessments in this phase occur during the student’s teaching internship and take a variety of forms: (a) observations, (b) reflective conversations, (c) seminars, and (d) portfolios. They also lead to summative assessments as the fourth point of assessment, indicating degree completion and conferral of a teaching license. See Figure 2 for a summary of the recommended revisions of the former points as they expand to both points and phases for the assessment protocol framework. A summary of the revised framework, based on these discussions can be found in Appendix C.

Figure 2.

Revised Framework.

Next Review and Future Research

Based on our discussions, we revised the assessment protocol framework to include four points of summative assessment and three phases of instruction and formative assessment. Additionally, we revised the language and terminology throughout the levels of achievement and descriptions in the rubrics. With an updated 2024 to 2025 framework, we will employ the protocols during the academic year and complete a review report at the end of the year for another round of revision. We recommend that this second-year review will expand to include a small, liberal arts college that offers a music teacher preparation program to determine how the protocols align with practices for small and/or BA in music programs. Additionally, the next review will include a 2-year community college to examine the efficacy of communicating protocols to a student population who will transfer to a 4-year institution. Additional broadening of sites could also lead to more robust feedback as the researchers endeavor to create applicable and efficacious guidelines generalized for a wider variety of music education faculty and programs.

Given recent developments in education in the United States, it is critical that music education program leaders continue to find ways to ground their practices in sound research. Developing a protocol framework that allows programs to quantify and track their success through implementation of both rubrics and narratives could be invaluable for the profession. This pilot has taken that next step in defining a protocol framework that outlines and measures successful completion of a music teaching license in the United States. Continuing to investigate its applicability, adaptability, and efficacy will provide additional support in these efforts to advocate for music teacher education. Our hope is to continue this line of research to ensure the longevity of our profession through sound, research-grounded, data-informed practice at all levels of music education.

Footnotes

Appendix A

Appendix B

Pilot Framework Overview.⁴

	Musical	Pedagogical	Professional
Admission	- Primary instrument audition - Music theory assessment - Aural skills assessment - Keyboard skills assessment	- Self-assessment of readiness for music education study - Teaching Identity Inventory - Embedded Measure following Intro to Music Course	- Motivations to pursue teaching to establish a baseline of expectations - Sharing of beliefs about teaching - Inventory of past teaching experiences (job shadow, drum major, section leader, student conductor)	Pedagogical and Professional admissions measured with embedded assessments occurring during the first year of study in Intro to Music Education.
Midpoint	- Primary instrument move to upper division - Music theory summative assessment - Aural skills proficiency assessment - Keyboard skills proficiency assessment	- Inventory of past teaching experiences - Reflection of teaching experiences to this point identifying strengths and areas of growth. - Interview to continue covering Knowledge, Skills, Dispositions, and plan of action	- Performance in class that demonstrate teacher identity and dispositions - Articulate beliefs about teaching through sharing a position paper on Music Teaching and Learning - Demonstrate effective relationships with peers and/or teaching	Pedagogical and Professional Skills assessed through embedded measures in the Gateway course during semester 3–4
Graduation	- Senior recital jury	- Music teaching unit, which is a culmination of musical and pedagogical skills - Portfolio that demonstrates mastery of all concepts necessary for success in the classroom	- Clearly articulated Philosophy of Music Teaching - Student Teaching Internship demonstrating delivery of beliefs and connecting with a community while establishing the value of music	Pedagogical and Professional Skills are assessed through the completion of the University Summative Assessment (e.g. Portfolio, etc.) and the successful completion of the student teaching semester.

Appendix C

Revised Framework Overview.⁵

Points and Phases	Musical	Pedagogical	Professional
Point 1: Admission	- Primary instrument audition - Music theory assessment - Aural skills assessment - Keyboard skills assessment	- Self-assessment of readiness for music education study - Teaching Identity Inventory - Embedded Measure following Intro to Music Course	- Motivations to pursue teaching to establish a baseline of expectations - Sharing of beliefs about teaching - Inventory of past teaching experiences (job shadow, drum major, section leader, student conductor)	Pedagogical and Professional admissions measured with embedded assessments occurring during the first year of study in Intro to Music Education.
Phase 1: Pre-education	- Lower-division applied music - Music Theory I–IV - Aural skills - Keyboard proficiency	- Self-reflection and discovery of their beliefs about teaching - Initial teaching experiences - Introduction/Orientation to Music Education	- Time management - Use of technology - Written and verbal communication - Foundations of Education
Point 2: Midpoint/Admission to Teacher Education	- Primary instrument move to upper division- Music theory summative assessment- Aural skills proficiency assessment- Keyboard skills proficiency assessment	- Inventory of past teaching experiences- Reflection of teaching experiences to this point identifying strengths and areas of growth.- Interview to continue covering Knowledge, Skills, Dispositions, and plan of action	- Performance in class that demonstrate teacher identity and dispositions- Articulate beliefs about teaching through sharing a position paper on Music Teaching and Learning- Demonstrate effective relationships with peers and/or teaching	Pedagogical and Professional Skills assessed through embedded measures in the Gateway course during semester 3–4
Phase 2: Professional/Pedagogical Development	- Upper-division applied music- Advanced theory/arranging- Music History Sequence- Address musical deficiencies	- Pedagogical research- Teaching techniques and strategies. Including Intensive Fieldwork in Music Education- Instructional planning and assessment	- Professional dispositions- Address professional deficiencies
Point 3: Approval for Student Teaching Internship	- Successfully address musical deficiencies	- Complete music education coursework	- Successfully address professional deficiencies
Phase 3: Pre-profession	- Demonstrate musical skills in classroom	- Demonstrate pedagogical knowledge and skills in classroom	- Demonstrate professional dispositions
Point 4: Graduation/Licensure	- Senior recital jury	- Music teaching unit, which is a culmination of musical and pedagogical skills- Portfolio that demonstrates mastery of all concepts necessary for success in the classroom	- Clearly articulated Philosophy of Music Teaching- Student Teaching Internship demonstrating delivery of beliefs and connecting with a community while establishing the value of music	Pedagogical and Professional Skills are assessed through the completion of the University Summative Assessment (e.g. Portfolio, etc.) and the successful completion of the student teaching semester.

Acknowledgements

No third-party writing or editing assistance was used in the creation of this manuscript.

ORCID iDs

Jeffrey Ward

Michele Henry

Ethical Considerations

This was a case study where the participants were the authors.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Author Contributions

Jeffrey Ward: Conceptualization: Data curation: Formal analysis: Investigation: Methodology: Project administration: Writing – original draft: Writing – review & editing.

Phillip Payne: Conceptualization: Data curation: Formal analysis: Investigation: Project administration: Writing – original draft: Writing – review & editing.

Michele Henry: Data curation: Formal analysis: Investigation: Writing – original draft: Writing – review & editing.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

All data collected in this case study are reported in the manuscript.

Notes

References

Alexander

Hollingsworth

Montgomery

(2022). Assessing dispositions: Student self-perception vs faculty perception of readiness for a music teacher preparation program. In Brophy

T. SM.

Haning

Lehmann-Wermser

(Chairs), Assessment in Music Education: Theory, Practice, and Policy: Selected Papers from the 8th International Symposium on Assessment in Music Education [Symposium].

Booth

Kuppan

Longtin

Nenadic

(2018). Assessing the International Principles of Assessment in Music Education. In Brophy

Fautley

(Chairs), Selected Papers from the 6th International Symposium on Assessment in Music Education.

Brand

(1987). The best and brightest: Admission requirements for music teachers. Music Educators Journal, 73(6), 32–36. https://doi.org/10.2307/3400259

Colwell

(2006). Music teacher education in this century: Part II. Arts Education Policy Review, 108(2), 17–29.

Fisher

R. A.

Summitt

N. L.

Koziel

E. B.

Hall

A. V.

(2021). Influences on teacher efficacy of preservice music educators. International Journal of Music Education, 39(4), 394–409. https://doi.org/10.1177/0255761420986241

Henry

Alexander

Hollingsworth

Montgomery

(2020). Evaluating Readiness: The Practice of Assessing Music Education Candidates. In Brophy

T.S.

Haning

(Eds.), Advancing Music Education through Assessment: Honoring Culture, Diversity, and Practice: Selected Papers from the 7thInternational Symposium on Assessment in Music Education. (245-259). Chicago, IL: GIA Publications.

Hime

L. J.

Miksza

Hunsucker

(2014). Admissions criteria and upper-divisional ratings as predictors of student teaching achievement in music education. Bulletin of the Council for Research in Music Education, 200, 7–22. https://doi.org/10.5406/bulcouresmusedu.200.0007

Hollingsworth

Alexander

Montgomery

(2024). Assessing dispositions: Student self-perception of teaching readiness before and after a music teacher preparation program. In Hanning

Lehmann-Wermser

(Chairs), Assessment in Music Education: Theory, Practice, and Policy: Selected Papers from the 9th International Symposium on Assessment in Music Education.

Kansas State Department of Education. (n.d). Licensure application. https://www.ksde.org/Agency/Division-of-Learning-Services/Teacher-Licensure-TL/Licensure/License-Requirements/Initial-License-Requirements

10.

Kelly

S. N.

(1988). Marketing Your College Music Program to students. Music Educators Journal, 75(3), 27–29.

11.

Motycka

(1969). Music admission policies and practices: The music student enters the NASM accredited institution of higher education. http://search.proquest.com.er.lib.k-state.edu/docview/64398518?accountid=11789

12.

Parkes

K. A.

Ritcher

G. K.

Doerksen

P. F.

(2019). Measuring dispositions in preservice music educators. In Brophy

(Ed.), The Oxford handbook of assessment policy and practice in music education (Vol. 1, pp. 870–899). Oxford University Press.

13.

Payne

Allen

Wacker

A. T.

Hoffman

E. C.

(2023). On the same page: Curricular clarity through shared language [Poster presentation]. Biennial Symposium on Music Teacher Education, Salt Lake City, UT.

14.

Payne

Burrack

Parkes

Wesolowski

(2019). An emerging process of assessment in music education. Music Educators Journal, 105(3), 36–44. https://doi.org/10.1177/0027432118818880

15.

Payne

Ward

(2020). An examination of admission and assessment of music teacher education candidates. Journal of Music Teacher Education, 29(2), 10–21. https://doi.org/10.1177/1057083719878388

16.

Raiber

M. A.

Teachout

(2022). Journey from Music student to teacher (2nd ed.). Routlege.

17.

Royston

N. S.

Springer

D. G.

(2015). Beliefs of applied studio faculty on desirable traits of prospective music education majors: A pilot study. Journal of Music Teacher Education, 25(1), 78–94. https://doi.org/10.1177/1057083714549467

18.

Sadler

D. R.

(2013). Assuring academic achievement standards: From moderation to calibration. Assessment in Education Principles Policy and Practice, 20(1), 5–19. https://doi.org/10.1080/0969594X.2012.714742

19.

Scott

S. J.

(2012). Rethinking the roles of assessment in music education. Music Educators Journal, 98, 31–35. https://doi.org/10.1177/0027432111434742

20.

Shellahamer

B. R.

(1984). Selection and retention criteria in undergraduate music teacher education programs: Survey, analysis, and implications (admission, audition) [PhD]. Available from ProQuest Dissertations & Theses Global. (303320044).

21.

Teachout

D. J.

(1997). Preservice and experienced teachers’ opinions of skills and behaviors important to successful music teaching. Journal of Research in Music Education, 45, 41–50. https://doi.org/10.2307/3345464

22.

Ward

Payne

(2018). A survey of admission standards and procedures of higher education institutions in the United States in comparison to National Core Music Standards. In Brophy

Fautley

(Eds.), Selected papers from the 6th international symposium on assessment in music education. GIA Publications, Inc.

23.

Ward

Payne

(2020). An examination of music teacher education faculty perceptions regarding indicators of potential success among music teacher education candidates. In Brophy

Haning

(Eds.), Selected papers from the 7th international symposium on assessment in music education. GIA Publications, Inc.

24.

Ward

Payne

(2024). A framework for non-linear assessment protocols for pre-service music teachers from admission to graduation. In Haning

Lehmann-Wermser

(Eds.), Assessment in music education: Unity and diversity. GIA Publications, Inc.

25.

Ward

Payne

Henry

(2024, August). A case study of the implementation of non-linear assessment protocols for pre-service music teachers [Conference presentation]. 36th International Society for Music Education (ISME) World Conference, Helsinki, Finland.

26.

Wayman

V. E.

(2006). Beginning music education students’ and student teachers’ opinions of skills and behaviors important to successful music teaching. Contributions to Music Education, 33, 29–42.

27.

Wiggins

McTighe

(2005). Understanding by design. Pearson.

28.

Wiggins

McTighe

(2007). Schooling by design: Mission, action, and achievement. Association for Supervision and Curriculum Development.