Abstract
The Council for the Accreditation of Educator Preparation (CAEP) Standards requires educator preparation programs (EPPs) to ensure instruments used to assess their candidates are both valid and reliable. Due to size and limited financial resources, this task may be challenging for some EPPs. In an effort to address CAEP’s expectations, 26 EPPs in one state formed a collaboration to develop and implement an instrument for use during student teaching, and then conducted analyses of its data to determine the validity and reliability. This article uses a case study methodology to investigate the EPPs’ motivations for participating in the collaboration, and the benefits, challenges, and learning that resulted from participation. The findings, principally related to aspects of individual program improvement, have implications not only for EPPs pursuing CAEP accreditation but also for any higher education institutions interested in collaborative assessment development.
Within the field of educator preparation, there have been discussions about the possibilities afforded by conducting large-scale research in a variety of institutional contexts (Cochran-Smith & Fries, 2005; Liston, Whitcomb, & Borko, 2007; Wilson, Floden, & Ferrini-Mundy, 2002). However, because such research has not been extensively pursued, there are few models to provide guidance for multi-institutional collaborations. Knowing collaboration among organizations is motivated by both internal and external factors (Duffield, Olson, & Kerzman, 2013; Kezar, 2005), it is perhaps not surprising that aspects of the 2013 Council for the Accreditation of Educator Preparation (CAEP) Standards (namely, a call for the development of valid and reliable instruments for use in educator preparation) have inspired a multi-institutional collaboration. Within this article, we describe the findings of a case study exploring the motivations, benefits, challenges, and learning that occurred as part of a collaboration among 26 educator preparation programs (EPPs) in Ohio and the potential implications for institutions that are interested in engaging in collaborations for accreditation or other purposes.
Motivation for Collaboration
In 2013, CAEP approved a set of standards for EPPs preparing candidates for initial teacher licensure (CAEP, 2015a). The standards brought about revisions to the accreditation process, including redefinition of the qualities and characteristics of the evidence submitted during the accreditation review. One of the six aspects of evidence specifies, “Educator preparation providers are responsible for the validity and reliability of evidence they offer to demonstrate that CAEP standards are met” (CAEP, 2013, p. 4). In the 2013
Because establishing the validity and reliability of instruments can require a significant investment of time and resources (e.g., content experts and psychometricians; Boone, Staver, & Yale, 2014; Goldberg, 2014), it is not realistic for each of the more than 700 CAEP accredited EPPs, especially for institutions smaller in size and scope, to establish true validity and reliability for self-developed instruments used. To facilitate the creation of these instruments, while also supporting CAEP’s mission to “advance excellent educator preparation through evidence-based accreditation that ensures quality and supports continuous improvement to strengthen P-12 student learning” (CAEP, n.d.), a group of faculty and staff members from 26 colleges and universities in Ohio organized as the valid and reliable instruments for educator preparation programs (VARI-EPP) Project. Over the course of 3 years, this group developed, implemented, and conducted analyses on a summative instrument to assess teacher candidates during their student teaching practicum: the Candidate Preservice Assessment of Student Teaching (CPAST) Form. The authors served as the coordination team for the project, overseeing planning and ongoing communications with participating institutions. The work was guided by Freeman’s (1993) definition of collaboration: “When two or more people or organizations join forces over a long period of time to produce something neither can achieve alone” (p. 33). The project was supported by limited funding, supplied by Race to the Top, the state
Knowing the success rate of collaborative ventures is less than 50% (Kezar, 2005), establishing a strong collaboration between 26 institutions of varying sizes (1,200-65,000 students) and contexts (public/private) on a limited budget was intimidating. To provide a framework for other institutions that may wish to engage in a similar endeavor, we reviewed literature on collaborations in higher education, and used the themes we discovered as the conceptual framework for a case study investigating the collaboration that occurred during the VARI-EPP Project. Within this article, we begin with a brief review of literature related to key factors that enable collaborations in higher education, followed by a summary of the CPAST Form developed by the VARI-EPP Project team, and a description of the development and implementation process. Next, we present the findings of the following research questions and the implications of their findings:
Literature Related to Collaborations in Higher Education
According to Kezar (2005), “In order to be considered a collaboration, it is key that the process entail an interactive process (relationship over time) and that the groups develop shared rules, norms and structures which often become their first work together” (p. 834). As previously mentioned, collaboration among organizations is motivated by both internal and external factors (Duffield et al., 2013; Kezar, 2005). For universities and P-12 schools, one potential external factor is “the challenge of being asked to pursue both excellence and equity in a climate of accountability and competition” (Butcher, Bezzina, & Moran, 2011, p. 30). For EPPs within universities, the “climate of accountability” is frequently established by the requirements of CAEP accreditation. Although collaboration can be a valuable tool to address such needs and develop new knowledge, Kezar notes “institutions are, generally, not structured to support collaborative approaches to learning, research, and organizational functioning” (p. 832). In the last 20 years, literature describing university/college collaborations, consortia efforts, and recommended best practices has become more prevalent (e.g., Andrews & Lind, 2007; Gottlieb et al., 1999; Plumb & Reis, 2007; Strieter & Blalock, 2006), including an article focusing specifically on a collaboration among three teacher education programs that “developed a common tool for formative and summative evaluation of student teachers” (Duffield et al., 2013, p. 247).
After reviewing literature focusing on multi-institutional collaborations in higher education, we found a dearth of information about collaborations intended to develop performance assessments to meet accreditation needs. Addressing this lack of information is necessary because, as Eddy (2010) advised, “it is important to understand more about partnerships to discern the reasons for their frequent failures and to highlight the structures and processes that promote success and sustainability” (p. 2). Based on our review of literature that described collaborative endeavors in higher education, we identified the following themes (or variations thereof) mentioned repeatedly as key factors necessary for collaboration: external pressures, mission/shared purpose, integrating structures/resources, mutual benefit, networks/relationships/governance, trust, membership of collaboration, learning, and communication.
Key Factors Enabling Collaboration
Collaborations between institutions are complex because it is not merely individuals collaborating to accomplish a common goal, but rather individuals who are embedded within (and representatives of) an institution with its own norms, rules, and structures. External pressures (e.g., messages from external groups, such as CAEP) are a key component for motivating collaborations because they create a strong rationale for the need to engage in collaborative work. In fact, “without a compelling external argument for why collaboration is necessary, it is unlikely to occur” (Kezar, 2005, p. 847). Organizational incentives for collaboration include a shared purpose, shared resources, and mutual benefits.
Once a collaboration is formed, it is important it be driven by a “mission” (Kezar, 2005; Strieter & Blalock, 2006) or “shared purpose” (Butcher et al., 2011). This mission may be motivated by the external pressures inspiring the collaboration, and creates common ground for the work to be completed.
Another foundational element necessary for collaboration in higher education is “integrating structures” (Kezar, 2005), or “resources” (Butcher et al., 2011; Gottlieb et al., 1999). Examples of integrating structures include formation of a center, creation of a central collaboration unit, and supporting technology systems (Kezar, 2005). Similarly, Butcher et al. (2011) identified resources as “the financial and human assets that partners contribute to a given project to ensure its success” (p. 38).
Finally, it is also important for the collaboration to be “mutually beneficial” (Duffield et al., 2013) for it to be sustainable. For this reason, Strieter and Blalock (2006) specified that collaborations should “provide benefits to members” (p. 2) and “define strategies to eliminate/minimize discrepancies in effort and benefits” (p. 3). These benefits, like rewards systems (e.g., tenure and promotion; Kezar, 2005), can increase the appeal of participation in the collaboration. Although the key factors for collaboration described above are organizational in nature, there are other key factors more closely associated with interpersonal interactions.
The first interpersonal key factor necessary for collaboration is “networks” (i.e., relationships between peers) that are established at the beginning of a collaboration with a core group of people genuinely dedicated to the success of the work. Throughout a collaboration, networks of individuals work together to solve problems and create intellectual resources as a way to overcome challenges that could potentially decrease collaboration (Kezar, 2005). The importance of these “relationships” was frequently highlighted in Duffield et al.’s (2013) description of their collaborative partnership. Similarly, Strieter and Blalock (2006) advised the importance of “[establishing] and [nurturing] trusting working relationships between collaborators” (p. 2). Duffield et al. extolled the importance of a governance structure to manage the relationships (e.g., specifying the partners’ roles within the collaboration, which should be “based on characteristics and skills”; Strieter & Blalock, 2006, p. 3).
A second key factor for collaboration is that networks and relationships are developed through “trust,” which Andrews and Lind (2007) noted as “a key element of success” (p. 2) in collaborations. However, networks and relationships cannot go unmanaged, as Duffield et al. (2013) spoke of the importance of building “trust” among collaborative partners, and the amount of time necessary to do so. Butcher et al. (2011) encouraged collaborative partners to “relate on a basis of trust” (p. 36), and also identified collaborative leadership as an important principle:
[Collaborative leadership] requires educative, relational, and evaluative strategies that involve all stakeholders. All need to be represented in the decision-making that occurs across organizations and all need to be aware of their capacity to influence the decision making. (p. 39)
A third key factor identified by multiple authors (Andrews & Lind, 2007; Duffield et al., 2013; Strieter & Blalock, 2006) was the importance of the membership of the collaboration. Specifically, Duffield et al. (2013) advised that “team members need to be carefully chosen because personal agendas or institutionally competitive interests that are not beneficial to all partners can push partnership work off track and create conflict” (p. 248). Strieter and Blalock (2006) recommended “[inviting] members from as many diverse segments of the community and compatible with your mission” (p. 3), and Andrews and Lind (2007) encouraged the formation of “collaborative . . . teams with a variety of skills and skill levels” (p. 2).
Two other key factors for collaboration were predominant in the literature: learning and communication. Although it has been described in different ways, “learning” has been identified as a key factor in collaborations. According to Kezar (2005), “learning” refers to the idea that collaborators need to be educated about “the skills of collaboration” (p. 827) and “the benefits of collaboration in order to motivate people to conduct collaborative work” (p. 847). In Butcher et al. (2011), however, the authors advise that participants in collaborations “remain open to learning and change” (p. 36) because,
given that initiatives are often implemented to create new knowledge and new growth, it should be expected that those involved will develop and grow. There needs to be a willingness to learn on the part of all participants. (p. 38)
Finally, “communication” is the key factor for collaboration that enables all of the other key factors to succeed. For example, Andrews and Lind (2007) stated, “Effective communication is crucial” (p. 2); Gottlieb et al. (1999) advised that collaborators “strive for open communication, flexibility, and patience” (p. 312); and Duffield et al. (2013) frequently discussed the role and importance of communication in their collaborative partnership.
The aforementioned list of key factors of collaboration is not exhaustive of all topics discussed in the literature. Instead, it reflects the most prevalent themes and those that resonated as most applicable to the collaborative VARI-EPP Project.
Summary of the VARI-EPP Project
As previously described, the VARI-EPP Project was comprised of a group of faculty and staff members from 26 colleges and universities in Ohio. They developed and implemented the CPAST Form, an instrument to assess teacher candidates during their student teaching practicum. The authors served as the coordination team for the project, and the project was supported by limited funding from external and internal sources. Within this section, we summarize the content of the CPAST Form, and describe the development and implementation process undertaken by the VARI-EPP Project.
Structure of the CPAST Form
The CPAST Form is a summative performance assessment instrument developed and tested over the course of 3 years. It is a rubric with 21 performance criteria, designed to assess pedagogical skills in four categories (Planning for Instruction and Assessment, Instructional Delivery, Assessment, and Analysis of Teaching) and dispositions in three categories (Professional Commitment and Behaviors, Professional Relationships, and Critical Thinking and Reflective Practice) that teacher candidates across disciplines are expected to demonstrate. The rubric is aligned to multiple sources including the CAEP standards, the Interstate Teacher Assessment and Support Consortium (InTASC) standards, and the Ohio Standards for the Teaching Profession (Ohio Department of Education, 2005).
Development and implementation of the CPAST Form
The CPAST Form was developed based on multiple widely accepted frameworks that are the crux of a preservice curriculum focused on effective pedagogy. First, the Form development team examined the Danielson Framework (Danielson, 2011) to address the following teaching domains and components: (a) Planning and Preparation, (b) the Classroom Environment, (c) Instruction, and (d) Professional Responsibilities. Next, the Form development team considered High Leverage Teaching Practices, which include practices used by teachers that are most likely to result in student learning (Ball, Sleep, Boerst, & Bass, 2009). Finally, the development team studied existing performance assessments such as edTPA (Stanford Center for Assessment, Learning, and Equity [SCALE], n.d.-a) and the Ohio Teacher Evaluation System for further insight into performance criteria selection.
In Year 1, faculty and staff members from the authors’ institution developed the first version of the rubric, which was implemented with more than 350 teacher candidates in multiple content areas. The following year, the initial rubric developed at the authors’ institution provided a starting point for the collaborative effort undertaken when volunteers from EPPs at eight public and private institutions of higher education (IHEs) from across the state participated in the development of a similarly structured performance assessment based on research and best practices in the field of teacher education (Ball & Forzani, 2010; Council of Chief State School Officers, 2011; Danielson, 2011; Gargani & Strong, 2014; Marzano, Pickering, & Pollock, 2001; SCALE, n.d.-a, among others). This second version of this rubric (the first CPAST Form) was piloted with the teacher candidates from the eight institutions who participated in the CPAST Form development, and validity and reliability analyses were conducted with the data. The findings from these analyses were used to revise the CPAST Form for the third year of the project. Faculty and staff members from 10 IHEs were involved in the CPAST Form revision, which was also vetted by content experts (including a P-12 teacher and a teacher educator external to the project). The Form was then implemented with teacher candidates at 26 IHEs, allowing for greater power in the second round of validity and reliability analyses. All participating institutions were provided with a 90-min online training (including an assessment). The training was mandatory for supervisors and optional for cooperating teachers, and was designed to provide instruction about how to use the CPAST Form to score teacher candidates. At the conclusion of the academic year, all participating institutions received their EPPs and the statewide mean scores by rubric row, which could then be submitted to fulfill accreditation requirements. Table 1 describes the context of the participating institutions and their level of involvement in the collaboration.
EPPs Developing and Implementing CPAST.
Method
According to Brown and Rodgers (2002), a case study is “[an] intensive study of the background, current status, and environmental interactions of a given social unit: an individual, a group, an institution, or a community” (p. 21). This study employed case study design (Yin, 2009) to examine the “environmental interactions of a given social unit” (Brown & Rodgers, 2002, p. 21), that is, the collaborative interactions of the institutions participating in the VARI-EPP Project. The multi-institutional collaborative effort of developing and implementing the CPAST Form served as a revelatory case because “the descriptive information alone [was] revelatory” (Yin, 2010, p. 49).
Case Study Participants
For this case study, participants were obtained through purposive sampling, which is commonly used when pursuing case study research in special populations (Bernard, 2006). In this case, the special population is comprised of the one faculty and/or staff member at each institution who served as the liaison on their campus for the VARI-EPP Project and implementation of the CPAST Form (i.e., the person who principally communicated with the project coordination team [the authors] to ensure implementation). As indicated in Table 1, 16 of the liaisons (11 from private institutions and five from public institutions) participated in this case study.
In Year 1, faculty and/or staff members from eight EPPs participated in the initial development of the CPAST Form (Development Team 1) and implemented the Form at their institutions. In Year 2, those same institutions plus two additional EPPs participated in the process of revising the CPAST Form (Development Team 2). Following the revisions in Year 2, 24 EPPs around the state ultimately implemented the CPAST Form with some or all of their student teachers (Implementation Team).
Data Collection
When designing this study, the goal was to collect data that would enable the development of a descriptive case study report (Merriam, 1998). Data were collected through multiple procedures to enhance the data credibility (Baxter & Jack, 2008). Specifically, this study employed methodological triangulation (Stake, 1995), that is, “using multiple methods to confirm emerging findings” (Merriam, 1998, p. 204). The three methods used were a survey, focus groups, and semistructured interviews. The questions for each were developed based on the themes we noted repeatedly as key factors necessary for collaboration: external pressures, mission/shared purpose, integrating structures/resources, mutual benefit, networks/relationships/governance, trust, membership of collaboration, learning, and communication.
Survey
A five-question qualitative survey was distributed to the case study participants via Qualtrics. The survey questions asked participants to describe the following: their EPPs’ motivation for participating in the VARI-EPP Project, the role of their EPP’s leaders in adopting and implementing the CPAST Form, external and internal pressures contributing to the EPP’s involvement in the project, and any additional information they would like to provide about their EPP’s participation in the VARI-EPP Project. As shown in Table 1, representatives from 13 institutions completed the survey (from 10 private and three public institutions).
Focus groups
Focus groups are “widely used to find out
Semistructured interviews
Using a list of researcher-developed interview prompts, and interview strategies and techniques suggested by Charmaz (2001), Kvale and Brinkmann (2008), Seidman (2006), and Talmy (2011), the researchers engaged in individual, approximately one hour semistructured interviews via telephone with five participants (Table 1), two of whom had been highly involved in the CPAST Form project from its inception (both from public institutions), and three of whom were involved in only the implementation stage (all from private institutions). The questions for these interviews were drafted based on themes that arose from the focus group discussions and survey responses. The interviews were audiorecorded and selectively transcribed.
Data Analysis
Because “the process of data collection and analysis is recursive and dynamic” (Merriam, 1998, p. 155) in qualitative case study research, analysis was ongoing throughout the data collection procedures. For example, once surveys were received, they were analyzed to inform the next steps in the data collection process (i.e., drafting appropriate questions for the ensuing focus groups and semistructured interviews). Data source triangulation was used throughout the data analysis process, in an attempt to corroborate and replicate findings across sources (Miles & Huberman, 1994). First, the data sources were analyzed using categorical construction (Merriam, 1998). The goal of this type of analysis was to identify themes that reflected recurring patterns found in the units of data (Merriam, 1998), which in this study included key words and sentences from the survey responses, focus group transcriptions, and semistructured interviews. The key words and sentences were organized into categories using the constant comparative method (Merriam, 1998). Clustering (i.e., “clumping together things that ‘go together’ by using single or multiple dimensions”; Miles & Huberman, 1994, p. 255) was used to organize the data into these categories and create the case study database (Yin, 2009).
Limitations
There are two principal limitations to this study. The first is data were collected from an emic viewpoint; that is, the researchers were part of the collaboration and therefore collecting data from other partners. An etic viewpoint (i.e., data collected from someone who was not participating in the collaboration) may yield different findings. The second limitation is that only a sample of the 26 institutions that participated in the development and/or implementation of the CPAST Form also participated in this case study. Responses from all participating institutions would have yielded more thorough data. However, the 16 case study participants represented both public (five) and private (11) institutions in urban (seven), suburban (six), and rural (three) contexts.
Results
The results of the analysis of the survey responses and focus group and interview transcripts are organized below by the themes of the study’s research questions:
Motivations for Participation
The data revealed there were three separate, yet interrelated motivations for institutions’ participation in the VARI-EPP Project including the necessity of revisions/updates to their current instruments for evaluating student teaching, the external pressure of accreditation requirements, and the increased resources afforded by implementing the CPAST Form.
Need for revisions/updates to current instrument
Of the 16 institutional representatives who participated in this case study, 11 identified their institution’s need for a new student teaching assessment instrument as a motivation for participating in the development and/or implementation of the CPAST Form. The reasons they were seeking a new instrument ranged from
noticing . . . we weren’t seeing a lot of differentiation in the ranking of some of our students, and it seemed we needed to have an overhaul of what was the purpose of assessing our student teachers . . . and we thought this would be a way to jumpstart the conversation (Private Rural D)
to a need for “student teaching forms that were more aligned with educator standards” (Private Suburban F) and/or the need for a “valid and reliable” instrument (mentioned by seven institutions: four private and three public) for student teaching. It is important to note the need for a revised instrument (especially one that was “valid and reliable”) was potentially due to the CAEP accreditation requirements for such instruments.
Accreditation requirements
The second most frequently mentioned motivation for participating in the CPAST Form development and implementation process was the need to comply with CAEP requirements. Nine institutional representatives explicitly mentioned “CAEP” or “accreditation” as a motivator for involvement in the project. Specific comments included the following: “First of all, CAEP was at the uppermost of our minds” (Private Suburban A); “We are all looking for valid and reliable instruments that CAEP will accept” (Public Suburban H); and “We were trying to get ahead of the CAEP situation, and the idea of having valid and reliable instruments is huge” (Public Urban I).
Increased resources
Finally, six of the participating institutions (five private and one public) identified the resources of the collaborative CPAST Form project as a motivator for participating. Meaning, by participating in the development of the Form and providing data for the statistical analyses, they were able to accomplish a task collaboratively that would have been incredibly challenging, if not impossible, to accomplish individually. The institutional representative from Private Suburban A noted, “Our institution is small, therefore we did not have the resources to develop the validity and reliability of any instrument in the same way we can do this as a collaborative effort.” The representative from Private Urban Institution B corroborated this comment in a separate focus group: “We are a small university so taking this on ourselves would be difficult, so having the support and collaborative efforts to arrive at a well-researched instrument is important.” Finally, the representative from Private Suburban D acknowledged in an interview:
I just think that it would take one of the bigger schools to be able to reach out and make something like this happen. I don’t think any of our smaller universities would be able to pull such an extensive piece of research off, and be able to create something that all of us could use for CAEP.
Benefits of Participation
Given the wide variety of the contexts and needs of the EPPs that participated in the VARI-EPP Project, the institutional representatives who participated in the survey, focus group, and interviews identified a variety of benefits they experienced from participating in the collaboration. The benefits described here are those stated most frequently by the institutions and included having an identical form at multiple institutions, and the positive impacts of the Form and its implementation processes within the individual EPPs.
Identical form across institutions
The representative from Private Suburban B stated, “I believe a statewide form to evaluate student teachers is best for all,” and her belief was supported by comments from other institutions who identified unpredicted benefits they found when implementing the same student teaching form as other institutions in the state. For example, the representative from Public Urban G mentioned, “We felt strongly that it would be a benefit to our candidates, our faculty, and our school partners if we were using the same instrument that other institutions would be using.” The Private Suburban F representative provided specific examples of what those benefits were when stating the following:
Consistency is kind of key, so . . . if we have transfer students who come in from Public Urban H, or another institution, and if there are several institutions using the same form across the board, it kind of makes it a little bit easier for those students coming in.
In a separate interview, the representative from Public Urban H (who is located in the same region of the state as Private Suburban F) made a similar observation:
Cooperating teachers loved the idea that [the Form] could be the same form for multiple institutions. Because we’ve got Private Suburban F, we’ve got Private Suburban C . . . we’ve got so many people up in our area. Public Urban J bumps up against us. Public Urban L often. Public Urban I. I mean, that’s not uncommon, to have all of our [P-12] schools having to deal with people from [multiple institutions]. And so when [P-12 school partners] said, “Oh my gosh. We can see a similar form even though the [CAEP Specialized Professional Association] assessments might be different,” they loved it. Any place where you’ve got institutions that tend to bump up because of location, I think there’s great benefit for that . . . if we really want to be able to share that resource of our P-12 partners, instead of having exclusiveness . . . I think that would break some of the barriers down for us as an institution.
In another focus group, Private Urban B and C noted that they shared some supervisors between them, and also with Public Urban F (i.e., supervisors were employed part-time by multiple institutions). Therefore, the continuity of forms across institutions enabled the supervisors to become CPAST Form experts and benchmark scores, even across institutions. In sum, using an identical form across institutions provided benefits for students/candidates, P-12 partner schools, and supervisors who are employed by multiple institutions.
Form impact on individual programs
Two institutional representatives acknowledged one benefit of participating in the collaborative project was being able to “to see what we’re doing in comparison to other institutions across the state” (Private Suburban F), and “see best practice, and get a better feel of where do we sit in the continuum of what’s going on across the state” (Public Urban H). Because the Form was “designed for the profession, by the profession” (SCALE, n.d.-b), “best practices” from individual institutions on the Development Team were incorporated into the CPAST Form content and implementation process. As a result, some institutional representatives mentioned the CPAST Form content and implementation process positively affected other assessments and processes in their program. For example, at three institutions (privates Urban B, Rural C, and Suburban D), the “Dispositions” section of the CPAST Form was used to inform revisions of dispositional assessments used earlier in their programs. The representative from Private Rural C stated,
We started using [the Dispositions section of the Form] with some of our clinical courses . . . so students . . . start to have a sense of the professionalism that’s required . . . in a way that’s consistent of how they’re going to be assessed down the road.
Private Suburban D noted they were now beginning to “look at the dispositions [as] more of a ladder . . . to see if we can move the dispositional part throughout the whole program,” while Private Urban B stated, “We decided to rewrite our dispositions as a result of looking at the dispositions of the [CPAST Form]; that was a definite benefit.”
Another program impact explicitly noted by four institutions was the requirement for the formative midterm and final summative Three-Way Conferences (i.e., two meetings of the candidate, university supervisor, and cooperating teacher where each used the CPAST Form to evaluate the candidate’s performance before the meeting, and then used the meetings to arrive at a consensus score for each row). In one case (Private Urban A), the requirement was the impetus to implement such a process: “I had never done a Three-Way Conference. It was something I was always wanting to try, so this gave me the catalyst to do that and I see lots of good benefits.” The institutional representation from Private Rural F noted,
We had a lot of favorable comments on the Three-Way Conferences. And while we used to always encourage them, we felt like there’s a little more pressure this year to go ahead and do [them], because that was specifically part of the [CPAST Form] process, and that ended up being very beneficial. We got a lot of positive comments from our public school mentors. Hearing more from the supervisors, the students, even though students were uncomfortable, they felt that it was good to hear from both mentor and supervisor. That was a positive, a benefit for our EPP as well.
The Three-Way Conference was also a brand-new process for the EPP at Private Suburban D, whose representative said, “It’s a benefit for everybody, because then they definitely [talk] about the rubric a little bit more, to understand it. Because there’s some parts, that I’m sure [student teachers] don’t really understand, not being out in the field that long.” Similarly, the Private Rural C representative mentioned the benefits for involving the candidates in the assessment process, “We really did like the fact that the student was included in the process, and it’s really helped to open their eyes to the depth of areas that they need to be cognizant of becoming more proficient in.”
Challenges of Participation
Although the institutional representatives were largely positive about the experience of participating in the collaborative CPAST Form development and implementation process, it is important to note “every rose has its thorn” (Michaels, DeVille, Dali, & Rockett, 1988), and the process of implementing the CPAST Form was not without challenges. As with the benefits, the challenges were varied, depending upon the contexts of the individual institutions. For example, three institutions mentioned they faced technological challenges (either with distributing the training to supervisors or collecting the data at the end of the study) when implementing the CPAST Form. Two institutions noted difficulties implementing the CPAST Form while also continuing to use their previous form. However, the one area where institutional representatives noted the most significant challenges was in relation to communicating about the CPAST Form with the cooperating teachers in the P-12 schools where the teacher candidates were placed.
Communication with cooperating teachers
Two institutional representatives cited cooperating teacher confusion about a CPAST Form row titled “Connections to Research and Theory.” This row is aligned to CAEP Standard 1.2 and assesses the student teacher’s inclusion and reflective articulation of teaching methods based on research and theory. In a focus group, the representative from Private Urban A reported “the cooperating teachers wouldn’t know if [the student teachers] were using educational research or not.” In a separate interview, the representative from Public Urban J mentioned that mentor teachers frequently asked questions about the “research row.” She said some expressed a belief that student teachers “shouldn’t be able to do that.” The Public Urban J representative then provided examples of the evidence the student teachers should provide to meet the expectations of the row. Based on her experience, she advised that institutions should provide “better framing [about the rubric] and communication . . . with the mentor teachers” because “things that I just took for granted because I understood the rubric, they didn’t really seem to understand what it really meant.” Likewise, the representative from Private Suburban D believed “the only part that is difficult is communication with the cooperating/mentor teachers. I hope that this collaboration will enable us to include some information that will help them understand the use of the form.” The representative from Private Suburban B also felt further communication with mentor teachers was necessary because
the challenge we had was one particular cooperating teacher—they spent probably an hour and a half during their [Three-Way Conference] . . . and I don’t think that’s the amount of time that’s necessary. So I guess the challenge for me is to make sure the cooperating teacher has a better understanding of the time commitment for this.
Although these institutions identified a need for increased communication with cooperating teachers, other institutions highlighted the challenge of communicating with cooperating teachers due to the time constraints. For example, the representative from Private Rural C mentioned,
We had some challenges with everybody really understanding . . . we offered [the training] to the cooperating teachers, but none of them took us up on that offer, and so there was some variance in terms of them understanding [the Form] as well
as the supervisors who completed the training. Private Rural F’s representative expressed a similar challenge when stating it was “a little hard to train all the . . . mentor teachers. It’s a little hard to ask them to watch all those videos [in the training] and do all those hoops.”
Learning That Occurred From Participation
The institutional representatives indicated that learning occurred for candidates, supervisors, faculty, and cooperating teachers in the P-12 schools as a result of developing and implementing the CPAST Form across a variety of institutions. The representative from Public Urban I (who was involved in the Form development process) stated a “real eye opener” was when she learned “how many different approaches there were across the state to observation and assessing students in the field . . . and the variability of the things that we think are important at the different institutions.” The examples of learning spanned a variety of topics including a need for increased internal communication, technology use, and supervisor reflection, among others. However, learning articulated most frequently by the institutions was in relation to how supervisors, student teachers, and P-12 mentors were scoring the student teachers’ performance using the form.
Use of “Exceeds Expectations”
When the CPAST Form was designed, the development team decided, consistent with the Ohio Teacher Evaluation System, the highest level of performance (Three—“Exceeds Expectations”) should be aspirational and reserved for the very highest performing student teachers. The Form’s levels of performance were designed accordingly, and the Form training advised Form users that “Exceeds Expectations” should be used sparingly, and only for those students demonstrating exceptional, or “rock star,” performance in the classroom. This transition proved to be challenging for supervisors, student teachers, and cooperating teachers in some programs and was educative for the EPP as a whole. When asked whether any learning occurred in the EPP as a result of using the CPAST Form, the representative from Private Suburban F noted,
We potentially learned that some of our supervisors are giving really high ratings. Not that we don’t think that our students are awesome and wonderful, but . . . they are still pre-service teacher candidates and they are still learning, and they’re still growing, but they shouldn’t necessarily be rock stars. And so we kind of learned where our supervisors, and when it comes to evaluating [student teachers] as a professional, what [the supervisors] are doing.
Meanwhile, at Private Urban C, the supervisors learned that the student teachers had inflated perspectives of their performance:
Some of our student teachers rated themselves a lot higher than the supervisors and their [cooperating] teachers did . . . where we thought, “Oh they’re meeting expectations,” [the student teachers] thought they exceeded it. And so it was the basis of some good conversation about, “You’re not fantastic, you’re still learning” . . . that’s an “ah ha” moment that several of the supervisors, including me, had.
The EPP leadership at Public Urban I had a similar realization with their student teachers:
The other piece that we learned was that a lot of the students didn’t understand, or had a hard time with that midterm. Why they didn’t get that highest rating. Again, that was a learning curve for us to talk about “This is a snapshot of where you are in week eight. You wouldn’t be at the [Exceeds Expectations level] in week eight.” But that was a conversation that was very difficult for many student teachers. Especially our grad level students, and our 4.0 students. Because they’re always used to having the highest score.
Finally, Private Suburban D acknowledged there was tension with the CPAST Form scale because the levels of performance on the previous instrument used for assessing student teachers
[don’t] quite synch with what the CPAST Form has. Like our “three” is sort of where we really want them to be by [the end of] student teaching . . . So I think it was just a little bit hard for our cooperating teachers and our supervisors to realize the difference between a “three” on the forms we’ve been using, isn’t necessarily the same as a three on the CPAST Form. So we had to address some of that.
These comments from the representatives of the VARI-EPP Project institutions demonstrated that participating in the collaboration promoted learning by supervisors, candidates, and program leaders in their individual EPPs.
Discussion
This case study examined a collaborative effort among 26 institutions (the VARI-EPP Project), and specifically explored their motivations for participation, the benefits and challenges of participation, and the learning that occurred within the individual institutions as a result of their participation. Table 2 summarizes the principal findings for each of the three research questions. These findings are representative of some, but not all, of the themes we noted as key factors necessary for collaboration according to literature about the topic (i.e., external pressures, mission/shared purpose, integrating structures/resources, mutual benefit, networks/relationships/governance, trust, membership of collaboration, learning, and communication), and have implications for multi-institutional collaborations and accreditation in the field of educator preparation.
Summary of Principal Research Findings.
Relationship of Findings and Key Factors for Collaboration in Higher Education
The formation of the VARI-EPP Project and the findings of this study are representative of some of the key factors for collaboration in higher education. First, participants noted the “external pressure” (i.e., CAEP requirements) played an important role in motivating this collaboration. This “external pressure” partially shaped the “mission/shared purpose” of the Collaboration, which was to develop valid and reliable assessment instruments for EPPs. Eddy (2010) emphasized the important role of “common goals” in establishing successful partnerships in higher education. For the partners in the VARI-EPP Project, the need for a revised student teaching form and to meet accreditation requirements created a common goal to motivate collaboration among these EPPs.
In addition, the “integrating structures/resources” were also a factor that motivated the institutions’ participation in the project. By implementing the CPAST Form, the participating institutions received access not only to the CPAST Form and were required to implement processes such as the Three-Way Conference, but they also received a series of training modules about the use of the Form at the beginning of the academic year. At the end of the academic year, they received the means of each row for all teacher candidates who were scored using the Form (which they could then use for comparative purposes for accreditation reporting). These data then served as one of the many “mutual benefits” afforded to VARI-EPP Project members.
The participants also mentioned two other “mutual benefits.” First, by using a Form and implementation process developed by a team of institutions based on scholarship from the field, the VARI-EPP Project participants were able to benefit from one another’s “best practices,” which then had positive influences on some programs (e.g., the development of longitudinal dispositions assessments). Second, participants noted using an identical form across institutions was a “mutual benefit,” both for the EPPs who shared supervisors and for the EPPs who shared P-12 school partners; in both cases, the supervisors and cooperating teachers had only one form to learn and use.
Although the data presented in this study did not directly reflect the themes of “networks/relationships/governance, trust, and membership of collaboration,” the themes were a key factor of the VARI-EPP Project. Specifically, it could not have taken shape without the already formed network of quality EPPs within the state that had long-standing relationships established on trust.
The key factor of “learning,” as defined by Butcher et al. (2011), was apparent in the case study data, where participants described the various types of learning that took place on their campus as a result of participating in the Collaboration (e.g., a need for increased internal communication, technology use, supervisor reflection, and how the Form was used to score). In the data presented here, the key factor of “communication” emerged as a challenge; while the case study participants noted that communication among the institutions during the Collaboration was positive, multiple institutions learned that communication beyond the immediate collaboration members (i.e., cooperating teachers) of the VARI-EPP Project was less successful, and efforts to share information more fully beyond the participating EPPs’ representatives should be explored. By educating P-12 partners about the CPAST Form as a best practice tool, the CPAST Form and implementation process provide a way for EPPs to connect with P-12 schools, as expected by CAEP Standard 2.
Implications
Schwarz (2015) reported that “one finds very little resistance among the teacher educators in the United States” (p. 105) to the mandates imposed by organizations such as CAEP. Indeed, although the CAEP requirements have elicited varying levels of consternation in EPPs and may not be entirely feasible and realistic as a whole in all contexts, there are components of the Standards that have the capability to move the profession forward. The VARI-EPP Collaboration would not have been conceived, nor had the sense of priority (Kezar, 2005) from participating institutions, without the external pressure of the CAEP Accreditation requirements. As stated by the representative from Private Suburban A,
We would much prefer to use our own student teaching form. However, with CAEP demands for reliability and validity studies that are concluded through research, it is more than our institution has time to do, so we are embracing the collaboration.
As shown by the data presented here, this collaborative project had numerous benefits beyond merely developing instruments that would produce valid and reliable data for accreditation purposes. The goal of the VARI-EPP Project was to create a transformational partnership of institutions “based upon genuine engagement and a focus on common goals and mutual benefits” (Butcher et al., 2011, p. 29), with the hope the collaboration enables “each of the member institutions to do more together than could be accomplished alone . . . an essential component of partnership” (Duffield et al., 2013, p. 249).
The process of establishing an instrument with valid and reliable results for the purposes of accreditation was just one component of the collaboration. Approaching this process as “one for all and all for one” reduced the workload for all involved and provided enhanced results in a variety of contexts. For the VARI-EPP Project, the collaborative process of developing and/or implementing an instrument designed to yield valid and reliable results for accreditation purposes encouraged discussion and learning across EPPs that improved other aspects of their programs (e.g., deeper reflection during Three-Way Conferences, and more consistent assessment of teacher candidates’ dispositions throughout the program). As noted by the institutional representative from Public Urban H, the VARI-EPP Project also provided participating institutions with momentum and traction to implement other program improvements. She described the process of participating in the VARI-EPP Project from its initiation as “rolling the boulder up the hill,” further elaborating,
By working as a team, it gave me the energy to be able to move it, first of all . . . I was just getting enough energy because it’s not something that I’m going to be moving alone, because I think moving it alone within my institution would have been at a snail’s pace . . . and even it felt like at one point it wasn’t just one boulder, but we started moving multiple pieces up at different levels, and addressing the [Specialized Professional Association] pieces, in different ways, and hearing where people were going with that, and addressing the IT component of the integrity of getting them into Tk20 or Taskstream. I think all of those things allowed for multiple pieces in my world to be moved at the same time, and also gave me energy from outside of my institution, that allowed for that momentum, and got me closer to tipping points, where I’m not putting it up, that we’re actually getting it to the point where we want it to be.
Her comments support that when EPPs participate in a multi-institutional collaboration, there are benefits beyond those related directly to the task at hand.
Conclusion
Within this article, we have described the findings of a case study exploring the motivations, benefits, challenges, and learning that occurred as part of a collaboration among 26 EPPs in Ohio. The findings demonstrate that conducting collaborative instrument development may be accomplished if there are common mission and values, by beginning on a small scale and with minimal funding. These findings have implications not only for institutions addressing accreditation requirements but also for any institutions or programs that may wish to engage in a collaborative effort for program enhancement. Plumb and Reis (2007) noted, “Administrators need to provide faculty with examples of collaborations that are already working, either from within the institution or from similar colleges and universities” (p. 29). Therefore, collaborations such as the VARI-EPP Project described here are much needed to inspire and perpetuate future multi-institutional collaborations. With more than 1,500 individual EPPs that are public, private, large, small, urban, suburban, brick and mortar, and/or online, there is a wealth of diverse wisdom in our profession, and uniting it in collaborative efforts will only help to move the profession forward.
Footnotes
Acknowledgements
The authors wish to acknowledge those individuals who contributed to the development of the CPAST Form throughout the project: Frank Beickelman, Mary Bendixen-Noe, Patty Bode, Mary Jo Fresh, Christine Warner, and M. Susie Whittington (The Ohio State University); Debra Gallagher (Bowling Green State University); Anne Price, Annamarie Crell (Cleveland State University); Joanne Arhar, Steven Turner (Kent State University); Connie Patterson (Ohio University); Wendy Jewell (University of Akron); Connie Bowman (University of Dayton); Victoria Stewart (University of Toledo); Martha Hendricks (Wilmington College); Sally Brannan, Tracy Whitlock (Wittenberg University); Tammy Kahrig (Wright State University) The authors would also like to thank Xiangquan Yao for his assistance in the preparation of this manuscript, the VARI-EPP collaborators who participated in this case study, and the anonymous reviewers for their feedback and suggestions.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: This research was supported in part by the Ohio Educator Preparation Program Performance Grant, the American Association of Colleges for Teacher Education (AACTE) State Chapter Support Grant, and the University Center for the Advancement of Teaching (UCAT) Seed Grant at the Ohio State University.