Considerations when using rating scales to support teacher professional development: A commentary on Weadman,Serry and Snow (2022)

Abstract

While observational rating scales such as the Emergent Literacy and Language Early Childhood Checklist for Teachers (ELLECCT) have potential to support Early Childhood Teachers in improving their practice, the existence of a valid and reliable tool is no guarantee of success. A carefully constructed model of teacher professional development will be required to ensure benefits for instructional practice and child language outcomes. This commentary considers the evidence available to guide intelligent design of such a model.

The field of early childhood has a long and rich tradition of developing and using observational rating scales to capture aspects of effective adult–child interactions. Consensus has largely been reached that such interactions can be assessed using systematic protocols in a manner which predicts child learning, even if effects remain moderate (Ulferts et al., 2019). The ELLECT (Weadman et al., 2021) reflects a valuable addition to measures targeting oral language pedagogy, and the current paper is a first step in measure development and psychometrics–although evidence of predictive validity has yet to be established.

We know that effective adult-child interactions can be identified and described. The key now is to establish how to promote these effective practices in a workforce with wide-ranging qualifications. Weadman et al. (2021) propose ELLECT as a means of enhancing Early Childhood Teachers’ (ECTs) shared book reading practices, and initial feedback from ECTs to the tool is positive. However, a good deal of further development, piloting, and research will be needed to establish how teachers might best be supported in using the ELLECT to identify gaps in practice, and gain the knowledge and skills needed to address those gaps. Developing a robust professional development (PD) approach to complement the ELLECT will require as much attention as the development of the scale itself. Although in-service PD shows promise in improving ECTs’ language-and-literacy practice and associated child outcomes it is not uniformly effective – and for every successful programme in the literature there is another with null or minimal effects (Markussen-Brown et al., 2017).

The first step for the ELLECT will be to establish the PD approach or model, drawing on prior research. There is some evidence that using observation rubrics as part of a peer support model can be effective. Burgess et al. (2021) evaluated an intervention in which English secondary school teachers observed and scored each other’s practice using a detailed rubric. Positive effects were found for student performance on maths and English exams, with effects identified for the children of both observers and observees (although effect sizes were small at .07 of a standard deviation [SD]).

Weadman and colleagues’ proposal of a coaching model offered by external professionals is more strongly supported by the literature. A recent meta-analysis concluded that instructional coaching has a pooled effect size of .49 SD on teacher’s practice and .18 SD on child outcomes (Kraft et al., 2018). Indeed, the authors note that effect sizes achieved through coaching are larger than those seen for almost all other school-based interventions, including general PD, teacher preservice training, and merit-based pay. Many of the studies included in the meta-analysis were language-and-literacy coaching programmes for preschool and primary school teachers in the United States, from which much can be drawn to inform the ELLECT model.

The extensive literature on effective characteristics of PD provides a firm foundation for design. For example, Elek and Page’s (2019) review of empirical research on coaching models for early childhood educators identifies observation, feedback, goal-setting, and reflection as common elements of successful programmes. The review further recommends that coaching programmes should align dose and content with educators’ characteristics, skills, and contexts; allow educators opportunities to apply new skills; and support them to reflect on their practice and set self-directed goals.

A further consideration will be to ensure that PD enhances ECTs’ pedagogical content knowledge in additional to focusing on teaching behaviours. Teachers’ domain-specific pedagogical knowledge is as important as instructional quality when it comes to predicting their effectiveness in supporting child outcomes (Coe et al., 2014), with the former underpinning the latter. Ideally, the ELLECT model would enhance ECTs explicit knowledge of oral language pedagogy rather than relying on implicit knowledge to develop as teachers use the tool. A recent exploratory study of ECTs shows that knowledge of language-supporting strategies is not enough to ensure good quality. Teachers with the highest quality classrooms were also able to connect their knowledge of strategies to their knowledge of child development practice, in order to understand the why as well as how (Mathers, 2021). This classroom ‘analysis’ may support teachers in employing pedagogical strategies (such as those detailed in the ELLECT) intentionally during classroom interactions in support of specific instructional goals.

Evidence from the same study indicates that the development of such classroom reasoning or analysis needs to be explicitly supported. Teachers took part in a large-scale intervention study involving 5 days of training and regular follow-up coaching over a 10-month period, focused on effective language-supporting practice and including use of observational rating scales. While teacher’s knowledge of language-supporting strategies improved compared with teachers in the control group, their analytical knowledge did not. While perhaps not surprising, given that the intervention did not set out to enhance such expertise, it illustrates that such learning cannot be taken for granted and that pedagogical reasoning must be explicitly nurtured. For example, teachers might be supported in reflecting their use of ELLECT strategies via video and relating this to child language and thinking (e.g. what was their pedagogical intention, what was the effect on the child, what could have been done differently). Again, there is a rich literature which might be drawn upon to support such work, including studies of teacher video clubs (e.g. Sherin & van Es, 2009).

Finally, when using structured observational tools to support quality improvement, care must be taken to focus teachers on professional learning and behaviour change rather than short-term improvement in scores. It is noteworthy that the researchers cited by Weadman et al. (2021) as arguing for use of observational assessment in supporting PD have latterly developed programmes which are informed by the Classroom Assessment Scoring Scale (CLASS), but which do not explicitly use ratings with teachers during coaching discussions (e.g. Pianta et al., 2017). While the ELLECT’s focus on frequency of strategy use is valuable for guiding improvement, ‘more’ does not always mean ‘better’ and teachers will also need support in reflecting on the appropriateness of their strategy use. For example, in a coaching programme based on the Classroom Strategies Assessment System (CSAS), teachers are provided with feedback on frequency counts and also on recommended frequency – that is, how often they should have used each strategy during the observation (Reddy et al., 2021).

In sum, while the existence of a valid and reliable observation tool cannot guarantee its successful application for improvement purposes, there is a wealth of research available to inform the development of an effective PD model. Any such model should ensure attention to the effective characteristics of PD, nurture teachers’ pedagogical knowledge (including classroom reasoning) and focus on professional learning and behaviour change rather than improvement in scores.