Improving 21st-century teaching skills: The key to effective 21st-century learners

Abstract

The development of competencies known as 21st-century skills are garnering increasing attention as a means of improving teacher instructional quality. However, a key challenge in bringing about desired improvements lies in the lack of context-specific understanding of teaching practices and meaningful ways of supporting teacher professional development. This paper focuses on the need to measure the social quality of teaching processes in a contextualized manner. We do so by highlighting the efforts made to develop and measure teacher practices and classroom processes using the Teacher Instructional Practices and Processes System© (TIPPS) in three different contexts: Uganda (secondary), India (primary), and Ghana (pre-school). By examining how such a tool can be used for teacher feedback, reflective practice, and continuous improvement, the hope is to pave the way toward enhanced 21st-century teacher skills and, in turn, 21st-century learners.

Keywords

Classroom observation classroom quality low- and middle-income countries 21st-century skills reflective practice teacher professional development

Classroom instructional quality (and its relationship to learning outcomes) can serve as a critical lever for educational change. However, there is much still to be learned about what actually goes on inside classrooms, particularly in low- and middle-income countries (LMICs). Though an abundance of observational instruments now exist, most have not undergone rigorous methodological development, and even fewer have been used across different contexts, cultures, and interventions (Bruns, 2011; Crouch, 2008). Many of these observational instruments have taken the form of checklists or time on task measures, which have traditionally been more popular for their cost-effectiveness and ease of use for intervention studies. Nevertheless, a recent comparative study of observational instruments by Bruns et al. (2016) states that time on task measures are too coarse to be used for teacher feedback or performance evaluation. Furthermore, time on task measures are unable to distinguish key aspects of the 21st-century classroom environment such as student engagement, the effective use of instructional strategies, or the emotional factors that support child development (Seidman et al., 2018). Thus, it follows that there is a need to turn away from checklists and time on task measures.

Global interest in how teaching practices and classroom processes affect student learning outcomes and their psychosocial development is growing – and with good reason. Instructional quality has proven to be more strongly associated with child learning than structural aspects of schools in both Western (Pianta et al., 2009) and developing countries (Chavan and Yoshikawa, 2013; Patrinos et al., 2013; Yoshikawa and Kabay, 2015). However, the breath of skills required for quality student learning, and concomitantly quality teaching, call for essential competencies and skills beyond literacy and numeracy, otherwise known as 21st-century skills.

The 21st-century skillset is generally understood to encompass a range of competencies, including critical thinking, problem solving, creativity, meta-cognition, communication, digital and technological literacy, civic responsibility, and global awareness (for a review of frameworks, see Dede, 2010). And nowhere is the development of such competencies more important than in developing country contexts, where substantial lack of improvements in learning outcomes has suggested that the task of improving instructional quality is urgent. A challenge in bringing about the desired improvements lies in the lack of context-specific understanding of teaching practices as well as meaningful ways of supporting teachers in their professional development (Seidman et al., 2018; UNESCO, 2016; Wolf et al., 2018). In other words, how can we improve teacher’s 21st-century skills to help produce 21st-century learners?

Feedback of performance has been demonstrated to be a powerful tool in improving practice in a wide array of arenas from individual behavior to organizational performance (Butler and Winne, 1995). In recent years, there has been ample demonstration of the power of feedback in teaching and other human services (Allen et al., 2011; Becker et al., 2013; Cappella et al., 2012; Glisson et al., 2006; Smith and Akiva, 2008). This paper focuses on: (a) the need to measure the social quality of teaching processes in a contextualized manner; (b) the efforts that we have made to develop and measure, with reliability and concurrent validity, teacher practices and classroom processes in secondary, primary, and pre-school classrooms in Uganda, India, and Ghana, respectively, with the Teacher Instructional Practices and Processes System© (TIPPS; Seidman et al., 2013); (c) how these tools can be fed back to teachers and trainees to facilitate reflective practice and continuous improvement; and (d) how such professional supports can lead to enhanced 21st-century teacher skills and, in turn, 21st-century learners.

The social processes (or how) of quality teaching

Classroom observations are being increasingly used in LMICs to improve education quality through information about current teacher/classroom practices or measuring change in practices over time (UNESCO, 2016). Yet in order to fully understand how we can best help teachers, we need to take a step back and learn to regard teachers as learners and to ensure that the learning we want to see in our children is taking place with our teachers. Learning as an active process is rooted in the educational philosophy of social constructivism (Vygotsky, 1962), which established the belief that knowledge itself is situated within a social context; an individual’s ability to learn is regarded as a series of social processes that are inextricably shaped and influenced by his or her context. Though the perspective is rooted in and remains a predominantly Western ideology, it has taken hold in many countries around the world, and constructivist beliefs for education remain widely relevant for teachers across the globe (OCED, 2009). Nevertheless, constructivist perspectives should not be assumed as ubiquitous in education. For example, in cultures where verbal exchange is not the primary means through which knowledge is conveyed (see Treviño, 2006), we must be mindful of how such cultural variation and nuances affect ways of learning. Differences in sociocultural practices could dictate how children (or in this case teachers) may better learn through practices such as observation, listening, or sharing responsibilities rather than verbalization or actions (Rogoff, 2003; Treviño, 2006).

Social constructivism puts greater emphasis on context and also highlights the important role of culture and how knowledge derived from social processes also exist within cultures (McMahon, 1997; Schunk, 2000). Culture becomes a great influence into not only what patterns of social processes can emerge within a context but also how they emerge (Rogoff, 2003). This perspective calls us to think more carefully, not only about social processes (e.g. classroom interactions) and the knowledge that is generated through those processes, but also about how highly dependent those processes are on the cultures and context in which they reside. For example, a study of a teacher in-service program in South Africa (Brodie et al., 2002) calls attention to how situational constraints, particularly in low-resource contexts, can heavily hinder the ways in which teachers can develop alternative practices that are more learner-centered; the authors draw a critical distinction between the “form” (i.e. techniques such as questioning or group work) versus the “substance” (i.e. content such as engaging with learners’ ideas and interests) of learner-centered teaching. Based upon this work and her own in Tanzania, Vavrus (2009) set forth the notion of a contingent constructivist pedagogy, which considers the pedagogical spectrum between formalism and constructivism and calls for the adaptation of pedagogy to the material conditions, local traditions, and the cultural politics of a context. Such considerations as the ones outlined here have large implications on how social processes could best be measured.

Additionally, the notion of teachers as learners calls us to define what it is we feel that teachers need to know. An increasingly globalized and complex world has propelled a movement toward a vast array of skills that fall under the label of “21st century.” Most frameworks focus on various types of higher-order skills such as complex thinking, communication, collaboration, and creativity (also known commonly as the 4Cs) (e.g. Dede, 2010; Saavedra and Opfer, 2012; Soulé and Warrick, 2015). These skills are increasingly being recognized as the gold standard for student abilities, as well as requirements to meet the demands for success in work and life (Binkley et al., 2012). Yet the practice of delivering knowledge to students via a transmission process (e.g. lecture, dictation) remains dominant in large portions of the world (OECD, 2009). Therefore, if what students are to learn needs to go beyond rote, then there needs to be a concomitant shift in teacher pedagogy to match. Twenty-first-century teachers need to know not only how to use a practice but also when to use a practice to accomplish their goals with students in varying contexts (Darling-Hammond, 2006). This requires teachers to have a deeper knowledge of how to address a diverse array of learners and more refined diagnostic abilities to inform their decisions (Darling-Hammond, 2006). The ability to communicate in such a complex environment requires constant information flow and adjustment (Levy and Murnane, 2004), and a skilled teacher should be adroit at regulating the flow of classroom discussion as it ebbs and flows (Dede, 2010).

Social settings frameworks have historically emphasized the importance of looking at teachers as facilitators of an individual’s learning experience (Cohen et al., 2003; Pianta and Hamre, 2009; Tseng and Seidman, 2007). Learning and development rests within the daily interactions and experiences that take place in the classroom (Seidman and Tseng, 2011; Tseng and Seidman, 2007; Wolf et al., 2018) in addition to being a product of the culture where the processes reside (Stigler et al., 2000). In any given classroom, the core processes and practices are working concurrently. However, from the perspective of classroom observations, there is a need to be able to make clear distinctions between concurrent behaviors because doing so will better enable us to discern how these behaviors relate to key dimensions of the classroom environment that support student learning. Focus is put toward processes and practices in the classroom, thereby reducing the singular focus on what is being taught to how something is being taught. This is no easy task, but this strategy is both conceptually and programmatically aligned to support rigorous evaluation.

How we can measure quality teaching processes

Quantitative approaches to assessing classrooms have the ability to be more systematic and can broadly be broken down into two major categories: checklists and rating/categorization scales. Checklists typically require the classroom observer to mark the presence or absence of the item in the classroom. They are very low inference methods and can catalogue a range of desired constructs from artifacts in the classroom to the practices of the teacher. Rating or categorization scales are often higher inference methods, capable of focusing on the quality of specific behaviors as well as on their frequency of occurrence in the classroom (Waxman et al., 2004). As assessments of quality, rating or categorization scales have much greater potential to be fed back to teachers to change their practices.

Instruments to assess teacher practices and classroom processes used in LMICs have primarily focused on measuring student use of their time in the classroom, primarily due to the framework put forth by the Global Campaign for Education (2002) that set the learner in the center of all education quality endeavors. In the past, however, instruments such as the Stallings (Stallings, 1978) were based on Carroll’s (1963) model of school-based learning that focused on the importance of a learner’s time engaged in learning as well as his or her learning rate. Though this modality could still provide a general picture of classroom alignment with policy and expectations, understanding specific behaviors run the risk of being underreported when using a snapshot method unless intervals are quite frequent (UNESCO, 2016).

At one time, time on task was the prevailing method behind classroom observation, but more recent findings question this mode of measurement. Even as they support it, Benavot and Gad (2004: 293) note that, “researchers disagree over the magnitude of this [time engaged in learning and learning rate] relationship, the relative importance of various intervening factors, and the nature of the socio-economic contexts in which the relationship is more or less salient.” Therefore, though defined as systematic observation instruments, time and frequency measurement instruments, such as the Stallings Snapshot, are far too narrow in scope to meet the need of measuring the quality use of classroom time.¹ Furthermore, time on task measures cannot capture teacher competencies around “soft skills” such as emotional support nor can they differentiate levels (i.e. quality) of instruction (Bruns et al., 2016).

Additionally, many internationally used instruments are intervention-specific. The main drawback to tailoring measures to evaluate the specific facets of an intervention is that the measure becomes insensitive to experimental contrast, and use and comparison of data in other evaluations and contexts are not possible. Only a few instruments – primarily the various iterations of the Stallings Observation System (SOS) – have been used across different contexts, cultures, and interventions (Bruns, 2011; Crouch, 2008). The issue is further compounded by the fact that most instruments in use internationally do not provide a clear conceptual framework that serves as a foundation for instrument validity. The development of a high-quality instrument for LMICs would require further psychometric development. Furthermore, while professional development and pedagogy need to be based on research and existing best practices, it is also of the highest importance that the research that informs it continues to be localized, adapted, and refined to the day-to-day realities of the teacher’s context (Burns and Lawrie, 2015; Vavrus, 2002).

Behavioral observation

Generally speaking, there is relatively little knowledge from LMICs about what happens in classrooms. A review of classroom research in developing countries supported by the World Bank (Venäläinen, 2008) continues to echo the recommendations of Schaffer et al. (1994) and goes on to outline the need for improved instruments and methodologies to gauge classroom quality (e.g. student engagement, effective instructional strategies) and other elements of a classroom that cannot be gleaned from the purview of a “snapshot.” Similarly, the need for greater quality teacher professional development has elicited a call for “a focus on the how of teaching” (Burns and Lawrie, 2015: 43), including a focus on more structured, facilitated opportunities for teachers to learn, a better understanding of the contexts in which they work, and the use of improved data to determine what really works (Burns and Lawrie, 2015).

Westbrook et al. (2013) suggest that one way to improve the knowledge gap on classrooms is through systematic behavioral observations to record teaching practices. Evidence from Western countries suggests that teachers and school leaders trust classroom observations more than other value-added measures (Harris and Herrington, 2015). This is perhaps due to the fact that what they see in classrooms directly relates to what teachers do in practice and is a more concrete basis for information that teachers need to improve (Harris and Herrington, 2015).

A body of knowledge on effective pedagogical practices and classroom processes has begun to emerge (Seidman, 2012). An existing Western research base (e.g. Danielson, 2011; Kane and Staiger, 2012; Mashburn et al., 2014; Pianta, 2011) and emerging international research base (e.g. Araujo et al., 2016; Hu et al., 2016; Leyva et al., 2015) are beginning to rigorously assess the educational quality in the classroom. In some cases, classroom process quality has even been successfully linked to student learning outcomes (Allen et al., 2013; Leyva et al., 2015; Pianta et al., 2008a; Wolf et al., 2018).

Contextualization

As the value of understanding process quality in classrooms has become recognized more broadly, behavioral observation instruments are now being used more regularly to evaluate classrooms – particularly the Classroom Assessment Scoring System (CLASS; Pianta et al., 2008b). CLASS research now has a global base in both developed countries (Bell et al., 2012; Gettinger et al., 2011; Pianta and Hamre, 2010; Tayler et al., 2013) and increasingly in LMIC contexts (Araujo et al., 2016; Hu et al., 2016; Leyva et al., 2015). An empirical base for the CLASS across all these various contexts has established a three (domain) factor structure of Emotional Support, Classroom Organization, and Instructional Support (Hamre et al., 2013).

However, it is unclear whether or not the cross-country similarities are the result of the tool (Pastori and Pagani, 2017) and a de facto predefinition of quality that has been defined by the tool that is measuring it (Vandenbroeck and Peeters, 2014). Some critical questions have been raised about whether standards-based instruments can be applied out of the context from which they originated without serious consideration for their cultural consistency and ecological validity (Pastori and Pagani, 2017). Considerations need to be made for underlying cultural complexities and the fact that values may manifest or be implemented differently across locales, though some may be similar or common (Pastori and Pagani, 2017; Rogoff, 2003).

Some emerging empirical evidence now suggests some psychometric inconsistencies with the CLASS three-factor structure outside of the US (see Pakarinen et al., 2010 and von Suchodoletz et al., 2014). Furthermore, in low resource contexts such as sub-Saharan Africa (SSA), the CLASS tool appears never to have been used. Wolf et al. (2018) suggest that based on some piloting exercises of the CLASS for an Early Childhood Education study in Ghana, the tool was not feasible due to the cost implications and potential need for significant adaptations. Much like what was suggested by Pastori and Pagani (2017), measuring classroom quality in a context such as SSA may require a contextually developed and anchored tool (e.g. allowing for incorporation of culturally specific examples) that is designed with the intent of adaptability and ease of use.

Teacher Instructional Practices and Processes System (TIPPS)

In any given classroom, the core processes and practices are working concurrently. However, clear distinctions need to be made between concurrent behaviors – isolating specific practices and processes can better enable us to discern how particular behaviors in the classroom environment serve to support student learning. Looking at processes and practices in the classroom shifts the focus from exclusively what is being taught to how something is also being taught. In other words, both “form” and “substance” of teaching (Brodie et al., 2002) are being considered to determine quality of classroom processes. Assessment of classroom process quality also necessitates a standard, in the form of a specific, conceptually based lens with which observers can view the classroom. Through such a lens, observers can be trained to observe more objectively, spot biases, and minimize subjectivity through clearly defined dimensional guides for rating.

The TIPPS is a classroom observation instrument designed with such considerations in mind and specifically to assess process quality in LMIC contexts. Moreover, TIPPS was developed to look at teaching processes in a granular, nuanced, and culturally relevant manner so that information could be fed back to teachers to improve their performance as well as student academic and social-emotional outcomes. Classroom observers are trained to connect what they observe, using concrete behavioral evidence, to indicators under each dimension of the tool. Observers are guided through common observational biases (e.g. sympathy scoring or appearance bias) so that they can better recognize them should such tendencies arise.

The nature and format of the instrument is meant to ease the process and reduce the cost of administration. We strove to keep the dimensions aligned with general classroom characteristics (rather than subject-specific) as well as to be succinct, focusing on only several key domains of classroom practices and processes. The tool includes items around classroom management, personal learning support, and cognitive activation of students (Praetorius et al., 2014) as well as items that address the importance of a teacher’s efforts to stimulate student interests (Patall et al., 2010) and promote inclusion, for example, gender parity (Stromquist, 2007), in the classroom.

In Table 1, a sample item from the training manual – scaffolding – is presented. First the concept/dimension is defined and next indicators are noted. The instrument’s “structured alternative format” was adapted from Susan Harter’s Perceived Competence Scale for children (1982) with the goal of facilitating more objective and reliable ratings (Seidman et al., 2018). Conceptually and visually, an observer is presented with a dimension and is required to make a dichotomous decision of whether, based upon observed behaviors, that particular dimension is absent or present in the classroom (Column A or Column B). Next, the observer is asked to make a second dichotomous rating, determining if the first dichotomous rating is “somewhat accurate” or “very accurate.”

Table 1.

TIPPS structured format and example.

Column A		Column B
Concept: Teacher uses scaffolding to promote student learning and understanding of subject matter.
Importance of Concept : A teacher can provide a step-by-step framework to help students understand and learn the subject matter; this process is known as scaffolding. Scaffolding can work in various ways – both by building on a base concept or by breaking down a concept into smaller parts. In either case, the teacher leads the learner through to the larger concept by providing prompts, hints, and assistance.
Indicators of scaffolding may include:
• teacher describes or models the thought process behind a concept;
• teacher elaborates on a part of the lesson that students are having difficulty with, and connect it to the larger concept;
• teacher gives examples, hints, and assistance appropriate to level of student understanding;
• teacher elaborates on knowledge student already possesses (vertical scaffolding);
• teacher shares routinized, predictable activities with students to provide structure for development (sequential scaffolding).
Teacher does not use scaffolding to provide a step-by-step framework to help students learn and understand subject matter.		Teacher uses scaffolding to provide a step-by-step framework to help students learn and understand subject matter.
Very Accurate	Somewhat Accurate	Somewhat Accurate	Very Accurate
Teacher writes out double-digit multiplication on the board. “Sarah, come to the blackboard and do problem one from the book,” says the teacher. When Sarah is stuck, teacher points to a part of the example and asks her to “do that step correctly.”	Teacher writes out double-digit multiplication on the board. “Sarah, come to the blackboard and do problem one from the book,” says the teacher. When Sarah is stuck, teacher repeats the same explanation of all the steps again.	Teacher writes out double-digit multiplication on the board. “I have broken it into three steps,” teacher says, “and each step must be completed in order to get to the answer, let’s do a few problems on the board.” Sarah, come to the blackboard and do problem one from the book.” When Sarah is stuck, teacher points out which step she is stuck in and explains the step again.	Teacher writes out double-digit multiplication on the board. “I have broken it into three steps,” teacher says, “and each step must be completed in order to get to the answer. Sarah, come to the blackboard and do problem one from the book.” When Sarah is stuck, teacher points out which step she is stuck in and explains the step again. Once Sarah completes her problem, teacher says, “this is a tricky step and many get stuck on it, let’s do another problem so that we can all review how this step is done” and focuses on the step that is difficult.

Finally, we felt an instrument that was meant to capture social processes needed to also allow for the integration of culturally unique processes (Hughes et al., 1993; Vavrus, 2009). The manifestations of “common” or genotypic processes would vary in each unique context (i.e. expressed phenotypically). Essentially, the way in which we structured the tool – particularly the concrete examples for each dimension – would allow for contextual-uniqueness or specificity.

Preliminary development of the instrument has taken place across multiple LMICs (Democratic Republic of the Congo, India, Tanzania, and Uganda) and at different developmental levels (primary, secondary, and early childhood) in a series of iterative endeavors. Dimensions could be added or deleted as appropriate to the context. In the following section, we present empirical data on the reliability and validity of the TIPPS in three different levels of schooling (secondary, primary, and early childhood) and countries (Uganda, India, and Ghana).

TIPPS secondary (Uganda)

Initial reliability and validity of the TIPPS instrument was established in Uganda (Seidman et al., 2018). In this first empirical study, data were collected from 197 secondary schools (i.e. 737 classrooms), across various regions of the country. Pairs of locally recruited observers were trained to observe live classrooms using the initial version of TIPPS and were asked to match their observations to a manual that outlines 18 behavioral indicators known as “TIPPS dimensions.” These dimensions were constructed to typify a core set of teacher practices and classroom processes, based upon a review of the most commonly used classroom observation instruments and pedagogical literature. The commonalities we found could be a result of educational reforms in many developing countries that have long been driven by notions of modernity (Inkeles, 1975). However, the overlap in core domains are more likely due to the similarity in the conceptual understanding of “teacher quality” (UNESCO, 2006), especially based on the type of professional development that teachers often receive driven largely by Western constructivist beliefs for education across the globe (OECD, 2009).

In spite of the theoretical and conceptual relevance of the dimensions, initial on-the-ground training in Uganda on TIPPS was met with some difficulties. Local enumerators were finding some of the terminology unfamiliar and difficult to internalize. To remedy this issue, the tool required some revisions based on qualitative work. Unfamiliar concepts were discussed to gauge whether such concepts did in fact exist within the Ugandan context; if they did exist, the appropriate wording was decided upon as a group to find the most relevant language in Ugandan English. The aforementioned process was critical to the contextualization of the instrument, and this process has been repeated for all subsequent versions since.

Classroom-level TIPPS dimensions were to be matched with 8th grade biology, English, and mathematics achievement scores. The results were promising in terms of the instrument’s coherence, rater-reliability, and concurrent validity (Seidman et al., 2018). Thirteen dimensions revealed sufficient variance and good to high levels of rater-reliability (range, median). Factor analysis revealed a three-factor structure: “Instructional Strategies,” which includes student-centered learning such as the encouragement of student questions and ideas; “Sensitive & Connected Teaching,” which refers to a teacher’s ability to connect lessons to everyday life and their sensitiveness to respond to students’ needs; and “Deeper Learning,” which characterizes a teacher’s ability to break down concepts to help facilitate student learning.

The factors revealed significant and intriguing subject-specific associations with biology, English, and mathematics scores, leading us to consider the possibility of a differential specificity hypothesis: student performance in an academic discipline could be differentially related to the successful enactment of particular practices within that discipline (Seidman et al., 2018). For example, a teacher’s ability to connect what students are learning to everyday life manifested a trend toward better performance in biology only. Additionally, individual items of the tool also had significant, meaningful associations to learning outcomes. For example, in English, a teacher’s use of specific feedback related directly to student performance in English.

TIPPS primary (India)

We developed and piloted a primary classroom version of the TIPPS as a precursor to developing a feedback loop for Kaivalya Education Foundation (KEF) to improve practice for teachers. All of the original dimensions from the TIPPS secondary version were mutually deemed relevant by New York University and KEF teams at the primary level. However, the concrete examples for each dimension were adapted to be more culturally and developmentally appropriate for primary classrooms. KEF’s design team was tasked with vetting the examples as well as translating the manual into Hindi. A full translation, back-translation protocol was undertaken, and in the end, a side-by-side English-Hindi manual was produced.

Data were collected from 256 classrooms throughout Rajasthan, India. Of these 256 classrooms, 148 were sampled to reflect variation by (a) region of the country, (b) rural/urban/tribal, (c) number of years of KEF programming received by school/principal, and (d) number of students in the classroom. Observers were locally recruited KEF staff (project managers from different KEF teams across India), who were trained to observe video-taped classrooms using the TIPPS. Low inter-rater reliability did not permit for higher-order analyses. Nevertheless, the study did provide important learnings for the development and implementation of the tool (see Jayaram et al., in press).

TIPPS ECE (Ghana)

Unlike the primary version of TIPPS, adaptation for an early childhood version required more than translation and a few adjustments in examples. Instruction in ECE classrooms required different teacher competencies and instructional foci. For example, a key principle of learning in ECE that was absent for primary and secondary was play, which provides young children the opportunity to acquire physical, cognitive, and social skills (Britto and Limlingan, 2012). Thus, the development of the ECE version of the TIPPS focused on theoretical constructs and key quality indicators in the classroom that support early child development, including the use of structured free-play in the classroom, ECE-focused instructional practices, social-emotional support, classroom management and environment (Wolf et al., 2018).

Reasoning and problem solving are among the most commonly included elements for the otherwise variedly defined term “critical thinking” (Lai, 2011), and research on young children’s thinking shows that it is possible for early childhood educators to promote children’s critical thinking skills through their classroom practice (Whittaker, 2014). Nevertheless, the teaching practices toward critical thinking need to manifest in a developmentally appropriate manner in order to facilitate effective learning.

Table 2 provides a comparative example of how a dimension (critical thinking) was adapted and made developmentally appropriate for the ECE classroom. The dimension is largely similar for primary and secondary. However, the wording of the dimension, while still maintaining an emphasis on reasoning and problem solving, is slightly altered. This is because in the case of an early childhood classroom, a teacher may need to do more to scaffold and lead a child through their reflection and thinking processes than an older student who may be able to reflect and evaluate with less prompting and more independence. Therefore, the dimension intentionally specifies “language,” and the importance of concept emphasizes the practice of encouraging students to think aloud. Consequently, the indicators also focus far less on the concept of discussion, which is developmentally more appropriate for older children in primary and secondary, and the concrete examples serve to ground the theoretical list of processes surrounding the item. Again, the concepts of form (critical thinking) and substance (how the child is engaged in critical thinking) play critical roles in the construction of the TIPPS dimension.

Table 2.

TIPPS complex thinking dimension by developmental level.

	Early childhood	Primary	Secondary
Dimension	Teacher encourages children to use language to reason and problem solve.	Teacher uses instructional strategies to aid students in complex thinking or problem solving.	Teacher uses instructional strategies to aid students in complex thinking or problem solving.
Importance of Concept	As children learn, it is important for them to process that information and reflect on it. Through use of expressive language and activities, a teacher will help children by posing questions, comparing and contrasting, or discussing the material that they are learning. By encouraging children to think for themselves and to share their reasoning aloud, a teacher is training children to process information and draw their own conclusions.	By using different forms of reasoning, a teacher helps expand the students’ understanding of the lesson being taught. Discussion, too, is an important instructional approach because it enables the teacher to further student learning beyond lecturing and memorization. Rather than pointing out an answer and limiting thinking around a topic, the teacher can provide ways for students to figure out an answer, promoting exploration and critical thinking.	By using different forms of reasoning such as contrasts and comparisons, a teacher also helps expand the students’ understanding of the lesson being taught. Discussion, too, is an important instructional approach because it enables the teacher to further student learning beyond lecturing and memorization. Rather than pointing out an answer, the teacher provides ways for students to figure out an answer.
Indicators	• Helping children verbalize their ideas and opinions, helping with word choice and sentence formation • Using of open-ended questioning to elicit children’s thoughts (to problem solve) in addition testing knowledge of facts through closed-ended questions • Reframing information to help children to problem solve • Providing time to reflect on work and activities (i.e. summarizing activities for the day)	• Teacher use of techniques such as comparisons and contrasts, story-telling and/or problem solving activities • Teacher use of questions or prompts to invite students to analyze, reflect, or problem solve • Teacher use of various tactics to facilitate discussion (teacher-to-student or student-to-student) • Teacher enables students to engage in discussion directly with one another	• Teacher use of techniques such as comparisons and contrasts, story-telling and/or problem solving activities • Teacher use of questions to invite students to problem solve • Teacher use of various tactics to facilitate discussion • Teacher enables students to engage in discussion directly with one another • Rote learning is used for new and or unfamiliar concepts, not for memorizing answers to questions
Examples	At the end of the school day, the teacher asks the children, “Before we go home, who wants to share what they enjoyed today?” As children give their answers, teacher asks, “Why did you enjoy it?” to prompt children to think about their reasoning. Teacher chooses several children so that many may contribute.	Teacher asks students, “In the morning, if your mom asks you to please get an umbrella before you go outside. What does that mean about the weather outside?” The teacher allows time for several students to respond. Student says, “because it is raining outside.” Another student responds, “but it could also be very hot.” “Good point, it could be hot or rainy,” says the teacher, “let’s thinks of the uses of an umbrella” and invites students to participate.	Teacher asks students to formulate their own definitions of an unfamiliar word from a text; he/she then helps them compare the different answers to come up with a definition everyone agrees upon.

In this empirical study, data were collected from 240 primary schools (i.e. 317 kindergarten classrooms) in Ghana, from six districts in the Greater Accra Region. A set of locally recruited observers were trained to observe classrooms using the TIPPS. Factor analysis revealed a distinct, but different three-factor structure: Facilitating Deeper Learning (FDL), which includes instructional support strategies used to encourage learning such as scaffolding and providing high-quality feedback; Supporting Student Expression (SSE), which includes considering student ideas and interests, as well as the development of higher-order thinking skills such as reasoning and problem solving, connecting concepts to students’ lives outside of the classroom, and language modeling; and Emotional Support and Behavior Management (ESBM), which includes concepts related to both student emotional support (e.g. sensitivity and responsiveness, tone of voice) and positive behavior management strategies (e.g. providing a consistent routine) (Wolf et al., 2018).

Two factors, Supporting Student Expression and Emotional Support and Behavior Management, predict classroom end-of-school-year academic outcomes. One factor, Supporting Student Expression, was shown to predict classroom end-of-school-year social-emotional outcomes. The findings reveal that the TIPPS was successful in identifying some critical elements of process quality in Ghanaian pre-primary classrooms. Furthermore, a teacher’s ability to support student expression during instruction is significant for improved classroom outcomes (Wolf et al., 2018).

Summary

TIPPS observers have been reliably trained and calibrated, with a median AC1 statistic (Gwet, 2002) of .86 in Uganda and an intraclass correlation coefficient (ICC) of 71.1% (variance shared across raters) in Ghana. While Ghanaian observers were trained to use the Early Childhood version of TIPPS, Ugandan observers were trained to use the Secondary version of TIPPS that overlaps with the primary version of TIPPS utilized in India. The training process itself was similar in all developmental levels and includes concept familiarity, bias awareness, and practicing to hone observer behavioral observation techniques. Rater reliability in multiple contexts is a testament to the fact that the TIPPS has been able capture some common dimensions of classroom quality across contexts (as inherent to the context) or, at the very least, that the instrument has appropriately expressed dimensions of classroom quality in a culturally appropriate manner (as external but learned constructs to the context). Yet in the case of India, we see that content and linguistic adaptation of the tool was not enough, as described below.

With the exception of three concepts (of the 19), the Indian observer cohort had low inter-rater reliability. We attribute the difference in rater reliability to the fact that Indian cohort of observers had a great deal of expertise in a particular curriculum and pedagogy. An individual’s quality of thinking around a particular topic is inextricably linked to the context in which that thinking must be done (Bailin et al., 1999). Part of that context includes not only background knowledge but also dispositions and situations (Bailin et al., 1999; Halpern, 2013; Han and Brown, 2013). Therefore, knowledge and experience in a given area of study or practice can be a significant determinant of the ability to think critically in that area (Bailin et al., 1999) and even a potential impediment to critical thinking (Nosich, 2012).

The pedagogical expertise of KEF staff often emerged as a barrier to enhancing objectivity in their classroom observations. This adherence to “practices that work” as well as “supporting teachers to do better” was counterproductive to our efforts to sharpen their observational lens to apply TIPPS as an effective heuristic for objective observation; they could not focus on “what was happening” rather they focused on trying to “fix what is happening” in the classroom. Also, knowledge of the curriculum being taught in the classroom hindered observer objectivity as some trainees pointed out that the teacher in the practice video was not teaching the curriculum as instructed. A final counter-productive factor was observers’ familiarity with the teachers being observed. During practice videos, many observers indicated that they were familiar with teachers being observed and made assumptions about the classroom that were not indicated in the classroom footage they were observing. While these well-intentioned critiques indicate the high level of commitment to KEF, their profession, and their determination to make a difference, it did not make for objective observers.

These initial studies provide insights into the previously unknown mechanisms of interventions and everyday practice. However, the studies also underscore what still needs to be done to fully realize the goals of an observational tool to assess pedagogical practices and classroom processes in LMICs that (a) is easy to use; (b) has cross-cultural and developmental reach; and (c) has potential as a feedback tool to improve teacher and student performance. Having methods that have some equivalence across countries could dramatically reduce costs of evaluation and be the catalyst for more regular and rigorous cross-national analyses, and an instrument that allows for the integration of culturally unique processes is especially desirable, particularly for its crucial contributions for quality teacher feedback. Yet to be fully ascertained, this may require further ethnographic approaches and contingent constructivist pedagogy as described by Frances Vavrus (2009), not only for the form and substance of the tool itself but also, as we have learned through experience, for the training process. We will continue to gather more information on the cultural equivalence of behavioral observation methods and persist in refining the various versions of TIPPS based on the data we have thus far. With knowledge in hand from a tool that has been contextualized in several cultural milieus and developmental contexts, we now turn to how it can be used via feedback and reflective practice to improving 21st-century teaching skills.

Developing 21st-century teacher skills

The education delivery system has a substantial impact on the way in which 21st-century skills develop in learners. Pedagogy, curriculum, school rules and climate, assessments, and benchmarking skill acquisition are all key factors in the way 21st-century skills develop and are monitored. Nevertheless, the classroom is the primary environment where the aforementioned factors culminate to bring knowledge acquisition and skills development. Furthermore, the classroom is the space where learners observe the modeling of these skills by their teachers and can practice themselves. Therefore, it is equally important to prepare and train teachers in not only the acquisition of 21st-century skills but also the dissemination of these skills. Measuring the classroom processes and teacher practices that are enabling and supporting the development of 21st-century skills in the classroom can serve as an important first step.

The role of feedback and reflective practice

We posit feedback to the teacher on his/her own teaching performance as a key deliverable of observation instruments. Thus, our focus shifts from earlier efforts that focused on utilizing classroom observation as a means for understanding and highlighting facets of teaching quality to that of identifying key contributors to teacher feedback in a cycle of continuous professional development (Arbour et al., 2015; Yoshikawa et al., 2015). However, for teachers to improve their practices, the manner in which professional feedback occurs needs to change. Sustained, meaningful changes in processes of the classroom requires feedback that is purposed for the continuous improvement and ongoing support of teachers. There is promising evidence to endorse coaching and feedback interventions, with an emphasis on actual performance or practices, as effective means to alter setting-level regularities (Seidman, 2012). Moreover, the tools used to measure complex social settings such as the classroom continue to require further development and contextualization as well as validation. For classroom assessment in particular, an effective tool would need to be sufficiently granular and nuanced (Pianta, 2011).

Critically, classroom observations must not focus on what training the teacher may have received but rather what the teacher does in class (Burns and Lawrie, 2015) and in a culturally appropriate contextualized manner. Therefore, receiving low-stakes, high-support feedback, such as from observation, allows the teacher to be able to reflect on his or her in-class performance and empower him/her to make changes. In addition, an observation instrument must deliver on two core aspirations – clarity and granularity. The need is to identify both the strong and the weak elements of the classroom in a manner that is both comprehensible and actionable for the teacher. Generic feedback is unhelpful toward improving core practices of the teacher because it is not providing information that is actionable. Granular feedback allows the teacher to reflect on past performance as well as support action for the future change; this degree of specificity has proven to be the most valuable information to feedback to teachers (see Allen et al., 2011; Jones et al., 2013; Rivers et al., 2013).

Individualized support for the teacher is also necessary to support a successful educational intervention or a professional development system. An observational instrument can become a key component in that effort by tying the teacher’s own growth and improvement to student learning and development. Using feedback of information on social regularities in a supportive and empowering context stimulates reflective practice and lasting and effective behavior change (Seidman, 2012). Long term, in-school integration of classroom observations, not just one-off or short term, is a key factor for an instrument’s utility for teachers. Teachers can chart their own progress, and not through data with which they cannot relate but through tangible evidence derived from live or video classroom observation, coaching, and feedback on performance by an experienced mentor. Teachers are empowered to be active participants in their professional development as well as in the refinement of their pedagogy to best serve the students (e.g. 4Rs: Brown et al., 2010; Jones et al., 2011; My Teaching Partner: Allen et al., 2011; RULER: Rivers et al., 2013) – two aspects currently underserved in SSA’s teacher education and continuing professional development programs (Hardman et al., 2011).

Mentoring and reflective practice are not new concepts in LMICs,² and reflective practice, in particular, emphasizes the ability to recognize and monitor one’s own thinking, understanding, and knowledge about teaching (Parsons and Stephenson, 2005). Here too, constructivist ideologies undergird and reinforce the importance of the dynamic, active process of learning for teachers. Throughout this process, it is fundamental that information gained from the acts of monitoring and recognition be mobilized for deep rather than rote thinking (Rodgers, 2002). Cursory practices, such as feedback from behavior checklists, should be eschewed in favor of thinking that highlights critical thinking and the rigorous analyses of information. This level of depth is often not easily achieved through intervention, and reflection that is forced may not cultivate true reflective practice because such exercises can lack critical thinking and trigger social desirability bias (Hobbs, 2007; Orland-Barak, 2005).

As pointed out previously, true reflective practice can be a powerful medium to affect behavior change. But oftentimes, teachers need assistance, guidance and/or support structures in order to improve their practice. This type of teacher support frequently comes in the form of teacher mentorship and coaching, which is generally understood to be intensive, ongoing, responsive, and collaborative job-support (Ackland, 1991). Western education research suggests that teachers who are provided with specific feedback and opportunities to practice these changes in the classroom are able to increase the effectiveness of their teaching (Allen et al., 2011; Jones et al., 2013; Rivers et al., 2013). However, there are worthy examples from LMIC contexts as well.

The Kaivalya Education Foundation has pioneered education leadership and behavior change management in India that works to foster meaning, learning, joy, and pride in every stakeholder in the system by focusing on aspects of self-motivation and engagement. Their work with teachers over the last decade has revealed several critical insights – more specifically that teachers want to access tangible support to improve their practice, broaden their perspectives of education, and further develop their ability to learn new topics. Teachers have also expressed struggles with their limited capacity for self-reflective practice and an inability to advance themselves or support the development of others.

To support teachers, KEF deploys Gandhi Fellows, community-embedded education improvement workers, for one-on-one engagement in each school. This training has included implementation and use of the TIPPS, generating tangible metrics and insights about teachers’ instructional practices and classroom processes (Jarayam et al., in press). Data are used by a Fellow to provide teacher feedback, focusing on areas of strengths and improvement. The combination of a Fellow’s support and TIPPS feedback has been used to empower teachers to self-assess their abilities in a low-stakes environment. Simultaneously, through KEF’s Principal Leadership Development Program (PLDP), a Fellow works with the principal of the school to envision the way they can use the TIPPS to devise an improvement plan for their school. The principal’s coaching and mentoring support becomes an integral, structural part of the school organization.

Conclusion

As competencies such as self-awareness, collaboration, and critical thinking continue to be emphasized as key competencies for sustainable development (see Rieckmann, 2017), it is likely that teacher training interventions based on Western constructivist beliefs will continue to pervade the education developmental landscape in LMICs. Nevertheless, as we have contended here, there is still a great need to proceed with greater consideration for the respective contexts in which teachers are evaluated and receive their professional development. In order to create 21st-century learners, we must focus on teachers’ 21st century skills and re-conceptualize how we can evaluate and train teachers. To achieve this, we have invoked constructivist understandings of what goes on in classrooms and, in particular, teachers’ practices. Beyond common dimensions of practices, we sought to discover and construct dimensions that were expressed in contextually and culturally meaningful ways. In this vein, we were able to identify reliable dimensions with concurrent validity that are capable of being fed back to teachers.

Even so, the process of developing TIPPS was not without its challenges as we navigated various developmental and cultural contexts. Some issues, such as that of translation and language adjustment for content validity, are perhaps simpler and more easily solved. Yet, translating the dimensions of the tool for observers and practitioners is a bare minimum first step. Many of the greater challenges have been in getting observers and practitioners to understand how what they assert they understand on paper actually manifests as actions in the classroom. For example, what does it look like for a teacher to incorporate student interests into a lesson? Is it simply to allowing a student to answer questions or give examples? Or is it something deeper than that? An observer, practitioner, or even a teacher who has never experienced this in a classroom themselves will likely have a very difficult time identifying the behavioral manifestations of such a dimension, let alone employing the practice.

As was mentioned earlier, the classroom is the space where learners observe the modeling of skills by their teachers. If teachers do not know how to identify teaching practices, they certainly will not know how to model them. This is a critical issue not only for observational training but also for feedback and professional development of teachers and a fundamental reason why identifying culturally relevant manifestations of teaching practices is necessary. Having observers, practitioners, and/or teachers study classrooms from their own cultures (whether through live observation or videos) has been supremely important for successful training, rater reliability, and overall relevance of the tool.

In addition, we are still attempting to determine the relevance of certain dimensions of the tool in various contexts if at all. For example, observing inequitable treatment of students (or favoritism) in the classroom has been generally difficult (i.e. invariant) in a few contexts thus far. We have yet to determine whether this is due to the particular contexts or because such dynamics are much more nuanced than can be discerned by an independent third party. A dimension such as cooperative learning also remains to be seen with much frequency, yet its importance to the learning process (as emphasized by many interventions) seems to merit its continued presence in the tool for now.

In spite of the challenges we have outlined, with the right tool in hand to support the process, we feel the key to improving teacher practices is with granular and clear feedback. This needs to be done on a frequent and regular basis to foster self-reflection and continuous improvement. Teaching skills such as critical thinking require that teachers be educated in a manner that is reflective of that process – through professional development that engages ongoing reflection and continuous learning (Han and Brown, 2013). Only with successful accomplishment of such 21st-century teaching skills will we be able to enhance the 21st-century learning of students in LMICs.

Footnotes

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The projects described in this manuscript were supported in part by grants from the Economic and Social Research Council (ESRC) [grant number ES/M004740/1]; the UBS Optimus Foundation, World Bank Strategic Impact Evaluation Fund (SIEF), and NYU Abu Dhabi Reseasrch Institute; and a contract from the Kaivalya Education Foundation.

Notes

Author biographies

Sharon Kim, MA, is a doctoral student in the Psychology & Social Intervention Program and project Co-director for TIPPS at New York University. Her research interests include how cultural contexts affect the factors that influence a quality learning environment as well as how they affect the implementation quality of interventions.

Mahjabeen Raza, MA, is the co-director of the Teacher instructional Practices and Processes System (TIPPS) Project at New York University. Her research focuses on improving measurement quality in international development, and creating meaningful professional development programming for teachers and principals.

Edward Seidman, PhD, is a professor of Applied Psychology and Director, Psychology & Social Intervention Doctoral Program, New York University. His current research interests focus on understanding and improving classroom, school and other social settings, especially in developing nations.

References

Ackland

(1991) A Review of the Peer Coaching Literature. Journal of Staff Development 12(1): 22–27.

Allen

Gregory

Mikami

et al . (2013) Observations of effective teacher–student interactions in secondary school classrooms: Predicting student achievement with the classroom assessment scoring system—secondary. School Psychology Review 42(1): 76.

Allen

Pianta

Gregory

et al . (2011) An interaction-based approach to enhancing secondary school instruction and student achievement. Science 333(6045): 1034–1037.

Araujo

Carneiro

Cruz-Aguayo

et al . (2016) Teacher quality and learning outcomes in kindergarten. The Quarterly Journal of Economics 131(3): 1415–1453.

Arbour

Yoshikawa

Atwood

et al . (2015) Quasi-experimental study of a learning collaborative to improve public preschool quality and children’s language outcomes in Chile. BMJ Quality & Safety 24(11): 727–727.

Bailin

Case

Coombs

et al . (1999) Common misconceptions of critical thinking. Journal of Curriculum Studies 31(3): 269–283.

Becker

Bradshaw

Domitrovich

et al . (2013) Coaching teachers to improve implementation of the good behavior game. Administration and Policy in Mental Health and Mental Health Services Research 40(6): 482–493.

Bell

Gitomer

McCaffrey

et al . (2012) An argument approach to observation protocol validity. Educational Assessment 17(2–3): 62–87.

Benavot

Gad

(2004) Actual instructional time in African primary schools: Factors that reduce school quality in developing countries. Prospects 34(3): 291–310.

10.

Binkley

Erstad

Herman

et al . (2012) Defining twenty-first century skills. In: Griffin

McGaw

Care

(eds) Assessment and Teaching of 21st Century Skills. Dordrecht: Springer, pp.17–66.

11.

Britto

Limlingan

(2012) School Readiness and Transitions. A Companion to the Child Friendly Schools Manual. New York: Unicef.

12.

Brodie

Lelliott

Davis

(2002) Forms and substance in learner-centered teaching: Teachers’ take-up from an in-service programme in South Africa. Teaching and Teacher Education 18(5): 541–559.

13.

Brown

Jones

LaRusso

et al . (2010) Improving classroom quality: Teacher influences and experimental impacts of the 4rs program. Journal of Educational Psychology 102(1): 153.

14.

Bruns

(2011) Building better teachers in the Caribbean. World Bank Regional Learning Event: Improving Teaching and Learning Outcomes in the English-speaking Caribbean Countries with ICT. Bridgetown, Barbados, 13–16 April.

15.

Bruns

De Gregorio

Taut

(2016) Measures of effective teaching in developing countries. Working Paper, 16. Oxford: Research on Improving Systems of Education (RISE).

16.

Burns

Lawrie

(eds) (2015) Where it’s needed most: Quality professional development for all teachers. Inter-Agency Network for Education in Emergencies. New York: INEE.

17.

Butler

Winne

(1995) Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research 65(3): 245–281.

18.

Cappella

Hamre

Kim

et al . (2012) Teacher consultation and coaching within mental health practice: Classroom and child effects in urban elementary schools. Journal of Consulting and Clinical Psychology 80(4): 597–610.

19.

Carroll

(1963) A model of school learning. Teachers College Record 64(8) 723–733.

20.

Chavan

Yoshikawa

(2013) The future of our children: Lifelong, multi-generational learning for sustainable development. Thematic Group 4 on Early Childhood Development, Education and Transition to Work. Sustainable Development Solutions Network.

21.

Cohen

Raudenbush

Ball

(2003) Resources, instruction, and research. Educational Evaluation and Policy Analysis 25(2)119–142.

22.

Crouch

(2008) The snapshot of school management effectiveness: Report on pilot applications. Prepared for USAID under the EdData II project, 20.

23.

Danielson

(2011) Enhancing professional practice: A framework for teaching. N.p.: ASCD.

24.

Darling-Hammond

(2006) Constructing 21st-century teacher education. Journal of Teacher Education 57(3): 300–314.

25.

Dede

(2010) Comparing frameworks for 21st century skills. In: Bellanca

Brandt

(eds) 21st Century Skills: Rethinking How Students Learn. Bloomington: Solution Tree Press, pp.51–76.

26.

Frisoli

PSJ

(2013) Teachers’ experiences of professional development in (post) crisis Katanga province, southeastern Democratic Republic of Congo: A case study of teacher learning circles. EdD Thesis, UMASS Amherst, USA.

27.

Gettinger

Schienebeck

Seigel

et al . (2011) Assessment of classroom environments. In: Bray

Kehle

(eds) The Oxford Handbook of School Psychology. New York: Oxford University Press, pp.265–270.

28.

Glisson

Dukes

Green

(2006) The effects of the ARC organizational intervention on caseworker turnover, climate, and culture in children’s service systems. Child Abuse & Neglect 30(8): 855–880.

29.

Global Campaign for Education (2002) A Quality Education for All: Priority Actions for Governments, Donors and Civil Society. Brussels: Global Campaign for Education.

30.

Gwet

(2002) Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Statistical Methods for Inter-rater Reliability Assessment, 1.

31.

Halpern

(2013) Thought and Knowledge: An Introduction to Critical Thinking. New York: Psychology Press.

32.

Hamre

Pianta

Downer

et al . (2013) Teaching through interactions: Testing a developmental framework of teacher effectiveness in over 4,000 classrooms. The Elementary School Journal 113(4): 461–487.

33.

Han

Brown

(2013) Effects of critical thinking intervention for early childhood teacher candidates. The Teacher Educator 48(2): 110–127.

34.

Hardman

Ackers

Abrishamian

et al . (2011) Developing a systemic approach to teacher education in sub-Saharan Africa: Emerging lessons from Kenya, Tanzania and Uganda. Compare: A Journal of Comparative and International Education 41(5): 669–683.

35.

Harris

Herrington

(2015) Value added meets the schools: The effects of using test-based teacher evaluation on the work of teachers and leaders [Special issue]. Educational Researcher 44(2): 71–76.

36.

Harter

(1982) The perceived competence scale for children. Child Development 53: 87–97.

37.

Hobbs

(2007) Faking it or hating it: Can reflective practice be forced? Reflective Practice 8(3): 405–417.

38.

Fan

et al . (2016) Applicability of the classroom assessment scoring system in Chinese preschools based on psychometric evidence. Early Education and Development 27(5): 714–734.

39.

Hughes

Seidman

Williams

(1993) Cultural phenomena and the research enterprise: Toward a culturally anchored methodology. American Journal of Community Psychology 21(6): 687–703.

40.

Inkeles

(1975) Becoming modern: Individual change in six developing countries. Ethos 3(2): 323–342.

41.

IRC (2011) Creating healing classrooms: A multimedia teacher training resource.

42.

Jayaram

Raza

Sharma

et al . (in press) People, not numbers: Using data to humanize and strengthen teacher support systems in India. New York,NY, USA: Department of Applied Psychology, New York University.

43.

Jones

Bouffard

Weissbourd

(2013) Educators’ social and emotional skills vital to learning. Phi Delta Kappan 94(8): 62–65.

44.

Jones

Brown

Lawrence Aber

(2011) Two-year impacts of a universal school-based social-emotional and literacy intervention: An experiment in translational developmental research. Child Development 82(2): 533–554.

45.

Kane

Staiger

(2012) Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Research Paper. MET Project. Bill & Melinda Gates Foundation.

46.

Kayabwe

Asiimwe

Nkaada

(2014) Successful Decentralization: The Roles and Challenges of DEOs in Uganda. Paris: International Institute for Educational Planning.

47.

Lai

(2011) Critical thinking: A literature review. Pearson Research Reports 640–641.

48.

Levy

Murnane

(2004) Education and the changing job market. Educational Leadership 62(2): 80.

49.

Leyva

Weiland

Barata

et al . (2015) Teacher–child interactions in Chile and their associations with prekindergarten outcomes. Child Development 86(3): 781–799.

50.

Mashburn

Downer

Rivers

et al . (2014) Improving the power of an efficacy study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes. Prevention Science 15(2): 146–155.

51.

McMahon

(1997) Social constructivism and the World Wide Web-A paradigm for learning. ASCILITE conference, Perth, Australia.

52.

Nosich

(2012) Learning to Think Things Through: A Guide to Critical Thinking across the Curriculum. Boston: Pearson.

53.

OECD (2009) Creating effective teaching and learning environments: First results from TALIS. Organisation for Economic Co-operation and Development.

54.

Orland-Barak

(2005) Portfolios as evidence of reflective practice: What remains ‘untold’. Educational Research 47(1): 25–44.

55.

Pakarinen

Lerkkanen

M-K

Poikkeus

A-M

et al . (2010) A validation of the classroom assessment scoring system in Finnish kindergartens. Early Education and Development 21(1): 95–124.

56.

Parsons

Stephenson

(2005) Developing reflective practice in student teachers: Collaboration and critical partnerships. Teachers and Teaching: Theory and Practice 11(1): 95–116.

57.

Pastori

Pagani

(2017) Is validation always valid? Cross-cultural complexities of standard-based instruments migrating out of their context. European Early Childhood Education Research Journal 25(5): 682–697.

58.

Patall

Cooper

Wynn

(2010) The effectiveness and relative importance of choice in the classroom. Journal of Educational Psychology 102(4): 896.

59.

Patrinos

Velez

Wang

(2013) Framework for the reform of education systems and planning for quality. Policy Research Working Paper 6701. Washington, DC: World Bank.

60.

Pianta

(2011) Teaching Children Well: New Evidence-Based Approaches to Teacher Professional Development and Training. Washington DC: Center for American Progress.

61.

Pianta

Barnett

Burchinal

et al . (2009) The effects of preschool education: What we know, how public policy is or is not aligned with the evidence base, and what we need to know. Psychological Science in the Public Interest 10(2): 49–88.

62.

Pianta

Belsky

Vandergrift

et al . (2008a) Classroom effects on children’s achievement trajectories in elementary school. American Educational Research Journal 45(2): 365–397.

63.

Pianta

Hamre

(2009) Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher 38(2): 109–119.

64.

Pianta

Hamre

(2010) Classroom environments and developmental processes: Conceptualization and measurement. In: Meece

Eccles

(eds) Handbook of Research on Schools, Schooling and Human Development. Abingdon: Routledge, pp.43–59.

65.

Pianta

La Paro

Hamre

(2008b) Classroom Assessment Scoring System™: Manual K-3: Boston: Paul H Brookes.

66.

Praetorius

A-K

Pauli

Reusser

et al . (2014) One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction 31(June): 2–12.

67.

Rieckmann

(2017) Education for Sustainable Development Goals: Learning Objectives. Paris: UNESCO.

68.

Rivers

Brackett

Reyes

et al . (2013) Improving the social and emotional climate of classrooms: A clustered randomized controlled trial testing the RULER approach. Prevention Science 14(1): 77–87.

69.

Rodgers

(2002) Defining reflection: Another look at John Dewey and reflective thinking. Teachers College Record 104(4): 842–866.

70.

Rogoff

(2003) The Cultural Nature of Human Development. Oxford: Oxford University Press.

71.

Saavedra

Opfer

(2012) Teaching and learning 21st century skills. Phi Delta Kappan 94(2): 8–13.

72.

Schaffer

Nesselrodt

Stringfield

(1994) The contributions of classroom observation to school effectiveness research. Advances in School Effectiveness Research and Practice 133–150.

73.

Schunk

(2000) Learning Theories: An Educational Perspective. Upper Saddle River, NJ: Prentice-Hall.

74.

Seidman

(2012) An emerging action science of social settings. American Journal of Community Psychology 50(1–2): 1–16.

75.

Seidman

Kim

Raza

et al . (2018) Assessment of pedagogical practices and processes in low and middle income countries: Findings from secondary school classrooms in Uganda. Teaching and Teacher Education 71: 283–296.

76.

Seidman

Raza

Kim

et al . (2013) Teacher Instructional Practices & Processes System–TIPPS: Manual and Scoring System. New York: New York University.

77.

Seidman

Tseng

(2011) Changing social settings: A framework for action. Empowering Settings and Voices for Social Change. Oxford: Oxford University Press.

78.

Smith

Akiva

(2008) Quality accountability: Improving fidelity of broad developmentally focused interventions. In: Shinn

Yoshikawa

(eds) Toward Positive Youth Development: Transforming Schools and Community Programs. Oxford Scholarship Online, pp.192–212.

79.

Soulé

Warrick

(2015) Defining 21st century readiness for all students: What we know and how to get there. Psychology of Aesthetics, Creativity, and the Arts 9(2): 178.

80.

Stallings

(1978) The development of the contextual observation system. Annual Meeting of the American Educational Research Association, Ontario, Canada, March 27–31.

81.

Stigler

Gallimore

Hiebert

(2000) Using video surveys to compare classrooms and teaching across cultures: Examples and lessons from the TIMSS video studies. Educational Psychologist 35(2): 87–100.

82.

Stromquist

(2007) The gender socialization process in schools: A cross-national comparison. Background paper prepared for the Education for All Global Monitoring Report 2008, Education for All by 2015: Will We Make It?

83.

Tayler

Ishimine

Cloney

et al . (2013) The quality of early childhood education and care services in Australia. Australasian Journal of Early Childhood 38(2): 13.

84.

Treviño Villarreal

(2006) Evaluación del aprendizaje de los estudiantes indígenas en América Latina. Desafíos de medición e interpretación en contextos de diversidad cultural y desigualdad social. Revista Mexicana de Investigación Educativa 11(28): 225–268.

85.

Tseng

Seidman

(2007) A systems framework for understanding social settings. American Journal of Community Psychology 39(3–4): 217–228.

86.

UNESCO (2006) Teachers and Educational Quality: Monitoring Global Needs for 2015. Paris: UNESCO Inst for Statistics.

87.

UNESCO (2016) Measures of Quality through Classroom Observation for the Sustainable Development Goals: Lessons from Low-and-Middle-Income Countries. Paris: UNESCO.

88.

Vandenbroeck

Peeters

(2014) Democratic experimentation in early childhood education. In: Biesta

De Bie

Wildenmeersch

(eds) Civic Learning, Democratic Citizenship and the Public Sphere. Dordrecht: Springer, pp.151–165.

89.

Vavrus

(2009) The cultural politics of constructivist pedagogies: Teacher education reform in the United Republic of Tanzania. International Journal of Educational Development 29(3): 303–311.

90.

Venäläinen

(2008) What Do We Know About Instructional Time Use in Mali? Assessing the Suitability of the Classroom Observation Snapshot Instrument for Use in Developing Countries. Washington, DC: World Bank.

91.

von Suchodoletz

Fäsche

Gunzenhauser

et al . (2014) A typical morning in preschool: Observations of teacher–child interactions in German preschools. Early Childhood Research Quarterly 29(4): 509–519.

92.

Vygotsky

(1962) Thought and Language. Cambridge, MA: MIT Press.

93.

Waxman

Tharp

Hilberg

(2004) Observational Research in US Classrooms: New Approaches for Understanding Cultural and Linguistic Diversity. Cambridge: Cambridge University Press.

94.

Westbrook

Durrani

Brown

et al . (2013) Pedagogy, Curriculum, Teaching Practices and Teacher Education in Developing Countries: A Rigorous Literature Review. London, UK: DFID/Centre for International Education, University of Sussex.

95.

Whittaker

(2014) Good thinking! Fostering young children’s reasoning and problem solving. Young Children 69(2): 80–89.

96.

Wolf

Raza

Kim

et al . (2018) Measuring and predicting process quality in Ghanaian pre-primary classrooms using the Teacher Instructional Practices and Processes System (TIPPS). Early Childhood Research Quarterly 4518–30.

97.

Yoshikawa

Kabay

(2015) The evidence base on early childhood care and education in global contexts. Education for All Global Monitoring Report 2015.

98.

Yoshikawa

Leyva

Snow

et al . (2015) Experimental impacts of a teacher professional development program in Chile on preschool classroom quality and child outcomes. Developmental Psychology 51(3): 309.