Sage Journals: Discover world-class research

Abstract

This article addresses aggregation as a fundamental practice in educational psychology and ties it into the idiographic/nomothetic distinction, that is, distinguishing between studying what once was and studying what always is. I address the underlying assumptions of seminal educational research (OECD’s large-scales assessment and Hattie’s synthesizing meta-analyses). I argue that educational psychologists assume a priori general educational principles akin to nomothetic laws without sufficiently scrutinizing the limitations of aggregation. I then contextualize this assumption within the history of psychology, and address how these assumptions shape how educational psychologists view, collect, and examine data. Furthermore, I contextualize this assumption with an example showing a peculiarity of educational research: the existence of multiple perspectives on constructs. Finally, I argue that investing time and resources in the debate on aggregation and the epistemic nature of the insights that educational psychologists generate will ultimately advance the field and help bridge the theory–practice gap.

Keywords

aggregation best practice educational psychology idiographic/nomothetic quantitative psychology

This article deals with the practice of aggregation—“combining and summarizing a set of scores into a smaller set of scores that capture an aspect of the original set” (American Psychological Association [APA], 2022, para. 2)—in educational psychology. I argue in this article that educational psychologists assume a priori general educational principles akin to nomothetic laws. This assumption, in turn, shapes how we view, collect, and examine data. Accordingly, educational psychologists should scrutinize their own assumptions and aggregate data to higher levels based on well-founded rationale. This scrutiny should also be accompanied by heeding the limitations of what type of insights we can achieve by only studying aggregates (Lamiell, 1998).

To develop this line of reasoning, I first discuss aggregation in educational psychology. I then go into the practice of aggregation in relation to striving for nomothetic laws. In doing so, I revisit the idiographic/nomothetic distinction and relate it to today’s educational psychology. Building on this, I address the underlying assumptions of seminal educational research. In doing so, I argue that educational psychologists assume a priori general educational principles akin to nomothetic laws. I then go into the epistemic implications of this approach to research and discuss the limitations inherently imposed by studying aggregates in samples and how aggregates are limited in yielding nomothetic insights. Furthermore, I contextualize these lines of reasoning by an example showing a peculiarity of educational research: the existence of multiple perspectives on constructs. Therein, I address how our assumptions on general educational principles can shape how educational psychologists view, collect, and examine data.

Lastly, I discuss the theory–practice gap in educational psychology. I do so by putting forth the notion that the transfer of educational findings into practice may be hindered by quantitative educational psychologists aggregating to ever higher levels without considering the type of insights this process generates. Before this backdrop, I argue that investing time and resources in this debate may advance the field by helping to bridge the divide between theory and practice.

I conclude the article with two recommendations for educational psychologists. I invite all psychological researchers to examine if these can be applied to their respective area of research.

On aggregation in educational psychology

The practice of aggregation is inherent to today’s quantitative educational psychology. In theory, we can aggregate data as soon as the number of observations exceeds one. From a statistical perspective, we can already aggregate data of two participants, most commonly, by averaging. While intuitively a mean of n = 2 is not meaningful, in theory n = 2 already enables nonzero variances and nonzero degrees of freedom; consequently, commonly used statistical procedures relying on variance become available (e.g., t-tests to structural equation models). Going from such minimum requirements for aggregation to the other extreme, educational studies have reached high aggregation levels of up to approximately 236 million students (Hattie, 2009), or entire countries (e.g., The Programme for International Student Assessment, i.e., PISA; Organisation for Economic Co-operation and Development [OECD], 2000). However, we can even observe aggregation within educational case studies (e.g., single-case experimental designs; Plavnick & Ferreri, 2013) or even research of individual teachers (cf. action research; Altrichter, 1990). Essentially, aggregation is present as soon as educational psychologists can use means.

This omnipresence of the aggregate is not a new phenomenon. Danzinger (1990) describes the historical perspective of how psychology shifted from experimentally studying individuals to making knowledge claims based on aggregation of either societal or artificially created groups. Therein, Danzinger highlights how presenting means became suggestive of general laws describing the individual. This shift was supported by aggregated social statistics repeatedly demonstrating regularities (e.g., crime rates in certain districts). Eventually, “psychologists took to presenting their data as the attributes of collective rather than individual subjects” (Danzinger, 1990, p. 87).

Furthermore, educational psychology is signified by closeness to other social sciences and their statistical methods (Gräsel, 2011). This closeness to statistical methods is reflected in the fact that among the psychological journals it was the Journal of Educational Psychology that most rapidly adopted the shift to means across the 1910s, 1920s, and 1930s (Danzinger, 1990, p. 81). For some subdisciplines of educational psychology, this development with its closeness to other areas of psychological research is even made explicit (e.g., comparative education research; Jornitz & Wilmers, 2021).

This triumph of the aggregate has also been the subject of critique in educational psychology. One side of this critique hails from a theoretical level. For example, Smeyers (2015) calls for a critical appraisal of common quantitative educational practices and the promise of simplicity (p. 1383). Questioning this promise can already be understood as criticism of the aggregate: it questions if the implicit simplicity of aggregates is warranted. After all, relying on aggregates can tempt us to dismiss the heterogeneity and thereby the variance that may be underlying the aggregate.

To illustrate with a concrete example, scholars have recently criticized PISA’s comparisons of countries’ average performances (Miller & Fonseca, 2021). The criticism is based on the fact that “an average is just an average” (Miller & Fonseca, 2021, p. 195). Consider, for example, that selected schools from so-called low performing countries can outperform selected schools from so-called high performing countries despite the average differences on the country level. Accordingly, aggregated country level differences in performance do not govern the discrete empirical entities underlying this aggregate.

We can also view this above-mentioned critique of the aggregate in terms of criticizing reductionism in educational psychology. However, we need to clearly distinguish this critique from the overarching discussion on reductionism in educational research. Such a distinction is necessary, as this line of critique encompasses many other lines of reasoning (for an overview, see Allen, 1991; Wrigley, 2019). This specific critique on the aggregate better fits into the narrative of scholars criticizing meta-analyses (Pant, 2014; Wrigley, 2018) and how they tempt us to simplicity at the price of obscuring existing heterogeneity (see Feinstein, 1995; Glass & Robinson, 2004). This promise of simplicity may also be seen as a response to the complexity of educational psychology. After all, this complexity has led seminal scholars to designate educational research as the hardest science of all (Berliner, 2002).

Despite the criticism outlined above, currently, there is a strong tendency in educational psychology to aggregate data to ever higher levels. This tendency is illustrated by synthesizing meta-analyses (e.g., Hattie, 2009) or large-scale assessments such as PISA (see OECD, 2000). Scholars are repeatedly outlining the superiority of higher levels of aggregation over lower levels (as a prominent example, see Hattie, 2009). Moreover, this tendency is also reflected in calls for more data science methods in educational psychology (e.g., Singer, 2019) or even specific recommendations for statistical analyses geared towards handling more and more data (e.g., machine learning; Lindl et al., 2020).

Unfortunately, this tendency does not seem to be accompanied by quantitative educational psychologists debating saliently what the underlying aggregation implies and what are its inherent limitations to knowledge generation. I argue that, to advance the field and its best practice, it is necessary that educational psychology leads this debate on aggregation in close collaboration with quantitative educational psychologists.

To this end, I put forth the notion that educational psychologists’ tendency to aggregate data to ever higher levels is driven by the desire to find general educational principles, that is, educational psychologists strive for their generated knowledge to yield nomothetic laws in the educational context. In the quest for such general educational principles, researchers are trying to remove effects of the context, the situation, and the zeitgeist. At the same time, educational psychology seems to fall short of acknowledging the limitations inherent to analysing aggregates.

Aggregating to strive for nomothetic laws

On the nomothetic/idiographic distinction

Let us first revisit the distinction of studying what once was and studying what always is. Windelband (1921) termed this distinction nomothetic and idiographic, respectively. Nomothetic deals with distinct classes of phenomena, and idiographic describes principles bound by time, context, and culture (Grice et al., 2006). It is part of a debate on our understanding of science and on what divides and unites different disciplines. Scholars often date Windelband’s distinction to his work in 1921, but, in fact, it goes back to a speech in 1894 (Hurlburt & Knapp, 2006; Kemmis, 1978; Lamiell, 1998; von Wright, 1971) or even 1876 (Salvatore & Valsiner, 2010). Therein, Windelband differentiated between nomothetic and idiographic research in an attempt to offer a common ground for the different fields of science.

Tracing back the steps, the idiographic/nomothetic distinction of Windelband was brought via Münsterberg to Stern to Allport and, finally, to psychology’s attention (Hurlburt & Knapp, 2006). The simplified notion in psychology seems to be that nomothetic laws apply to groups while idiographic laws explain the individual (Beck, 1953; Falk, 1956); a prevailing view (see, e.g., Diemer & Gore, 2009; Lazarus et al., 2020; Renner et al., 2020). Consequently, nomothetic psychological laws are derived by reducing an individual to a selected subset of traits, which we then study in groups of individuals. The credo is to study interindividual differences. By contrast, idiographic psychological laws are derived by studying an individual and all their traits. To do so, we acknowledge that individuality is made up of an entire universe of traits, which can all interact with each other. The credo is to study intra-individual differences.

The idiographic/nomothetic distinction has led to the assumption that idiographic and nomothetic are opposites. On the one hand, this assumption led to research aiming to bridge this divide (e.g., Grice et al., 2006) and scholars calling to reconsider existing paradigms (Molenaar, 2004). This distinction also gave rise to different approaches to research, each with distinct research aims concerned with either variables or persons (Howard & Hoffman, 2018). In the more extreme case, this view has led to scholars stressing that idiographic research is an “antiscience” (Nunnally, 1978).

On the other hand, the above-mentioned distinction is an inherent controversy which scholars have tried to homogenize. For example, Meehl (1954) proposed that laws of (human) behaviour are nomothetic for a given group, idiographic when applied to the individual, and finally, strongly idiographic when considering response properties of organisms. Others stress that idiographic and nomothetic research need to go hand in hand (Beck, 1953). Another approach is to find nomothetic laws in idiographic approaches, that is, through collecting more data on intra-individual differences to derive nomothetic laws (e.g., Bakan, 1955; Renner et al., 2020).

Going back to Windelband, he would be very surprised how the debate in psychology around nomothetic and idiographic research deviates from what he intended. Lamiell (1998) gives an overview of how we can understand the original distinction of Windelband. In doing so, he draws a very different picture than contemporary psychology. Far beyond the simplified notion of “nomothetic = group & idiographic = individual,” he stresses that nomothetic was never limited to groups and interindividual differences, nor idiographic to the individual and intra-individual differences. Put more generally, nomothetic refers to the sciences of laws (i.e., what always is) and idiographic to the sciences of events (i.e., what once was). The former concerns itself with what always is, the latter concerns itself with what once was. The former referred to laws for distinct classes of phenomena, the latter to principles bound by time, context, and culture (Grice et al., 2006).

When we ourselves as humans become the object of inquiry, we should look for general laws that we can apply to the individual, even though this may never reach the goal of explaining the individual. Windelband’s underlying assumption was that even after applying general laws to the individual, something will be left unexplained (Lamiell, 1998). Nevertheless, this view implies that there are nomothetic psychological laws. This implication offered psychologists the possibility to position themselves in a domain where they could claim to study human behaviour and derive general laws. Thereby, psychology could see itself as a special case of the natural sciences (Mos, 1998; Salvatore & Valsiner, 2010).

The nomothetic/idiographic distinction in educational psychology

Within the educational debate, we can also find the distinction between nomothetic and idiographic research. However, the nomothetic/idiographic debate is much less dominant in educational psychology and focuses on the idiographic aspect. This focus is rooted in the context and situation specificity of educational processes. For example, scholars have advocated shifting to case studies due to the shortcomings of quantitative educational psychology (Codd, 1981). Other evaluation theorists have called for a focus on the individual to understand the educational milieu in which learning takes place (Kemmis, 1978). Others stress that educational psychology needs to heed individual ways of learning, as learning appears in specific and unique forms (Eisner, 1993). Even others point to researchers handling idiographic synonymously with case studies and action research while delimiting these lines of research from “traditional–quantitative–empirical” research (Altrichter, 1990, p. 136). In the broader scope, we can also find this focus on the individual in comparative education research in which the individual is a whole education system (e.g., application of the idiographic function of Hörner in school leadership research; Brauckmann-Sajkiewicz et al., 2021).

However, these educational examples only partially reflect how the distinction between nomothetic and idiographic was intended by Windelband. These examples adequately reflect that idiographic is bound by time, context, and culture. In doing so, testimony is given to the specificities of educational processes. On the other hand, all above-mentioned examples suffer from the same distortion of the nomothetic/idiographic distinction that can be found in psychological research. This distortion is that idiographic is viewed as inherently limited to studying the individual. Consequently, educational psychology draws the same line psychological research does: either study groups (nomothetic) or study individuals (idiographic).

I argue that this divide has done and will do a disservice to educational psychology. Reconsidering how the nomothetic/idiographic distinction was intended by Windelband (i.e., studying what always is vs. what once was) may be an asset for educational research. I put this argument forth as such reconsideration may prompt educational psychologists to scrutinize their own assumptions on the type of insights we aim to gain. Doing so may, in turn, keep educational psychologists from hastily aggregating to ever higher levels, while erroneously striving to thereby distil general educational principles akin to nomothetic laws. On the contrary, such reconsideration may let us take a step back from the highest level of aggregation, without having to abandon aggregation in favour of henceforth only studying individuals.

The nomothetic striving of educational psychology

I will argue in this section that educational psychology is striving to find general educational principles akin to nomothetic laws by analysing aggregates. Educational psychologists readily assume such general educational principles without scrutinizing this assumption nor how aggregates are supposed to unveil these principles, nor how aggregates cannot unveil these principles for the discrete empirical entities underlying aggregates. I want to first scrutinize this assumption by referring to the PISA studies of the OECD (2000) and the synthesis of meta-analyses by Hattie (2009).

Please bear in mind that even though the OECD’s (2000) PISA and Hattie’s (2009) synthesis of meta-analyses have been highly impactful in the educational field, they are of course examples. Accordingly, they are bound by their context, situation, and zeitgeist. Tapping into the idiographic/nomothetic distinction, these examples serve as studying what once was and are limited by not being able to express what always is. Therefore, conclusions derived from these examples should not be viewed as general educational principles akin to nomothetic laws. They should be viewed as examples of highly impactful studies in the field, which address two important stakeholders in education: students and teachers.

PISA has established itself as an important player for policy reforms and is widely used for benchmarking students’ performance worldwide (Breakspear, 2012). Thus, scholars view PISA as a story of success for the betterment of educational systems worldwide (e.g., Schleicher, 2018). Having said that, scholars also hotly debate and heavily criticize PISA in its basic assumptions (Zhao, 2020), its usefulness and implications (Sjøberg, 2015), and its methods (Fernandez-Cano, 2016). It seems the only thing that proponents and opponents of PISA can agree upon is that PISA yields great influence. At the same time, all stakeholders involved in debating the pros and cons of PISA have more differences than commonalities, resulting in a debate that is rarely eye to eye or beyond one’s own respective field and view (Gorur, 2017).

Setting the controversies surrounding PISA aside, taking a closer look at PISA can be informative for discerning some explicit and implicit views of the educational psychologists involved. Schleicher (2018) strived to “compare the achievements of . . . school systems with those of other countries” (p. 18). This is aimed very high and, once achieved, undoubtedly offers unprecedented possibilities for the betterment of educational systems. At the same time, this ambitious scheme rests on underlying assumptions strongly grounded in what type of knowledge researchers think can be generated. A basic assumption underlying PISA and the educational psychologists involved is that it is possible to measure the quality of educational systems by common and comparable indicators (Sjøberg, 2015). In turn, this assumption is rooted in an even more fundamental assumption on education: the existence of a set of skills and knowledge universally valuable in societies, regardless of that society’s past and future (Zhao, 2020). These are what PISA claims to assess: not what once was but what always is. Accordingly, the striving of PISA is to tease out general educational principles from data, such as distinguishing features of high-performing school systems (Schleicher, 2018). Without assuming such general educational principles akin to nomothetic laws, the striving of PISA is unattainable from the start. Furthermore, PISA evidently assumes that analysing aggregates can yield such general educational principles.

Next, let us consider the synthesis of meta-analyses by Hattie (2009). Hattie’s synthesis is undoubtedly a milestone in the effort to understand successful learning in schools (Terhart, 2011). As such, much has been spoken and written due to and about Hattie’s synthesis (for a recent overview, see Cramer, 2021). Going hand in hand with being well-received worldwide, Hattie’s synthesis was and is also the subject of heavy criticism on the methodological (Bergeron & Rivard, 2017) and theoretical (Rømer, 2019) level. Not unlike PISA, scholars have criticized Hattie’s synthesis as having created a cult (Eacott, 2017).

Without having to pick a side in this controversy, taking a closer look at Hattie (2009) can again be very informative for discerning some explicit and implicit views of the educational psychologists involved. Already at the beginning of his book, Hattie (2009) attests to differences in classrooms, pointing out that today’s classrooms are “busy, multifaceted, culturally invested, and changing” (p. 4) and that “every moment of learning is different” (p. 2). Despite appreciating the fact that that each classroom is variously different, Hattie (2009) ascribes to the notion that “effective teaching can occur similarly for all students, all ethnicities, and all subjects” (p. 239). On the one hand, this notion may testify to believing in educability. On the other hand, this notion implies that studying effective teaching can go beyond studying what once was into discerning what always is. This notion of an underlying general educational principle is a necessary precondition to take a step towards the goal of Hattie’s (2009) synthesis: “we need a barometer of what works best” (p. ix).

Following the train of thought outlined above, the underlying notion in Hattie (2009) is that all differences across teachers, learners, schools, and so forth can be dealt with by aggregating enough student data. Educational psychologists seem to agree with this notion, as criticism is predominantly directed towards Hattie’s flawed methodology and lacking statistical rigour (e.g., Bergeron & Rivard, 2017). On the other hand, theorists have opposed this underlying notion avidly (Rømer, 2019). But, as with PISA, assuming that we can distil such general educational principles given enough data is also a necessary precondition for Hattie’s synthesis. Otherwise, Hattie’s aspiration would have been unattainable from the start and aggregation would not have been his method of choice.

Taken together, both the PISA studies of the OECD (2000) and the synthesis of meta-analyses by Hattie (2009) need to assume the existence of general educational principles akin to nomothetic laws. This assumption is rarely scrutinized within the field of quantitative educational psychology. Yet, it profoundly shapes the way we view, collect, and examine data. After all, to educational psychologists holding this view, such nomothetic laws are hidden within the data. To attain these nomothetic laws, educational psychologists are ever refining their methods and measurements and are collecting more and more data. Better measurement and more aggregation seems to be the chosen path to find these nomothetic laws in education, that is, to go beyond studying what once was and to gain insights into what always is. Herein lies the nomothetic striving of educational psychology.

On nomothesis in today’s educational psychology

This nomothetic striving of educational psychology can be considered as the discipline’s efforts towards developing its self-understanding as a natural science. Such a development in educational psychology is encouraged by the influence of other psychological disciplines. For subdisciplines of educational psychology, such as comparative education research, scholars have made this development with its closeness to other psychological disciplines explicit (for an overview, see Jornitz & Wilmers, 2021). This shift in self-understanding entails two distinct characteristics: the population-centredness (Salvatore & Valsiner, 2010) and the triumph of the aggregate (Danzinger, 1990). Both are as common in educational psychology as they seem unquestioned in common publishing practice.

First, let us consider population-centredness (Salvatore & Valsiner, 2010). Along this line, scholars have suggested binding nomothetic laws by context, situation, and zeitgeist. Doing so, scholars can claim for any finding that it is nomothetic in nature. To give this appearance, researchers must simply specify the exact population where the derived law applies. Such a population may not only be limited by its characteristics, but also by its context, situation, and zeitgeist. This line of reasoning, in particular, renders the nomothetic striving absurd. Consider, for example, Brown et al. (2022) studying what characteristics and circumstances are beneficial for research-informed educational practice in educators. With population-centredness, we can easily lift this claim up to the status of a nomothetic law. The presumably nomothetic law would be that the finding always holds in exactly the studied sample of N = 147 educators at the point in time when the study was carried out in Northern England, Southern England, Midlands, and London (UK). To alleviate this limitation, researchers could go a step further and study whether this finding holds repeatedly across multiple populations. However, following this path would be obstructed by the need for the finding to hold empirically for every particular instance. Taken together, such a law would not depict a general educational principle of a phenomenon that always is, but would be a description of a phenomenon that once was—or at best a phenomenon that was several times in aggregates.

The scientific community would of course reject a nomothetic law derived in the above-described manner. However, educational psychologists would do so given the limited contribution stemming from the severely limited generalizability (for a similar line of reasoning, see Robinson et al., 2013). We would not explicitly reject that it is nomothetic in nature; which, however, would be the correct approach. This is akin to educational psychologists specifying in publications the limitations in generalizability of their own findings and samples, thereby seemingly alleviating the criticism that their findings may not be nomothetic. In common publishing practice, furthermore, this limitation of generalizability is often specified after the research was conducted. What, in fact, is being done here is that we study what once was and claim nomothetic nature by stating that what once was is a general law for what once was and thereby describes something that always is. However, such a line of reasoning does of course not produce general educational principles akin to nomothetic laws.

Bearing population-centredness in mind, let us move on to the triumph of the aggregate (Danzinger, 1990). After all, assuming populations that can be studied by collecting sufficiently sizeable subsets of their individuals is already grounded in the triumph of the aggregate. Danzinger (1990) describes the historical perspective of this triumph. This development was strong within educational psychology: the Journal of Educational Psychology rapidly adopted this practice across the 1910s, 1920s, and 1930s (Danzinger, 1990, p. 81). To reiterate, researchers “took to presenting their data as the attributes of collective rather than individual subjects” (Danzinger, 1990, p. 87), while not evaluating critically enough that regularities on the collective level need not represent regularities on the individual level (Lamiell, 2019). Consequently, researchers lost interest in applying insights to individuals. For educational psychology, this lack hinders applicability of findings to specific problems and individual students. This is unfortunate, as it is exactly this lack of applicability that teachers criticize about educational psychology (Hinzke et al., 2020; Joram et al., 2020), and that scholars have suggested should be increased to facilitate educators’ research-informed educational practice (Brown et al., 2022).

Besides problems and challenges of aggregation discussed by Danzinger (1990), aggregation has a fundamental epistemic implication for the debate on nomothesis. As scholars have pointed out, studying aggregates is severely limited in the striving for discerning nomothetic laws, as insights derived from aggregation may be true for the aggregate but need not apply to every single instance (Lamiell, 1998). Consequently, it would not only be necessary that this transfer of the aggregate to the individual is given, but research would need to investigate this empirically for every particular instance. To be nomothetic in nature, a law derived from the aggregate would need to apply to every single individual from which the aggregate was formed—and more. One way to try to circumvent this issue would be to specify that a law derived from the aggregate is only nomothetic for the aggregate in the situation, the context, and the zeitgeist when it was assessed. Doing so would be as fallacious as described above for population-centredness and would not yield nomothetic laws. Another way to try to circumvent this issue would be to show that a law derived from the aggregate holds repeatedly for each of many aggregates. However, doing so would also be insurmountable, as researchers would need to investigate this empirically for every particular aggregate.

While this inherent nonnomothetic nature yields a strong limitation to aggregation, it also offers a very alluring benefit: laws derived from the aggregate are easily defensible against individuals not adhering to the law. Thereby, these laws are not refutable by individual cases. The exception to the rule does not refute the rule (Lamiell, 1998). The school for which PISA recommendations do not change student achievement cannot refute Schleicher (2018). Akin to this, the student for whom feedback does not work cannot refute Hattie (2009).

Now what are the implications for educational studies where researchers analyse aggregates? On an epistemic level, the implications are that analysing aggregate statistics such as means, standard deviations, correlations, and so forth cannot yield general educational principles akin to nomothetic laws. Consequently, researchers need to acknowledge having gained an insight that was bound by context, situation, and zeitgeist, and is thus inherently idiographic in nature.

Furthermore, we cannot overcome this idiographic nature by aggregating to higher levels. No matter how much researchers strive to nullify the boundedness of an aggregate statistic of a sample by more aggregation, the results will still not be nomothetic in nature.

Moreover, applying meta-analyses can also not alleviate this limitation. Consider first that meta-analyses aggregate aggregates, for example, by estimating an underlying effect size based on multiple studies, each analysing standardized mean differences. What researchers are thereby doing is taking a study that studied what once was, and merging it with other studies that also studied what once was. This process cannot elevate studying what once was to having studied what always is, that is, the sum of multiple idiographic findings does not yield a nomothetic finding.

This idiographic nature of meta-analyses and their underlying primary studies becomes even more clear when considering that researchers per best practice analyse individual studies’ heterogeneity/homogeneity in meta-analyses (Feinstein, 1995). These analyses are geared towards discerning whether individual studies differ in their results and if, consequently, researchers should heed moderators in their meta-analyses (Song et al., 2001). Accordingly, any meta-analyses discerning moderators shows that an effect varied, for example, by context. Moreover, educational psychology is affected by powerful context effects and interactions even more than other fields, as is bound to be reflected by heterogeneity within meta-analyses (cf. Berliner, 2002; Pant, 2014). Consequently, we are not even tasked yet with contemplating if accumulating sufficient idiographic findings with sufficient homogeneity may discern something nomothetic, as the idiographic studies already differ heavily.

This challenge of results varying by context, situation, and zeitgeist also shines a light on another tradition in educational psychology: the lack of replications. This lack has been criticized in educational psychology (Makel & Plucker, 2014; Plucker & Makel, 2021; Rost & Bienefeld, 2019; Travers et al., 2016), and unfortunately ties into psychology’s tradition of preferring novel and statistically significant findings over thoroughly researched findings (Pigott et al., 2013; Sterling, 1959). This tradition goes hand in hand with an excessive number of positive findings (91.5% in psychiatry/psychology; Fanelli, 2010) and a lack of reproducibility (Open Science Collaboration, 2015). This tendency to not replicate findings and to generate findings that do not replicate shines a critical light on the idiographic/nomothetic distinction: how can we dare to make any nomothetic postulates before this background?

On a positive note, scholars have begun to advocate Open Science and to call for more robustness within psychological research (e.g., Wentzel, 2021). This call to Open Science is also echoed within educational psychology (Gehlbach & Robinson, 2021; Krammer & Svecnik, 2020; van der Zee & Reich, 2018). Heeding this call to Open Science and to therefore conduct research more transparently and openly while undertaking more replications may lead educational psychology to at least gauge the robustness of insights into educational issues.

At this point, I want to clarify that this article should not be misunderstood as a call to abandon or to intensify quantitative educational psychology. Neither is it a call to give up on striving to find universal insights into educational issues. However, educational psychology should scrutinize itself and its assumptions on which knowledge it strives to attain. After all, today’s educational psychology is studying what once was while making strong claims about what always is.

A concrete example: Students’ and teachers’ perspectives on teaching quality

After viewing the practice of aggregating in the context of striving for nomothetic laws in educational psychology, I want to place this debate into one specific field of educational psychology—students’ and teachers’ perspectives on teaching quality. Doing so highlights how assumptions on general principles in educational psychology can shape the research process, and thereby the type of light we shed onto educational issues.

A peculiarity of educational psychology is the existence of multiple perspectives on constructs. While there are many constructs in education and educational psychology with different perspectives, I want to limit this example to the perspectives of students and teachers. Scholars can and have assessed constructs from these two perspectives. To name a few: strictness, leadership, giving responsibility, differentiation, promoting students’ independence, uncertainty, rule clarity, task difficulty, and cohesiveness (see overview of den Brok et al., 2006), dimensions of mastery goal structures (Bardach et al., 2018), school climate (Konold & Shukla, 2017), teacher effectiveness (Hill et al., 2011), classroom management (Göllner et al., 2020), and aspects of quality of teaching (Fauth et al., 2020; Krammer et al., 2019, 2021).

The fact that we can view and assess constructs from different perspectives raises profound questions for educational psychology. These questions address the nature of the constructs themselves. Do different perspectives examine the same construct, but merely from different points of view? Or do different perspectives fabricate different constructs? Or are different perspectives a testimony to distinct and different constructs?

Answers to and assumptions on these above-posed questions have implications for what level educational psychologists aggregate their data, that is, it shapes how we handle and interpret data and thus shapes the insights we derive. Not being wary of these assumptions may implicitly dictate the level of aggregation chosen by researchers. In this example, the question is whether one phenomenon is assumed—and consequently researchers aggregate different perspectives; or if two phenomena are assumed—and consequently researchers do not aggregate different perspectives.

Regarding the questions raised above, we can placed answers on a continuum between (a) supposing one phenomenon and thus striving for perspective-free insights and (b) supposing multiple phenomena and letting perspectives stand in their own right. I argue that aiming for perspective-free insights is a hallmark of striving for general educational principles akin to nomothetic laws. On the other hand, letting perspectives stand in their own right respects that educational phenomena may be bound by the context.

Let us further contrast these two poles of one continuum. Supposing one phenomenon, one phenomenon is viewed from different perspectives. Supposing multiple phenomena puts forward that there is one phenomenon for each perspective. Supposing one phenomenon assumes that different perspectives bear a challenge for statistical modelling, and aggregating multiple perspectives can distil the shared basis of these perspectives. Supposing multiple phenomena assumes perspectives standing in their own right, and does not superimpose a perspective-free trait. Supposing one phenomenon can be found in the vein of educational psychology pointing to the benefit of multiple perspectives for better measurement. For example, researchers using mixed methods to assess teacher effectiveness (Hill et al., 2011), or multitrait-multimethod modelling to assess school climate (Konold & Shukla, 2017). Supposing multiple phenomena can be found in the vein of educational psychology that ascribes distinct value to each perspective. For example, researchers emphasizing perspective-specific validities (Fauth et al., 2014; Wettstein et al., 2017), or the contribution of different perspectives to professional development (Fraser, 1998).

Assuming that there is one phenomenon, we may be inclined to affirm this assumption no matter what the data show us. After all, a null correlation between perspectives does not have to challenge the view of one phenomenon; it may just point towards flaws in measurement. Consequently, differences in and lacking precisions in measurement may explain null correlations between perspectives (e.g., not heeding items’ reference frame: Fauth et al., 2020). Thus, the aim of research becomes to find the assumed correlation; thereby, the null correlation becomes merely the absence of this correlation due to measurement issues. This train of thought becomes even more salient in the framework of null hypothesis testing. Within this framework, a H₀ can never be proven (p-values cannot lend support to the H₀; Aczel et al., 2018; Passon & von der Twer, 2020). Consequently, a null correlation can merely imply that no correlation was found.

On the other hand, a similar risk is present when we assume two phenomena. Researchers working under this assumption may not be persuaded otherwise by a perfect correlation between perspectives. After all, a perfect correlation can exist even between unrelated phenomena (cf. spurious correlations; Calude & Longo, 2017).

Consequently, educational psychology should tread very carefully regarding assumptions on different perspectives. These assumptions need to be questioned in research and only perspectives aggregated to a higher level with full intent and purpose. Observing a classroom from different perspectives is not the same as watching an apple fall from a tree from different perspectives. Instead of aggregating to perspective-free traits, perspectives on educational phenomena can stand in their own right.

Aggregation’s impact on theory-to-practice

In the following, I want to outline an additional detrimental effect that aggregation can have for educational psychology. This specific detrimental effect feeds into the theory–practice gap of educational psychology and—to the best of my knowledge—has not been readily addressed in quantitative educational psychology: is the transfer of educational findings into practice hindered by quantitative educational psychologists aggregating to ever higher levels without considering the type of insights this process generates?

First, I argue that more aggregation may not always be of advantage. To this end, let us consider that aggregation in educational psychology inherently opens up the question of which observational unit should be chosen (Hayman et al., 1979). For example, effects may be different for students, classes, and schools. Consequently, educational psychologists need to address at what level an effect should be theorized and studied. Correspondingly, the most appropriate level of aggregation is determined by the research question. Therefore, more aggregation cannot per default be the appropriate option.

Second, I argue that striving for higher levels of aggregation may make findings less applicable to situations at hand. For instance, Hattie’s (2009) finding on the importance of teachers’ feedback is such a broad finding that it is readily applicable to today’s educational systems in general. At the same time, this broadness comes at the cost of generating a finding that may need to be applied very differently to different settings, that is, it neither does justice to the complexity of the subject researched nor to the power of the context (Berliner, 2002). In this example, educational psychologists are confronted with the trade-off of gaining insights that are widely generalizable, but less applicable to a classroom at hand, or are less generalizable but seem more readily applicable (for a similar line of reasoning in research on personnel selection, cf. the bandwidth-fidelity dilemma; Ones & Viswesvaran, 1996).

Consider in a next step two things: how do teachers embrace educational research and how can educational psychologists view this process? Addressing the first question, studies show that when teachers were confronted with the evidence, they very often refuted the evidence (cf. the theory–practice or research–practice gap in education; Brown et al., 2022; Cramer, 2014) or encountered it with open resistance (Terhart, 2013). One source of this reluctance is rooted in the difficulties that arise when applying the evidence to the classroom (for an overview of the difficulties, see Neuweg, 2002, 2015). Furthermore, teachers do not feel that the evidence is transferable to individual students (Hinzke et al., 2020; Joram et al., 2020), which in turn reduces educators’ willingness to implement research-informed educational practices (Brown et al., 2022).

At this point, please bear in mind that this lack of transferability should be considered a problem for educational research. After all, transferability is a fundamental characteristic of educational research (e.g., Elliott, 1986). In other words, educational research should aim at producing scientific insights that can help improve educational systems (Cain et al., 2019; Gräsel, 2011). Before this background, the lack of transferability has long been discussed and has not shone a favourable spotlight on quantitative educational psychology (for overviews, see Brown et al., 2022; Cain et al., 2019; Wrigley, 2018).

This problem is reinforced by educational psychologists attributing teachers’ refusal to transfer theory to practice as a testimony to practitioners’ lack of appreciation and understanding for generalizable educational findings and the underlying research. Yes, on the one hand it can be a problem that teachers do not readily revisit their beliefs when presented with contrary-to-belief evidence (cf. prevailing subjective theories in Neuweg, 2004; or devaluation of educational research in Thomm et al., 2021). However, viewing the theory–practice gap in this light inherently makes practitioners shoulder the burden of the theory–practice gap.

By contrast, it should not only be the practitioners’ responsibility to make sense of educational psychologists’ findings; it should also be educational psychologists’ responsibility to facilitate transferring educational insights into practice. Along this line, scholars have shown that carefully framing and preparing educational insights can indeed lead teachers to trust insights from educational research more than from other sources (Schmidt et al., 2022).

Having said that, educational psychologists seem to forget that educational psychology is studying what once was in aggregates to make strong claims about what always is even on the individual level. Before this backdrop, I urge educational psychology to be more aware of the inherent limitation this places on the derived insights and their transfer to practice (for a summary of a similar critique regarding practising clinicians and clinical psychological research, see Lamiell, 2019). Furthermore, I ask educational psychologists to acknowledge that studying ever higher levels of aggregation does not facilitate transferring theory into practice.

It also stands to reason that educational psychology could profit from listening more to teachers. Arguably, teachers have a deep understanding of educational processes (cf. deriving a workable theory of the case; Elliott, 1986). At the same time, tapping into this understanding may be hindered by not being able to explicitly communicate in the framework of educational research (for the problem of teachers not knowing what teachers can do, see Neuweg, 2002). However, this should not discourage educational psychologists to meet teachers eye to eye.

Taking all of this together, I urge educational psychologists to strive for the most appropriate level of aggregation. Investing time and resources into this debate may advance the field by helping to bridge the divide between theory and practice.

Final note

I want to conclude with two recommendations. I invite all educational psychologists to examine if these can be applied to their respective area of research.

Furthermore, I see no reason not to extend the derived recommendations to other areas of quantitative psychological research. Therefore, I also invite all noneducational psychologists to assess if heeding these recommendations and regarding their basis can contribute to advancing psychological research.

First, educational psychologists should question the nomothetic striving. Educational psychologists should clarify their own assumptions on the existence of general educational principles akin to nomothetic laws. These assumptions profoundly shape the way we view, collect, and examine data.

Second, educational psychologists should scrutinize aggregation. In doing so, we should continuously scrutinize which level of aggregation is the most appropriate. Defaulting to the highest levels of aggregation may advance the field less than respecting the context, the situation, and the zeitgeist.

Footnotes

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Georg Krammer

Author biography

Georg Krammer is University College professor for educational measurement and applied psychometrics at the University College of Teacher Education Styria. He identifies as a research methodologist and social scientist (trained psychologist). He is coeditor of the Zeitschrift für Bildungsforschung [Journal of Research on Education]. Due to his focus on methods, he has a breadth and depth in his fields of research: from applicant faking to classroom leadership, from arithmetic performance to neuromyths, from simulation studies to purely theoretical papers. At the end of the day, he wants to apply stats and methods to further our understanding of substantive issues. Recent publications include: (with H. Burgsteiner; Eds.) Impacts of COVID-19 Pandemic’s Distance Learning on Students and Teachers in Schools and in Higher Education – International Perspective (2022) and (with J. A. Schermer, R. Rogoza, M. Branković et al.), “Humor Styles are Related to Loneliness Across 15 Countries,” in Europe’s Journal of Psychology (2022).

References

Aczel

Palfi

Szollosi

Kovacs

Szaszi

Szecsi

Zruba

Gronau

Q. F.

Van Den Bergh

Wagenmakers

E.-J.

(2018). Quantifying support for the null hypothesis in psychology: An empirical investigation. Advances in Methods and Practices in Psychological Science, 1(3), 357–366. https://doi.org/10.1177/2515245918773742

Allen

R. T.

(1991). Reductionism in education. Philosophical Inquiry in Education, 5(1), 20–35. https://journals.sfu.ca/pie/index.php/pie/article/view/796

Altrichter

(1990). Ist das noch Wissenschaft? Darstellung und wissenschaftstheoretische Diskussion einer von Lehrern betriebenen Aktionsforschung [Is this still science? Presentation and scientific–theoretical discussion of action research conducted by teachers]. Profil Verlag.

American Psychological Association. (2022). Aggregation. In APA dictionary of psychology. https://dictionary.apa.org/aggregation

Bakan

(1955). The general and the aggregate: A methodological distinction. Perceptual and Motor Skills, 5, 211–212. https://doi.org/10.2466/PMS.5.7.211-212

Bardach

Khajavy

G. H.

Hamedi

S. M.

Schober

Lüftenegger

(2018). Student- teacher agreement on classroom goal structures and potential predictors. Teaching and Teacher Education, 74, 249–260. https://doi.org/10.1016/j.tate.2018.05.010

Beck

S. J.

(1953). The science of personality: Nomothetic or idiographic? Psychological Review, 60(6), 353–359. https://doi.org/10.1037/h0055330

Bergeron

P.-J.

Rivard

(2017). How to engage in pseudoscience with real data: A criticism of John Hattie’s arguments in visible learning from the perspective of a statistician. McGill Journal of Education / Revue des sciences de l’education de McGill, 52(1), 237–246. https://doi.org/10.7202/1040816ar

Berliner

D. C.

(2002). Comment: Educational research: The hardest science of all. Educational Researcher, 31(8), 18–20. https://doi.org/10.3102/0013189X031008018

10.

Brauckmann-Sajkiewicz

Pashiardis

Goldring

(2021). Comparing school leadership practices in Germany and the United States: Contexts, constructs and constraints. In Wilmers

Jornitz

(Eds.), International perspectives on school settings, education policy and digital strategies—A transatlantic discourse in education research (pp. 31–46). Verlag Barbara Budrich. https://doi.org/10.2307/j.ctv1gbrzf4.4

11.

Breakspear

(2012). The policy impact of PISA: An exploration of the normative effects of international benchmarking in school system performance. OECD Journals, 71, 1–32. https://doi.org/10.1787/19939019

12.

Brown

MacGregor

Flood

Malin

(2022). Facilitating research-informed educational practice for inclusion. Survey findings From 147 teachers and school leaders in England. Frontiers in Education, 7, Article 890832. https://doi.org/10.3389/feduc.2022.890832

13.

Cain

Brindley

Brown

Jones

Riga

(2019). Bounded decision-making, teachers’ reflection and organisational learning: How research can inform teachers and teaching. British Educational Research Journal, 45(5), 1072–1087. https://doi.org/10.1002/berj.3551

14.

Calude

C. S.

Longo

(2017). The deluge of spurious correlations in big data. Foundations of Science, 22(3), 595–612. https://doi.org/10.1007/s10699-016-9489-4

15.

Codd

J. A.

(1981). The concept of educational criticism. Discourse, 2(1), 28–30. https://doi.org/10.1080/0159630810020103

16.

Cramer

(2014). Theorie und Praxis in der Lehrerbildung [Theory and practice in teacher education]. DDS–Die Deutsche Schule, 106(4), 344–357. https://elibrary.utb.de/doi/abs/10.31244/dds.2014.04.04

17.

Cramer

(2021). Zwölf Jahre “visible learning” im wissenschaftlichen Diskurs [Twelve years of “visible learning” in scientific discourse: An interim review of the Hattie study]. Pädagogische Rundschau, 75(4), 393–408. https://doi.org/10.3726/pr042021.0036

18.

Danzinger

(1990). Constructing the subject: Historical origins of psychological research. Cambridge University Press.

19.

den Brok

Bergen

Brekelmans

. (2006). Convergence and divergence between teachers’ and students’ perceptions of instructional behaviour in Dutch secondary education. In Fischer

D. L.

Khine

M. S.

(Eds.), Contemporary approaches to research on learning environments: World views (pp. 125–160). World Scientific.

20.

Diemer

M. A.

Gore

P. A.

(2009). Culture and assessment: Nomothetic and idiographic considerations. Career Development Quarterly, 57(4), 342–347. https://doi.org/10.1002/j.2161-0045.2009.tb00120.x

21.

Eacott

(2017). School leadership and the cult of the guru: The neo-Taylorism of Hattie. School Leadership and Management, 37(4), 413–426. https://doi.org/10.1080/13632434.2017.1327428

22.

Eisner

E. W.

(1993). Forms of understanding and the future of educational research. Educational Researcher, 22(7), 5–11. https://doi.org/10.3102/0013189X022007005

23.

Elliott

(1986). Educational theory, practical philosophy and case study. In Hutchinson

White-House

(Eds.), Teacher research and INSET (pp. 74–89). University of Ulster.

24.

Falk

J. L.

(1956). Issues distinguishing idiographic from nomothetic approaches to personality theory. Psychological Review, 63(1), 53–62. https://doi.org/10.1037/h0047811

25.

Fanelli

(2010). “Positive” results increase down the hierarchy of the sciences. PLOS ONE, 5(4), Article e10068. https://doi.org/10.1371/journal.pone.0010068

26.

Fauth

Decristan

Rieser

Klieme

Büttner

(2014). Student ratings of teaching quality in primary school: Dimensions and prediction of student outcomes. Learning and Instruction, 29, 1–9. https://doi.org/10.1016/j.learninstruc.2013.07.001

27.

Fauth

Göllner

Lenske

Praetorius

A.-K.

Wagner

(2020). Who sees what? Conceptual considerations on the measurement of teaching quality from different perspectives. Zeitschrift für Pädagogik Beiheft, 66(1), 138–155. https://doi.org/10.3262/ZPB2001138

28.

Feinstein

A. R.

(1995). Meta-analysis: Statistical alchemy for the 21st century. Journal of Clinical Epidemiology, 48(1), 71–79. https://doi.org/10.1016/0895-4356(94)00110-C

29.

Fernandez-Cano

(2016). Una crítica metodológica a las evaluaciones PISA [A methodological critique of the PISA evaluations]. RELIEVE, 22(1), 1–17. https://doi.org/10.7203/relieve.22.1.8806

30.

Fraser

B. J.

(1998). Science learning environments: Assessment, effects and determinants. In Fraser

B. J.

Tobin

K. G.

(Eds.), The international handbook of science education (pp. 527–564). Kluwer.

31.

Gehlbach

Robinson

C. D.

(2021). From old school to open science: The implications of new research norms for educational psychology and beyond. Educational Psychologist, 56(2), 79–89. https://doi.org/10.1080/00461520.2021.1898961

32.

Glass

G. V.

Robinson

D. H.

(2004). An interview with Gene V. Glass. Educational Researcher, 33(3), 26–30. https://doi.org/10.3102/0013189X033003026

33.

Göllner

Fauth

Lenske

Praetorius

A.-K.

Wagner

(2020). Do student ratings of classroom management tell us more about teachers or about classroom composition? Zeitschrift für Pädagogik Beiheft, 66(1), 156–172. https://doi.org/10.3262/ZPB2001156

34.

Gorur

(2017). Towards productive critique of large-scale comparisons in education. Critical Studies in Education, 58(3), 341–355. https://doi.org/10.1080/17508487.2017.1327876

35.

Gräsel

(2011). Was ist Empirische Bildungsforschung? [What is educational research?]. In Reinders

Ditton

Gräsel

Gniewosz

(Eds.), Empirische Bildungsforschung (pp. 13–27). VS Verlag für Sozialwissenschaften.

36.

Grice

J. W.

Jackson

B. J.

McDaniel

B. L.

(2006). Bridging the idiographic-nomothetic divide: A follow-up study. Journal of Personality, 74(4), 1191–1218. https://doi.org/10.1111/j.1467-6494.2006.00407.x

37.

Hattie

(2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge.

38.

Hayman

Rayder

Stenner

A. J.

Madey

D. L.

(1979). On aggregation, generalization, and utility in educational evaluation. Educational Evaluation and Policy Analysis, 1(4), 31–39. https://doi.org/10.3102/01623737001004031

39.

Hill

H. C.

Kapitula

Umland

(2011). A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal, 48(3), 794–831. https://doi.org/10.3102/0002831210387916

40.

Hinzke

J. H.

Gesang

Besa

K. S.

(2020). Exploration of the use of research results by teachers. Relevance of research between theory and practice. Zeitschrift fur Erziehungswissenschaft, 23(6), 1303–1323. https://doi.org/10.1007/s11618-020-00982-6

41.

Howard

M. C.

Hoffman

M. E.

(2018). Variable-centered, person-centered, and person-specific approaches: Where theory meets the method. Organizational Research Methods, 21(4), 846–876. https://doi.org/10.1177/1094428117744021

42.

Hurlburt

R. T.

Knapp

T. J.

(2006). Münsterberg in 1898, not Allport in 1937, introduced the terms “idiographic” and “nomothetic” to American psychology. Theory & Psychology, 16(2), 287–293. https://doi.org/10.1177/0959354306062541

43.

Joram

Gabriele

A. J.

Walton

(2020). What influences teachers’ “buy-in” of research? Teachers’ beliefs about the applicability of educational research to their practice. Teaching and Teacher Education, 88, Article 102980. https://doi.org/10.1016/j.tate.2019.102980

44.

Jornitz

Wilmers

(2021). Transatlantic encounters: Placing education research interests in an international context. In Wilmers

Jornitz

(Eds.), International perspectives on school settings, education policy and digital strategies—A transatlantic discourse in education research (pp. 195–219). Verlag Barbara Budrich. https://doi.org/10.2307/j.ctv1gbrzf4.4

45.

Kemmis

(1978). Nomothetic and idiographic approaches to the evaluation of learning. Journal of Curriculum Studies, 10(1), 45–59. https://doi.org/10.1080/0022027780100104

46.

Konold

T. R.

Shukla

(2017). Estimating school climate traits across multiple informants: An illustration of a multitrait–multimethod validation through latent variable modeling. Educational Assessment, 22(1), 54–69. https://doi.org/10.1080/10627197.2016.1271705

47.

Krammer

Pflanzl

Lenske

Mayr

(2021). Assessing quality of teaching from different perspectives: Measurement invariance across teachers and classes. Educational Assessment, 26(2), 88–103. https://doi.org/10.1080/10627197.2020.1858785

48.

Krammer

Pflanzl

Mayr

(2019). Using students’ feedback for teacher education: Measurement invariance across pre-service teacher-rated and student-rated aspects of quality of teaching. Assessment & Evaluation in Higher Education, 44(4), 596–609. https://doi.org/10.1080/02602938.2018.1525338

49.

Krammer

Svecnik

(2020). Open Science als Beitrag zur Qualität in der Bildungsforschung [The contribution of Open Science to the quality of educational research]. Zeitschrift für Bildungsforschung, 10(3), 263–278. https://doi.org/10.1007/s35834-020-00286-z

50.

Lamiell

J. T.

(1998). “Nomothetic” and “idiographic”: Contrasting Windelband’s understanding with contemporary usage. Theory & Psychology, 8(1), 23–38. https://doi.org/10.1177/0959354398081002

51.

Lamiell

J. T.

(2019). Psychology’s misuse of statistics and persistent dismissal of its critics. Palgrave Macmillan. https://doi.org/10.1007/978-3-030-12131-0

52.

Lazarus

Sened

Rafaeli

(2020). Subjectifying the personality state: Theoretical underpinnings and an empirical example. European Journal of Personality, 34(6), 1017–1036. https://doi.org/10.1002/per.2278

53.

Lindl

Krauss

Schilcher

Hilbert

(2020). Statistical methods in transdisciplinary educational research. Frontiers in Education, 5, Article 97. https://doi.org/10.3389/feduc.2020.00097

54.

Makel

M. C.

Plucker

J. A.

(2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304–316. https://doi.org/10.3102/0013189X14545513

55.

Meehl

P. E.

(1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press.

56.

Miller

D. C.

Fonseca

F. T.

(2021). An average is just an average: What do we know about countries’ low- and high-performing students in mathematics? In Wilmers

Jornitz

57.

Molenaar

P. C.

(2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement, 2(4), 201–218. https://doi.org/10.1207/s15366359mea0204_1

58.

Mos

L. P.

(1998). On methodological distinctions: Nomothetic psychology, or historical understanding. Theory & Psychology, 8(1), 39–57. https://doi.org/10.1177/0959354398081003

59.

Neuweg

G. H.

(2002). Wenn die einen nicht können, was sie wissen, und die anderen nicht wissen, was sie können [When some can’t do what they know and others don’t know what they can do]. In Peter

Welte

(Eds.), Reflektierendes Lernen. Beiträge zur Wirtschaftspädagogik (pp. 86–103). Studien Verlag.

60.

Neuweg

G. H.

(2004). Figure der Relationierung von Lehrerwissen und Lehrerkönnen [On relations of teacher knowledge and teacher skills]. In Hackl

Neuweg

(Eds.), Zur Professionalisierung pädagogischen Handelns (pp. 1–26). LIT.

61.

Neuweg

G. H.

(2015). Evidenzbasierte Lehrerbildung [Evidence-based teacher education]. Journal für LehrerInnenbildung, 15(4), 54–58.

62.

Nunnally

J. C.

(1978). Psychometric theory (2nd ed.). McGraw Hill.

63.

Ones

D. S.

Viswesvaran

(1996). Bandwidth–fidelity dilemma in personality measurement for personnel selection. Journal of Organizational Behavior, 17(6), 609–626. https://doi.org/10.1002/(SICI)1099-1379(199611)17:6<609::AID-JOB1828>3.0.CO;2-K

64.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

65.

Organisation for Economic Co-operation and Development. (2000). Measuring student knowledge and skills: The PISA 2000 assessment of reading, mathematical and scientific literacy. OECD Publishing.

66.

Pant

H. A.

(2014). Aufbereitung von Evidenz für bildungspolitische und pädagogische Entscheidungen: Metaanalysen in der Bildungsforschung [Preparation of evidence for educational policy and pedagogical decisions: Meta-analyses in educational research]. Zeitschrift für Erziehungswissenschaft, 17(4), 79–99. https://doi.org/10.1007/s11618-014-0510-9

67.

Passon

von der Twer

(2020). Evidenz, Signifikanz und das kleine p [Evidence, significance and the small p]. Zeitschrift fur Bildungsforschung, 10, 377–395. https://doi.org/10.1007/s35834-020-00282-3

68.

Pigott

T. D.

Valentine

J. C.

Polanin

J. R.

Williams

R. T.

Canada

D. D.

(2013). Outcome-reporting bias in education research. Educational Researcher, 42(8), 424–432. https://doi.org/10.3102/0013189X13507104

69.

Plavnick

J. B.

Ferreri

S. J.

(2013). Single-case experimental designs in educational research: A methodology for causal analyses in teaching and learning. Educational Psychology Review, 25(4), 549–569. https://doi.org/10.1007/s10648-013-9230-6

70.

Plucker

J. A.

Makel

M. C.

(2021). Replication is important for educational psychology: Recent developments and key issues. Educational Psychologist, 56(2), 90–100. https://doi.org/10.1080/00461520.2021.1895796

71.

Renner

K. H.

Klee

von Oertzen

(2020). Bringing back the person into behavioural personality science using big data. European Journal of Personality, 34(5), 670–686. https://doi.org/10.1002/per.2303

72.

Robinson

D. H.

Levin

J. R.

Schraw

Patall

E. A.

Hunt

E. B.

(2013). On going (way) beyond one’s data: A proposal to restrict recommendations for practice in primary educational research journals. Educational Psychology Review, 25(2), 291–302. https://doi.org/10.1007/s10648-013-9223-5

73.

Rømer

T. A.

(2019). A critique of John Hattie’s theory of visible learning. Educational Philosophy and Theory, 51(6), 587–598. https://doi.org/10.1080/00131857.2018.1488216

74.

Rost

D. H.

Bienefeld

(2019). Nicht replizieren: publizieren!? [Don’t replicate: publish!?] Zeitschrift für Pädagogische Psychologie, 33(3–4), 163–176. https://doi.org/10.1024/1010-0652/a000253

75.

Salvatore

Valsiner

(2010). Between the general and the unique: Overcoming the nomothetic versus idiographic opposition. Theory & Psychology, 20(6), 817–833. https://doi.org/10.1177/0959354310381156

76.

Schleicher

(2018). World class: How to build a 21st-century school system, strong performers and successful reformers in education. OECD Publishing.

77.

Schmidt

Rosman

Cramer

Besa

K.-S.

Merk

(2022). Teachers trust educational science—Especially if it confirms their beliefs. Frontiers in Education, 7, Article 976556. https://doi.org/10.3389/feduc.2022.976556

78.

Singer

J. D.

(2019). Reshaping the arc of quantitative educational research: It’s time to broaden our paradigm. Journal of Research on Educational Effectiveness, 12(4), 570–593. https://doi.org/10.1080/19345747.2019.1658835

79.

Sjøberg

(2015). PISA and global educational governance—A critique of the project, its uses and implications. Eurasia Journal of Mathematics, Science and Technology Education, 11(1), 111–127. https://doi.org/10.12973/eurasia.2015.1310a

80.

Smeyers

(2015). Making sense of layers of interpretation in quantitative educational research. In Smeyers

Bridges

Burbules

N. C.

Griffith

(Eds.), International handbook of interpretation in educational research (pp. 1381–1408). Springer.

81.

Song

Sheldon

T. A.

Sutton

A. J.

Abrams

K. R.

Jones

D. R.

(2001). Methods for exploring heterogeneity in meta-analysis. Evaluation & the Health Professions, 24(2), 126–151. https://doi.org/10.1177/016327870102400203

82.

Sterling

T. D.

(1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54(285), 30–34. https://doi.org/10.1080/01621459.1959.10501497

83.

Terhart

(2011). Has John Hattie really found the holy grail of research on teaching? An extended review of visible learning. Journal of Curriculum Studies, 43(3), 425–438. https://doi.org/10.1080/00220272.2011.576774

84.

Terhart

(2013). Widerstand von Lehrkräften in Schulreformprozessen: Zwischen Kooperation und Obstruktion [Teacher resistance in school reform processes: Between cooperation and obstruction]. In McElevany

Holtappels

H. G.

(Eds.), Empirische Bildungsforschung (pp. 75–92). Waxmann.

85.

Thomm

Gold

Betsch

Bauer

(2021). When preservice teachers’ prior beliefs contradict evidence from educational research. British Journal of Educational Psychology, 91(3), 1055–1072. https://doi.org/10.1111/bjep.12407

86.

Travers

J. C.

Cook

B. G.

Therrien

W. J.

Coyne

M. D.

(2016). Replication research and special education. Remedial and Special Education, 37(4), 195–204. https://doi.org/10.1177/0741932516648462

87.

van der Zee

Reich

. (2018). Open education science. AERA Open, 4(3). https://doi.org/10.1177/2332858418787466

88.

von Wright

G. H

. (1971). Explanation and understanding. Routledge & Kegan Paul.

89.

Wentzel

K. R.

(2021). Open science reforms: Strengths, challenges, and future directions. Educational Psychologist, 56(2), 161–173. https://doi.org/10.1080/00461520.2021.1901709

90.

Wettstein

Ramseier

Scherzinger

Gasser

(2017). Unterrichtsstörungen aus Lehrer-und Schülersicht [Classroom disturbances from the perspective of teachers and students]. Zeitschrift für Entwicklungspsychologie und pädagogische Psychologie, 48(4), 171–183. https://doi.org/10.1026/0049-8637/a000159

91.

Windelband

(1921). An introduction to philosophy. Unwin.

92.

Wrigley

(2018). The power of “evidence”: Reliable science or a set of blunt tools? British Educational Research Journal, 44(3), 359–376. https://doi.org/10.1002/berj.3338

93.

Wrigley

(2019). The problem of reductionism in educational theory: Complexity, causality, values. Power and Education, 11(2), 145–162. https://doi.org/10.1177/1757743819845121

94.

Zhao

(2020). Two decades of havoc: A synthesis of criticism against PISA. Journal of Educational Change, 21(2), 245–266. https://doi.org/10.1007/s10833-019-09367-x

A cautionary note on aggregation in educational psychology and beyond

Abstract

Keywords

On aggregation in educational psychology

Aggregating to strive for nomothetic laws

On the nomothetic/idiographic distinction

The nomothetic/idiographic distinction in educational psychology

The nomothetic striving of educational psychology

On nomothesis in today’s educational psychology

A concrete example: Students’ and teachers’ perspectives on teaching quality

Aggregation’s impact on theory-to-practice

Final note

Footnotes

Funding

ORCID iD

Author biography

References