Uncertainty in Education: Policy Implications

Abstract

Uncertainty in education, both general and special, has long troubled educational researchers, reformers, and practitioners. Responding to students with special (atypical) educational needs is an example of decisions that are prone to error. Although some efforts to reduce uncertainty in education are reasonable and helpful, efforts to eliminate uncertainty completely are not only certain to fail but sometimes make matters worse. Changing the structure or framework of education, such as using educational tiers, does not reduce and may increase uncertainty. We offer recommendations for reducing uncertainty in education, given that it cannot be eliminated.

Introduction

Like all other professions and crafts, education requires many judgments. Our purpose is to discuss the particular judgments required of educators in deciding whether to refer a student for evaluation for special education and how a multi-tiered system of supports (MTSS) could make such judgments more challenging. That is, we consider how MTSS may inadvertently make special education referral more difficult, complex, and uncertain. Uncertainty is fundamental to any science, including a science of teaching (e.g., Kauffman, 2011; Kauffman & Farkas, 2021). Judgments must be made to have special programs of any kind—for example, in medicine, law, teaching, tax policy, finance, education, the military, triage, and so on. Uncertainty was the theme of an entire issue of Scientific American (September, 2019). Yet, teaching was not included in Scientific American as an example of an occupation involving “tough calls.” Nevertheless, education is a profession in which unexplained variability or “noise” (Kahneman et al., 2021) is inevitable. Teachers’ daily, “noisy” work involves many difficult and uncertain judgments, but the pervasive and perennial problem of the uncertainties faced by teachers, general or special, are seldom considered.

Teaching is “risky business” (Fischhoff, 2019), and it is one in which we (authors) have many combined years of experience in both public and private K-12 schools, both general and special education, and preparation of teachers in both general and special education. Teachers must make judgments about many different things in the course of their daily work. The decisions teachers make inevitably involve uncertainty, and therefore error in judgment is also inevitable (Kahneman et al., 2021).

Teachers are most likely to make mistakes or be criticized when the call is close—that is, when a student is really close to or on the line that is drawn in making a judgment (Kauffman & Lloyd, 2017). Every teacher, from preschool to graduate school has to draw a line (choose a criterion) and make a judgment about things like these: the student is doing ok/not ok; pass/fail; satisfactory/unsatisfactory; grade A/B/C . . . .

Those who teach higher grades and brighter students usually have the luxury of more time to consider their judgments—even the possibility of what Kahneman (2011) calls “slow” thinking. The younger and less capable the student, the more quickly judgments about performance must be made. Teachers of young children and teachers of students with learning difficulties often have to make judgments about their students’ performances while they are instructing. For example, they must decide whether to repeat something, alter their presentation in a particular way, return to a previously mastered task, take a break from the task, move on to the next task, and so on (see Pullen & Hallahan, 2015).

In the course of teaching, virtually all teachers must at some time weigh this decision: Should I refer this student for evaluation for special education; yes or no? The decision is not an easy one in many cases, especially given the variability in students, the expectations of teachers and students, and administrative procedures.

Referral and Identification for Special Education

Referral of students for possible identification for special education, as well as the identification process itself, is complex, often frustrating both school personnel and parents. Referral is typically, though not always, initiated by a teacher, usually prompted by concern for an academic problem, sometimes for a problem related to the student’s behavior (Lloyd et al., 1991).

Initiating the process typically involves specifying the teacher’s particular concerns and documenting what the teacher has tried but found unsuccessful. Evaluation for special education typically involves testing and observation by a school psychologist, presentation of the results to a multidisciplinary committee, and, if the student is found by that committee to have a disability and be eligible for special education, then preparation of an individual education program (an IEP; Bateman, 2017). The details of the process are not provided here, as they are described in more detail in most introductory textbooks in special education (e.g., Hallahan et al., 2023) and in even greater detail in writings on special education law and regulation (e.g., Bateman, 2007, 2017; Bateman & Linden, 2012; Rodriguez & Murawski, 2022; Yell, 2019). The federal law now called the IDEA (Individuals with Disabilities Education Act) makes every student with a disability eligible for special education if that disability affects educational performance.

Identification for special education and preparation of IEPs are prescribed by IDEA and related federal regulations (Bateman, 2017; Rodriguez & Murawski, 2022; Yell, 2019; Yell et al., 2017; Yell & Prince, in press). It is a laborious process, expensive in dollars, time, and emotion. Therefore, even if it is essential to having appropriate education for all students, it is something to be avoided when possible. Naturally, all involved are concerned about making mistakes or errors in either direction—not doing something they should (e.g., not referring a student who has a disability), or doing something they should not (e.g., referring a student who does not have a disability). The uncertainty around the referral and identification process is a concern not only of teachers but also of school administrators, psychologists, parents, and student themselves (Vannest et al., in press). Educators must be aware of different types of errors and consider which type is worse, which better represents the principle “better safe than sorry.”

Practitioners of any profession, including those who teach, worry about mistakes and think about the mistakes that will haunt them most. Conscientious practitioners are guided by the principle “better safe than sorry.” Teaching, like practicing any profession or craft, involves many judgments, and it is therefore important to consider types of errors.

Preferences for Types of Error

Judgments may be right or wrong. Errors in judgment may be classified as false positive or false negative. Judgments (decisions) may be yes (positive) or no (negative), yielding either a false positive error (yes when no is correct) or a false negative error (no when yes is correct).

Classifying errors of judgment can be quite confusing. Figure 1 may help in understanding error types. Begin with the question, “Was the judgment yes or no?” Then ask, “Was it right or wrong?” False positive means yes, but wrong; false negative means no, but wrong.

Figure 1.

Classification of judgments and errors.

A particular profession may have a preference for false positive or false negative judgments. That is, practitioners of a given profession may think one type of error is preferable to or worse than the other, given that an error is going to be made.

Preferences for different types of errors depend to a great extent on how the consequences of error in judgment are viewed. Typically, false negatives are preferred when the consequences of false positives are unjustified punishment, unjustifiable suffering, or needless restriction of an individual who, were it not for the mistaken judgment, would not have been subjected to harsh or inappropriate treatment (Kahneman et al., 2021). That is, if a positive or “yes” judgment carries with it something quite undesirable, then the false negative is preferred. However, if a positive judgment carries with it something beneficial, then the false positive is preferred, and the opposite error is considered undesirable.

In medicine, for example, a false positive judgment is usually preferred and considered less awful than a false negative. A judgment that, yes—a person has a disease—and being wrong (because later the person is found not to have it) is considered less terrible than a false negative judgment. If a physician decides no—that person does not have the disease—and is wrong (because later it was found that the person really does have it), that false negative is a worse error. And the more serious the disease, the more this is the case. Physicians think their treatments in the vast majority of cases actually help more than they harm, even if they cause some pain or discomfort during treatment. People guided by a medical model try to play it safe by preferring the false positive because they do not want to miss a medical problem and let it go untreated, especially if the untreated condition results in something very harmful. Of course, depending on the matter in question, a false positive is horrific (e.g., amputating the wrong limb or prescribing an unneeded drug that has serious side effects and/or is addictive).

In contrast to a preference for false-positive errors in medicine, a false negative error in law is preferred. A false-positive error in law might be a verdict of guilty when the suspect was really innocent. Lawyers tend to think it is better to let a guilty person go free than to convict someone who is innocent, and the more serious the legal charge and punishment the more this is the case. In the practice of criminal law, especially, a false-positive error is considered the more terrible, even horrific type, the type that should be most carefully avoided. The false negative is preferred.

Because of uncertainty about errors in identification for special education, the issues involving racial and ethnic disproportions are extremely fraught. Although the disproportionate number of African American students receiving special education has typically been considered unfairly discriminatory, some research strongly suggests that African American students are often unfairly denied special education. Part of the issue is whether special education is viewed as helpful or harmful (see Anastasiou & Kauffman, 2019; Farkas et al., 2020; Harris, 2022; Kauffman & Anastasiou, 2019; Kauffman & Farkas, 2021; Whitford & Carrero, 2019). Views of special education add weight to the false-positive/false negative error problem; the more beneficial special education is thought to be, the greater the fear of the false negative, and the more harmful special education is thought to be, the greater the fear of the false positive.

Concern for avoiding false-positive errors (i.e., misidentification for special education) in identifying ethnic minorities have made many educators hypersensitive to such mistakes (e.g., Harry & Klingner, 2014; Whitford & Carrero, 2019). Over-representation or misidentification of minority students is, indeed, a serious problem, although that difficulty can also be used as an excuse or rationale for denying special education for children with special needs (e.g., Harris, 2022).

One of the major considerations in special education is that both false positives and false negatives can have horrific consequences for the students involved. A student’s not receiving appropriate special instruction can negatively affect his or her acquisition of important skills and life course. However, false identification for special education risks unnecessary stigma, inappropriate expectations, possible degradation of instruction and lack of exposure to appropriate curriculum, and life-long lack of opportunities. Thus, the matters of referral and identification for special education are particularly weighty.

At least two ways of tackling this problem of weighty uncertainty deserve full consideration, and they are not mutually exclusive. One is improving special education so that the negative consequences of false-positive errors are minimized (e.g., Kauffman, in press). Another is improving the teaching process in general education to try to reduce errors of both kinds (e.g., Lane et al., 2020). Errors of judgment, regardless of whether they are false positives or false negatives, are to be avoided whenever possible (Kahneman et al., 2021). Therefore, various professions take precautions designed to reduce errors.

Given what we know about human beings, we expect that some individuals will make more mistakes than others. One way to protect against mistakes is by increasing the number of judges (Kahneman et al., 2021). Uncertainty is the rationale for a jury system in law, so that verdicts in a jury trial do not depend on a single person’s judgment. The requirement of unanimity in jury verdicts is a guard against the false positive. Second opinions are often appropriate and encouraged in medicine to try to avoid errors of both types.

Federal special education regulations require that interdisciplinary committees make judgments about eligibility for special education, and although unanimity is not required, a consensus is. The message to educators is that a single person may not make important decisions about special education, and in this regard special education is more like law than like medicine. A clear preference for false negatives and abhorrence of false positives also makes special education more like law than like medicine in weight of errors.

The independence of multiple judgments requires that individuals not reveal their judgment to avoid influencing the group (Kahneman et al., 2021). Medicine and other professions concerned about errors now often build in procedures designed to safeguard judgments by requiring or recommending multiple judges, multiple judgments, clearer communication, more and better teamwork, or redundancies whenever possible.

Uncertainty, Prevention of Errors, and MTSS as Framework

Among recent attempts to reduce uncertainty and prevent false-positive errors is the idea of more tiers or MTSS (Wiley et al., in press). The basic idea is that typical or general education should be structured around the idea that students vary in their need for supports, that some need more support than others, and that individual students will be taught in the tier (or with the level of support) most appropriate for them.

The essential message of MTSS and its variants to teachers is this: “If a child is doing ok in your class, we’ll call that Tier 1. Most of your students will do well in Tier 1 if you’re using effective, evidence-based instructional and behavior management practices. However, if a student still isn’t doing well after you’ve made sure you are using evidence-based practices, then you should try some modifications that intensify your instruction, and that will be called Tier 2. Tier 2 involves doing something special, a step-up of intensity. If Tier 2 doesn’t work either, then the student should receive Tier 3 interventions, which involve interventions even more specialized or precisely targeted." Tier 3 might or might not be considered special education, and it does not necessarily mean a change of place (i.e., instruction somewhere other than the general education classroom). However, as Fairbanks et al. (2007) noted, “guidelines are still needed for knowing when to implement and when to increase the intensity of interventions across the three-tiered logic” (p. 308).

MTSS may well be a general framework and be supported by philosophical propositions that are consistent from one place to another, but specified processes vary greatly from one school to another when it comes to implementation. Educators use the framework’s philosophy to generate their own unstandardized processes. Judgments that are required in these processes include the following, all sources of uncertainty:

• When can an intervention be discontinued because the student no longer needs it?

• What constitutes an adequate response to a support?

• Why are particular interventions or supports provided in one tier but not others?

Evaluating MTSS empirically and quantitatively may be impossible, so long as it is ill-defined as a standard process. That is, the quality of the decisions cannot be evaluated and replications/comparisons across studies cannot be accomplished without clear definitions and consistent implementation. If MTSS is intended to be merely a framework that educators use to generate their own unstandardized processes, then the effect of using that framework will be diffuse. It will be impossible to distinguish statistical signal (effect) from noise (random variability).

Sources of variability in education include at least these:

• Nature of the individual student’s learning difficulty

• Magnitude or severity of the individual student’s learning difficulty

• The individual student’s personality and family support

• Teacher/curriculum effectiveness

• Teacher-student interactions

• Timing of decisions

• Tools used to make decisions

• Background, knowledge, training, and bias of the decision maker(s)

• Number of opinions used to make decisions

• Level of dependence among multiple opinions used to make decisions

• Method used to combine multiple opinions to make decisions

• Suitability of interventions for the nature of the difficulties

• Effectiveness of interventions

• Fidelity of implementation of interventions

• Tier distinctions and definitions

Furthermore, the outcomes by which the effectiveness of any framework may be judged include at least these:

• Number/proportion of students referred for special education

• Number/proportion of students diagnosed with a disability

• Time elapsed between identification of a learning difficulty that is a disability and referral for special education

• Level of academic achievement, self-care, independence, and post-school adjustment of students with disabilities

Tiered systems of education do have definite advantages (Kauffman, 2021; Kauffman et al., 2019; Lane et al., 2020; Wiley et al., in press). MTSS and its variants are intended to improve instruction and to be preventive as well, so that problems are caught in their incipient stages and intervention is not long-delayed. Certainly, catching problems as early as possible is a good idea. It is not wise to let problems fester until they become obvious disabilities and everyone agrees the student is having extreme difficulty and should be identified for special education. Surely, providing help for both students and their teachers is an excellent idea. Reducing unnecessary referrals, the kinds of mistakes known as false positives, also is a fine objective (e.g., see Kauffman et al., 2019; Walker & Gresham, 2014).

We do not suggest that MTSS and related multi-tiered frameworks have no merit, that they are simply bogus attempts to improve education. True, adherents of MTSS may contend that students who would benefit most tier 2 in a three-tiered system are consistently misclassified in the traditional system of general and special education. Under this assumption, the standard system would create as many or more errors than MTSS. Tier 2 in three-tiered MTSS would include students at risk, such as students who are English language learners (ELL), some culturally and linguistically diverse students, and/or students from low socioeconomic backgrounds who need extra attention and targeted support but do not have disabilities and should not be classified as eligible for special education (e.g., see Carlson & Parshall, 1996; Carlson et al., 2008; Holden-Pitt, 2005).

Although tiers may be a good idea for doing some things, there are things they cannot do, such as providing all possible supports for all possible students in a general education environment. Moreover, Kauffman (2021) has raised the issue of “tracking” and its possible existence as part of MTSS.

One thing not clear is precisely how the implementation of tiers is not a form of “tracking.” In one sense, it appears to be a refined form of tracking, in that observed differences in students’ learning and behavior are used to justify different designations and instruction. Perhaps it is more explicit, defensible, flexible, but relabeled tracking.

We also need a descriptive decision framework to study agents’ choices under conditions of uncertainty. “Agents” in this case are decision-makers in the matter of special education, including parents as well as educators. What an agent chooses to do on any given occasion is determined by rational calculation and by beliefs, values, and desires (Steele & Stefansson, 2020). A purely rational calculation is not the only factor involved in decision-making; affective and predisposition factors are also important (Bunge, 1996). Behavioral economists have found that people tend to be situation minimizers rather than situation maximizers (i.e., to minimize risk of harm rather than maximize risk of help; Bunge, 1996). Whether parents are education-minimizers or special education-maximizers when seeking additional services is unexplored. But if special education is portrayed as non-beneficial, this might favor education-minimizers and intermediate Tier(s). This may partially explain why the percentage of students aged 6–21 who received special education in the category of learning disabilities dropped over a 17-year period from 48.3% to 37.1% (U.S. Department of Education, 2006, 2022, pp. 21–42).

MTSS and its variants may well be good-faith efforts to provide a different framework for students with relatively mild educational disabilities and those whose responsiveness to interventions is ambiguous (Wiley et al., in press). Those students whose educational difficulties are more severe and obvious, whose needs for special instruction are unambiguous, could then be served immediately in Tier 3 or whatever tier is determined to be special education.

This interpretation of MTSS suffers from at least two problems. First, there is the insistence of some that “all means all” (e.g., Slee, 2018, 2020; SWIFT Schools, 2021) and that even those students with severe and obvious educational disabilities are to be accommodated in general education, such that special education does not result in any physical separation or placement in a dedicated environment. This seems certain to reduce inclusion to an ideological struggle (see Anastasiou, et al., 2018; Imray & Colley, 2017; Kauffman, 2020; Kauffman et al., 2019; Kauffman & Badar, 2020). Second, the problem of a line separating a mild from a severe educational disability does not avoid the problems of judgment, bifurcation of the population of students, labeling, stigma, and other problems. The line separating mild from severe disabilities or special needs will remain dependent on human judgment, as will the line between disability and no disability. Furthermore, it is important to recognize that some disabilities judged severe are not educationally severe (e.g., quadraplegia).

MTSS and Uncertainty

Perhaps by adding judgment points we add more uncertainty or noise in any endeavor. However, people sometimes inadvertently make additional problems for themselves and others in the course of trying to make things better. This can be done by adding more judgments to those required. It can also happen when a rule precludes noise but adds injustice (Kahneman et al., 2021). MTSS and its variants might add noise (random or unwanted variability) to educational decisions, perhaps even those involving special education. The problem is that a line and attendant judgments are required for every tier. Traditional special education involving only two tiers requires only one judgment—that between special education and general education. But all three-tiered systems, require two judgments, not one. With MTSS, one judgment is required to distinguish between Tier 1 and Tier 2 and a second between Tier 2 and Tier 3. That is, as more tiers are added, more judgments are required, introducing the possibility of more errors. Nevertheless, the proposition that all errors are equal is not justifiable on either logical or moral grounds. That is, some errors are more consequential than others.

The concept involving numbers of tiers and judgments may seem intuitive. That is, someone has to draw a line and make a judgment for each tier, and every one of these has a margin of error such that judgments may be wrong. Therefore, more tiers mean not only more judgments but more chances for error, but those errors may not be equal in importance for students or teachers. Criticisms of special education include the error problem, especially false-positive mistakes. Criticisms of all education include the criteria used to draw lines, the issues of sorting students into categories and giving those categories names or labels based on judgments about their relationships to the judgments and stigma associated with identification for special services—all those things having to do with making decisions, chances for error, bias, and making things worse rather than better.

Although MTSS could be a step in the right direction, it could also create even more problems. Creating a “sort-of” special education (which might be Tier 2 or Tier 3 or both) may be a very good idea. But ideas are not good just because they are new and different. Something new and different from status quo can be worse than status quo—a change could make things worse than the way things are. Change can be regressive. Perhaps special education would be better off as a policy and profession if it focused on making two tiers work well before trying three. Perhaps not. Uncertainty looms again.

MTSS might solve many problems, though not the core problems of special education—deciding not only which students to serve, but with what services and where (Kauffman, 2021). Perhaps the typical three tiers of MTSS are not enough and more tiers are better, such that 3 are better than 2 and 4 are better than 3 and 5 are better still. Uncertainty about the optimal number of tiers is highly predictable.

A related question about MTSS is whether students in Tier 2 have any of IDEA’s protections, and, if so, then which ones. Tier 3, then, presumably would be reserved for those who have been identified as having disabilities and granted all the protections of IDEA. However, that, too, is open to interpretation.

An open question is whether MTSS is subject to interpretation by local education authorities, and another such question is who should decide, and how, that a problem or disability obviously requires special education, regardless of interpretation by federal or local education authorities. That is, when, if ever, should a student receive instruction in the highest tier? What is not in doubt is that tiers add to the complexity of teaching and education policy

“Advocates of tiered education faced with criticism or skepticism often claim they are suggesting only a framework, a way of describing an array of interventions and resources” (Kauffman et al., 2019, p. 18). Granted, tiers are a framework, but that framework is implemented using a philosophy about what will work best in teaching. Indeed, guidelines are one approach to reducing error or noise in any system (Kahneman et al., 2021). There is also the matter of simplicity and the question of the extent to which something is doable in education. Two education researchers concluded this

By suggesting that some schools consider implementing a two-tier rather than three-tier framework, we are not saying that less complex frameworks are as effective as more complex ones. In principle, we would expect a three-tier approach to be more successful with more children. But in reality, many schools are not deciding between three and two tiers. They are struggling just to make Tier 1 work. . . . In considering a two-tier alternative, it is important to remember that the conventional three-tier approach, however logical and consistent with best practices, is without empirical validation. We do not say this dismissively or to be contentious or provocative. Rather, our point is simply that there is nothing sacrosanct about a three-tier framework. (Fuchs & Fuchs, 2017, p. 266, p. 266)

Educators often seem to forget that traditional special education involves two tiers: general education and special education. That is, students are either in general education, unidentified as having a disability, or special education, in which case IDEA applies. Today’s IDEA (and its precursor in law) seeks to bring consistency to the special education tier. It simply does not address students in Tier 1. IDEA thus presents a dilemma in education reforms in which funding supported by it are applied in Tier 1.

Conclusion

All education is likely characterized by a lot of unwanted variability or noise. No problem-free response to the differences we see among students now exists, and it is highly predictable (a virtual certainty) that none will be found—ever. Restructuring will not solve this problem. Placing all students in general education will not solve it. Moreover, the fundamental problem of special education—the very old observation that students differ widely in what they know and what they need to learn (Horn, 1924)—is never going to disappear.

Uncertainty is certain about many aspects of education, including identification of children for special education as well as for any other level of support. The optimum level of intensity of instruction or tier for any child is always in question. Judgments are required in all cases. This is merely a recognition of the facts: (a) teachers are humans and will make errors in judgment, (b) humans, including students and their teachers, are extremely varied in characteristics that are important in education, and (c) special educators are concerned about the effective instruction of all children, but especially those with education-related disabilities. Educators are stuck with discomforting uncertainties, but those uncertainties are much like those encountered in all human endeavors.

Recommendations

More research on decision-making in education is needed. Particularly important are studies of false positive and false negative errors and their implications for various educational projects. Identifying and reducing errors of judgment are important, and the costs of reducing errors or noise must be weighed carefully to avoid adding injustices to a noisy system.

Regarding policy related to special education, we recommend a focus on how educators can make the IDEA work as it should rather than invent new frameworks, structures, or philosophies or try to reform only general education to make it appropriate for all learners. That is, the policy already in place (IDEA) has been described as “a solution hiding in plain sight” (Cannon et al., 2013). The most helpful strategy is, perhaps, better implementation of the existing policy, not a policy change that is likely to increase uncertainties without improving justice.

Evaluating MTSS empirically may be impossible, so long as it is defined as a framework open to local interpretation and implementation. That is, the quality of the decisions cannot be evaluated and replications/comparisons across studies cannot be accomplished without clear definitions. If MTSS is intended to be merely a framework that educators use to generate their own unstandardized process, then the effect of using that framework will be diffuse. It will be practically impossible to distinguish statistical signal (effect) from noise (random variability).

All education is, in many ways, a particularly uncertain business and one in which greater accuracy and consistency of judgment is desirable. Reducing uncertainty to the optimal level of noise is a scientific task of great importance. Noteworthy also is the realization that reducing noise can be costly and that noiselessness is not necessarily an appropriate goal.

Good hygiene in making judgments includes slow and careful deliberation, aggregating judgments, and making sure multiple judges make their decisions independently. But, even so, it is essential to remember that “Decision hygiene practices also have their downsides: if poorly managed, they risk bureaucratizing decisions and demoralizing professionals who feel their autonomy is being undermined” (Kahneman et al., 2021, p. 349).