Abstract
Facilitating students’ acquisition of higherorder thinking skills is imperative in the 21st century. Although some types of text have been shown to enhance higherorder thinking, the effects of many novel forms of text have yet to be investigated. As such, the purpose of the present study was to explore the extent to which a relatively novel form of text (i.e., intratextual persuasive message) served as a catalyst for students’ higherorder thinking as evidenced in the quantity, quality, and content of their arguments before and after reading. The findings revealed that the quantity of students’ arguments increased from prereading to postreading and the content of the reasons provided by the students was more in line with those of the authors, whereas the quality of students’ arguments decreased over time. Interestingly, relatively few students altered their position on the central question from the text. Rather, the nature of the data indicated that students engaged in case-building as they read the text. As a result, the intratextual persuasive message was only minimally effective at enhancing students’ higherorder thinking. Implications for research and practice are forwarded.
Arguably, during no other time in documented history have individuals been inundated with as much or as varied kinds of information as they are in modern society (Weare & Lin, 2000). The sheer vastness and complexity of information places new demands on learners. Individuals must learn to be facile in comprehending, analyzing, synthesizing, and evaluating complex, multidimensional texts (Coiro & Dobler, 2007), that is, individuals must be adept at higherorder thinking (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956). Without a doubt, higherorder thinking is a fundamental 21st-century skill.
In the present study, we explore the effects of a relatively novel form of a complex text (i.e., intratextual persuasive message) on college students’ higherorder thinking. Intratextual persuasion refers to a single message consisting of two or more contrasting perspectives or text-segments written by different authors on a controversial topic (Andiliou, Ramsay, Murphy, & Fast, 2012). In addition, given that two text-segments written by two opposing authors on two opposing perspectives are delivered as a single message, it is important to denote that this is unique in that it cannot be classified into one of the traditional persuasive argument text structures. Because particular types of text have been shown to serve as catalysts for higherorder thinking in the classroom (e.g., Allen, 1991; Guzzetti, 1990; Guzzetti, Snyder, Glass, & Gamas, 1993; Leu et al., 2007), our exploratory investigation examines the extent to which a particular form of complex text (i.e., intratextual persuasive message) serves as a catalyst for students’ higherorder thinking as evidenced in the quantity, quality, and content of arguments formed at pre- and postreading.
Introduction
Higherorder Thinking
Within educational research and classroom contexts, the term higherorder thinking is most often attributed to the pioneering work of Benjamin Bloom and his colleagues in the 1950s (Bloom & Krathwohl, 1956), where they set forth a taxonomy for cognitive learning. Interestingly, Bloom’s original taxonomy has undergone only two substantive revisions since its inception. First, the revision of the taxonomy, led by Bloom’s former student and his original coauthor, served to update and enhance the categories of the taxonomy to better align with various types and levels of knowledge, including declarative, conceptual, procedural, and metacognitive knowledge (Anderson & Krathwohl, 2001). Later, Churches (2008) incorporated additional examples of ever-evolving digital technologies (e.g., posting and linking) into the categories of Anderson and Krathwohl’s revised taxonomy. Indeed, this taxonomy has had a profound influence on educational research, such that the levels of knowledge, comprehension, and application are traditionally understood as lowerorder thinking skills, and the function levels of analysis, synthesis, and evaluation are understood as higherorder thinking skills (e.g., Wimer, Ridenour, Thomas, & Place, 2001; Zohar & Dori, 2003).
Of course, not all researchers have conceptualized higherorder thinking in the same way as Bloom. Recognizing that higherorder thinking does not lend itself to a straightforward definition, Resnick (1987) identified characteristics of higherorder thinking to outline the construct in greater detail. According to Resnick (1987), higherorder thinking is effortful, complex, nonalgorithmic; requires uncertainty, self-regulation, nuanced judgment, the application of multiple criteria, imposing meaning; and possibly even yields multiple solutions (p. 3). Furthermore, as Resnick (1987) and Schraw, McCrudden, Lehman, and Hoffman (2011) suggested, it is important to recognize that higherorder thinking is not prompted exclusively by the traditionally conceptualized advanced cognitive activities. Instead, activities such as reading, critical thinking, or argumentation often afford the learner with opportunities for higherorder thinking and prompt deeper understanding. Particularly relevant to the present study is the emphasis that reading may act as a catalyst for higherorder thinking or that an argumentation task could serve as an assessment tool for gauging higherorder thinking, which will be addressed later in further detail.
Finally, Alexander and colleagues (Alexander et al., 2011) offered a reconceptualization of higherorder thinking that is especially pertinent, emphasizing that it cannot be understood without due consideration of the multidimensional, dynamic, and fluid interplay between the intellectual processes and one’s orientation toward knowledge and knowing. Specifically, Alexander et al. (2011) have defined higherorder thinking as “. . . the mental engagement with ideas, objects, and situations in an analogical, elaborative, inductive, deductive, and otherwise transformational manner that is indicative of an orientation toward knowing as a complex, effortful, generative, evidence-seeking, and reflective enterprise” (p. 53). Furthermore, Alexander et al. stipulate that while higherorder thinking can be understood similarly across domains, it is fundamental to recognize that the domain in which the task is situated gives rise to “characteristics of what is to be known (the what) as well as to the circumstances under which it is encountered (the when and the where and by whom)” (Alexander et al., 2011, pp. 53-54).
As conceptualized by Alexander et al., the what refers to the object of intellectual focus, which in the domain of reading refers to the nature of the text. Like Resnick (1987), Alexander et al. emphasized the task of reading in stimulating higherorder thinking; importantly, Alexander et al. (2011) contended that higherorder thinking is more apt to occur when the inherent or perceived complexity of the text invites higherorder thinking (e.g., contrasting opinions on a controversial issue). In comparison, the who is best understood as the defining features of the learner, including one’s knowledge, beliefs, and motivations that influence thinking and processing of information. The where dimension can best be understood in relation to the educational, social, and cultural context (i.e., place), while the when dimension might be best interpreted as the learner’s developmental trajectory in a domain like reading.
As repeatedly underscored in the extant text processing literature, there is necessarily interplay between the what and the who in higherorder thinking and processing (Dole & Sinatra, 1998; Murphy & Mason, 2006). In the present study, we have focused our investigation on the complex interplay between the what and the who aspects of higherorder thinking, while experimentally attempting to hold the where and when relatively constant across our participants by using a systematic data collection process and selecting learners at similar developmental stages.
The What Dimension of Higherorder Thinking: An Intratextual Persuasive Text
For a given reading task, the what is the text. The present study is part of a larger project (Andiliou et al., 2012) examining an intratextual persuasive message. Specifically, this text presents a single controversial issue with a novel argument structure (i.e., two one-sided text-segments expressing opposing views). Because two one-sided text-segments addressing the same issue are presented simultaneously within a single overall message, intratextual processing is required. For the purpose of this article, we draw from two facets of existing research: intratextuality and persuasion; it is at the crossing of these facets where the what dimension of higherorder thinking lies. In addition, to articulate the complexity and relevance of this novel text type, information regarding text structure and potential situations where readers may encounter this type of text is also provided.
Intratextuality
In the current study, a reader is presented with a single message in which two opposing one-sided text-segments are presented on a controversial issue with each one written by a different author. Deep comprehension of the message places an additional demand on the reader, as a complete understanding requires the reader to make intratextual connections between both opposing text-segments. As such, we will draw some of our framing from relevant theoretical and empirical research in the multiple documents literature.
Both Rouet’s (2006) Task-Based Relevance Association and Content Extraction (TRACE) Model and Rouet and Britt’s (2011) updated Multiple Documents Task-based Relevance Association and Content Extraction (MD-TRACE) Model directly address many of the critical components of complex, multiple situation models. Components from the older Documents Model (Perfetti, Rouet, & Britt, 1999) remain relevant. The Situations Model recognizes the plurality of situations developed from the complex documents (Perfetti et al., 1999), purporting that unique and overlapping information may be presented with multiple documents (Britt, Perfetti, Van Dyke, & Gabrys, 2000); the Intertext Model indicates the importance of information accompanying the text, including the documents’ source, content, and rhetorical goals, along with the relationship between the documents. This model is critical to the research presented here, as within the intratextual persuasive message, two text-segments are presented by two different authors. Readers must navigate through both segments’ content and source information, thus creating both a Situations and Intertext Model of the message.
For example, Hartman (1995) collected and analyzed think-aloud data from eight participants reading a set of five naturally occurring passages and derived three stable profiles of readers. The first profile was that of the intratextual reader. Students classified as intratextual readers generated a substantial number of think-aloud utterances that identified connections within a text (i.e., related the currently read information to information that was read previously in that same text). Furthermore, these intratextual links were classified as local, regional, or global, depending on the distance between connected information. Local intratextual links were made between sentences within a paragraph, regional intratextual links were made between paragraphs, and global intratextual links were made across the passage as a whole; the generation of local links was more common than both regional and global links. This finding is particularly relevant to the present study. With respect to our intratextual persuasive message, readers would be required to generate connections across the two one-sided text-segments, thus generating intratextual links at the global level. The second profile was that of an intertextual reader. The intertextual reader was characterized by a pattern of situating the current text with respect to previously read texts. While reading a text, the information from previously read texts, “influence[d] their ongoing understandings . . . while they read” (Hartman, 1995, p. 545). Finally, the last profile was for the extratextual reader. Extratextual readers focused on generating connections between the currently read text and that of previous experiences or prior knowledge. Hartman (1995) called this “read[ing] beyond” the text, explaining how one reader situated “her understandings mostly in terms of her own personal preferences, and occasionally to her own experience or knowledge” (p. 546). Indeed, the multiple document literature is primarily concerned with research on multiple texts. Additional research is needed to consider the effects of intratextual persuasive messages and to determine whether students reading intratextual persuasive messages also engage some of the heuristics present in intertextual reading. The current study attempts to address that limitation.
Persuasion
In the present research, we have adopted a broad definition of persuasion that draws on both the conceptual change and persuasion literatures. More specifically, we define persuasion as an interactive process through which the knowledge and beliefs of the receiver (e.g., student) are modified or altered in some way as a result of hearing, reading, or otherwise engaging an oral or written communication (Murphy & Alexander, 2004). As such, any oral or written message designed to counter the existing knowledge and beliefs of a learner and then present new ones can be referred to as a persuasive text (Chambliss & Garner, 1996).
The rich literatures from which this definition was drawn highlight the effects of particular text features on learner processing. For example, empirical evidence suggests that the comprehensibility, interestingness, author credibility, and understandability of a text influence the extent to which students process or engage the text (e.g., Alexander & Jetton, 1996; Buehl, Alexander, Murphy, & Sperl, 2001; Murphy & Alexander, 2004). Simply put, to prompt higherorder thinking, persuasive texts must be considerate and offer a message that is clear, understandable, and even useful to the reader (Armbruster & Anderson, 1984).
Persuasive texts are best described as having an argument structure (Meyer, 2003). Numerous studies have shown that both the structure and content of the argument influence the persuasiveness of a text (e.g., Allen et al., 1990; Guzzetti et al., 1993). At least three distinct argument-based text structures have been studied in the extant literature: one-sided, two-sided nonrefutational, and two-sided refutational texts (Allen et al., 1990). A one-sided text presents a single viewpoint while a two-sided nonrefutational text presents two conflicting positions yet keeps a neutral perspective (Guzzetti et al., 1993). In contrast, a two-sided refutational text presents two opposing stances but discounts one in favor of the other (Guzzetti, 1990; Sinatra & Broughton, 2011). Findings from numerous studies (Allen, 1991) and meta-analyses (Allen, 1993; Guzzetti et al., 1993) on argument-based text structure suggest that two-sided refutational texts are more persuasive than either two-sided nonrefutational or one-sided texts in changing readers’ views and in correcting misconceptions. By contrast, there have been mixed findings regarding the persuasiveness of two-sided nonrefutational versus one-sided texts (Allen et al., 1990). In one meta-analysis, there were no differences between the effectiveness of two-sided nonrefutational texts and one-sided persuasive texts (Jackson & Allen, 1987), while in another study, one-sided texts were more effective in changing readers’ positions (Allen, 1991).
To further explore the effectiveness of two text structures, Buehl et al. (2001) examined readers’ beliefs and responses to text features, both before and after reading a two-sided nonrefutational text about V-Chips and a one-sided persuasive text about educational reform. They found that the two-sided nonrefutational text changed the readers’ positions more than the one-sided text, but the one-sided text “was more effective at maintaining or changing readers’ views relative to the author’s premise” (Buehl et al., 2001, p. 285). Essentially, Buehl et al. found tentative evidence that case-building and biased processing occurred as readers selectively attended to arguments that aligned with their prior perspectives. It seems that without refutation, readers are not as compelled to consider alternative perspectives.
A current limitation in the persuasion literature, however, is that research exploring the effects of students’ interaction with varied types of argument structures has primarily focused on the one-sided, two-sided nonrefutational, and two-sided refutational texts (e.g., Jackson & Allen, 1987; Murphy & Alexander, 2004). Additional research is necessary to consider the effects of persuasive texts that do not fall into any of the three previously described argument structures. The current study, in line with our previous research using an intratextual persuasive message (Andiliou et al., 2012), attempts to address this limitation.
The Who Dimension of Higherorder Thinking: Characteristics of the Learner
Irrespective of the what, specific characteristics of the who interacting with a text impact the degree of higherorder thinking. Alexander et al. (2011) reported various learner characteristics that have been connected with higherorder thinking (e.g., background knowledge, self-regulatory behaviors, or goal orientations). Of particular interest in the present study are individual differences in prior knowledge and beliefs.
Prior Knowledge
Knowledge is a multidimensional concept defined in many different ways in the literature on concept and belief change (Murphy & Alexander, 2008; Murphy, Alexander, & Muis, 2012; Southerland, Sinatra, & Matthews, 2001). We adopted the perspective that knowledge “refer[s] to all that is accepted as true that can be externally verified and can be confirmed by others on repeated interactions with the object (i.e., factual)” (Murphy & Mason, 2006, p. 306). Furthermore, knowledge is often associated with lower levels of affect and importance (Alexander & Dochy, 1995; Alexander, Murphy, Guan, & Murphy, 1998).
Educators have difficulty modifying or altering the knowledge of students with either relatively limited or extensive topic-based understandings. Individuals with a limited topic knowledge base tend to integrate inconsistent information and experience difficulty with identifying their own misconceptions of evidenced, scientific knowledge (Murphy & Mason, 2006). Individuals with relatively lower levels of knowledge may not modify existing knowledge because they fail to identify discrepancies between their current knowledge and more accurate information (Limón, 2001, 2003). By contrast, individuals with high levels of topic knowledge can more efficiently analyze oral or written arguments critically (e.g., Johnson, 1994; Kardash & Scholes, 1995). These individuals are also more likely to defend their current knowledge and to identify the limitations of alternative perspectives (Murphy, 1998). As a result, individuals with higher levels of topic knowledge are more resistant to persuasion after instruction than individuals with lower levels of topic knowledge (Murphy & Mason, 2006).
Beliefs
In the present study, we have conceptualized beliefs as referring to “all that one accepts as or wants to be true. Beliefs do not require verification and often cannot be verified (e.g., opinions)” (Murphy & Mason, 2006, pp. 306-307). Beliefs, unlike knowledge, are more strongly related to affect and are characterized by a higher level of importance to the individual. As a result, individuals tend to embrace their beliefs when presented with conflicting evidence and are more likely to behave in accordance with their beliefs than with the new concepts (Garner & Hansis, 1994). However, as highlighted by Lord, Ross, and Lepper (1979), there are times when individuals are willing to look beyond their initial beliefs and weigh evidence presented in a given message. Some researchers (e.g., Murphy & Alexander, 2004) have suggested that belief change is due in part to the interaction of person characteristics (e.g., level of prior knowledge or beliefs) with characteristics of the text (e.g., nature of the argument). For example, Nussbaum and Kardash (2005) found that students who espoused extreme beliefs were unaffected by balanced textual arguments. Instead, the balanced arguments actually enabled students with extreme beliefs to generate additional one-sided arguments in support of their position. Jonassen and Kim (2010) suggested that students were more likely to argue solely based on their own beliefs because teachers did not provide them with the chance to engage in argumentation due to the multitude of pressures to cover content in a limited time frame.
Interplay of What With the Who
Both the extant literature on persuasion (e.g., Murphy & Alexander, 2004; Murphy & Mason, 2006) and Alexander et al.’s (2011) reconceptualization of higherorder thinking highlight the complex interplay between the what and the who dimensions of text processing. Certain text characteristics, such as the credibility and unbiased nature of the source, comprehensibility of the text, and the perceived use of the text, influence learners’ processing of a text (e.g., Murphy, Holleran, Long, & Zeruth, 2005; Murphy, Long, Holleran, & Esterly, 2003; Murphy & Mason, 2006). One example of the interweaving of the learner and text is evidenced by readers evaluating emotional appeals as more persuasive than rational appeals when they have relatively low or moderate knowledge on a topic (Murphy, 2001). Also, readers with limited topic knowledge rely more heavily on either the credibility of the author or on the emotionality of the appeal rather than on the rationality of the claim (Murphy, 2001; Murphy et al., 2003). In contrast, individuals relatively high in knowledge and beliefs are much more likely to provide factual, rational evidence when examining an argument, whereas those with a lower level of knowledge and beliefs are more likely to consider affective experiences when examining arguments (Murphy, 2001; Murphy et al., 2003).
Argument Tasks: A Vehicle for Assessing Higherorder Thinking
As suggested by Brookhart (2010), it is imperative that higherorder thinking is assessed using tasks, activities, or measures aligned with the central tenets of the construct of interest. In the present study, our construct of interest was higherorder thinking about and with the text. As such, we identified student-generated argumentation as the type of higherorder thinking skill of central interest (Schraw et al., 2011). To be clear, we are not conflating higherorder thinking with argumentation, implying that they are two synonymous terms. Instead, we are operationalizing argumentation as a vehicle for assessing higherorder thinking. We adopted Kuhn’s (1991) conceptualization of argumentation skills as the ability to formulate and weigh arguments for and against a course of action, a point of view, or a solution to a problem. Argumentation skills include the skill to generate reasons, offer evidence, and provide counterarguments and rebuttals. In an argument, reasons provide an explanation for the position that is held (i.e., reasons expound upon the claim by including information about why a person holds a given position), while evidence bolsters the argument by including a verifiable justification for the argument (i.e., evidence augments the argument by including information about how or what a person knows). As such, we selected an argument task to assess student argumentation because it provides an opportunity for students to exhibit evidence of their higherorder processing (e.g., analysis, integration, or evaluation) of the opposing viewpoints within the intratextual persuasive message.
Researchers who study argumentative reading and writing have used a variety of argument-based tasks and practices to gauge argumentation (e.g., reflective essays, collaborative reasoning, computer-supported collaborative argumentation, dialogic student–teacher and student–peer interactions), which are synthesized by Newell et al. (2011) in their review of literature on teaching and learning argumentative reading and writing. The use of some form of written argument task as a mechanism for gauging higherorder thinking and critical reasoning has been well substantiated in the literature (e.g., Jonassen & Kim, 2010; Nussbaum, 2005, 2008; Nussbaum & Schraw, 2007; Reznitskaya, Anderson, & Kuo, 2007). Given that we aim to examine the effects of the intratextual persuasive message on higherorder thinking assessed through an argument task, we draw on recent findings in the argumentation literature, particularly in relation to the effects of task instructions and adjunct aids (i.e., visual representations).
Based on the previously reviewed findings and our goal of creating a higherorder thinking assessment, we decided that it was important to provide the necessary argument supports (e.g., explicit directions), while limiting the extent to which our task actually promoted higherorder thinking (Jonassen & Kim, 2010; Nussbaum, 2005; Nussbaum & Kardash, 2005; Nussbaum & Schraw, 2007; Scheuer, Loll, Pinkwart, & McLaren, 2010). That is, we wanted to provide students with explicit directions and a minimal argument outline or structure so that they would be able to produce arguments. At the same time, we opted not to provide a graphic organizer or visual representation tool so as to limit the influence of the task on students’ higherorder thinking (Nussbaum, 2008). Finally, we gave the argument task both prior to and after reading so that we could gauge changes in students’ argumentation, thereby minimizing the effects of the task structure, while also accounting for differences in participants’ prior knowledge.
Purpose
In our previous research, we explored the extent to which readers altered their positions on an issue after reading an intratextual persuasive message (Andiliou et al., 2012). Specifically, we examined how undergraduate students’ positions on a controversial issue changed differentially based on their knowledge and beliefs before and after reading an intratextual persuasive message. The present study extends these findings by examining the degree to which intratextual persuasive messages serve as a catalyst for higherorder thinking as evidenced by students’ self-generated arguments on a compelling issue before and after reading. To address this purpose, we posed the following three research questions:
Method
Participants
Undergraduate students (n = 261) enrolled in educational psychology courses at a mid-Atlantic university in the United States participated in the study. They were compensated with extra credit for their participation. The gender distribution of the participants was 81% female and 19% male. There was a predominance of students in their first year (63%) or second year (27%) of study. The majority of the students were Caucasian (92%) with others reporting as Hispanic (2%), African American (2%), Asian (1%), and Other (3%). The average age of the participants was 19 years, and over half of the participants were education majors. The self-reported grade point average was 3.39 (n = 239) on a 4.0 scale.
Materials
Intratextual persuasive message
An editorial commentary section from a local newspaper was distributed in its authentic form to the study participants. The editorial was a single message comprised of two text-segments, written by two authors with opposing views on the following overarching question: Is shunning products from China’s pollution-belching factories the best way Americans can reduce global warming? The question was displayed at the top of the page, with the No Boycott text-segment and the Yes Boycott text-segment below the question. Brief bylines were provided for both authors, and information pertaining to their expertise was comparable (i.e., a research fellow and a contributing writer).
The No Boycott text-segment (i.e., 19 paragraphs, 571 words) supported the position that boycotting is not an effective approach for engaging China on environmental issues, whereas the Yes Boycott text-segment (i.e., 15 paragraphs, 618 words) presented the position that boycotting has the potential to encourage China to reduce its emissions. Presented together, the opposing text-segments comprising the intratextual persuasive message were one page in length and each included two cartoon illustrations. The author of the No Boycott text-segment presented only arguments in support of his position and no counterarguments, whereas the author of the Yes Boycott text-segment forwarded two noncausal counterarguments.
Establishing text-segment equivalency
Given that the overall message (i.e., overarching question and both opposing text-segments) was used unmodified as it occurred in the newspaper editorial section, we conducted two preliminary studies 1 to establish the equivalency of the two opposing text-segments, which constituted the intratextual persuasive message (Andiliou et al., 2012). First, in the pilot study, 26 master’s and senior-level undergraduate students from an educational psychology class at a large mid-Atlantic university volunteered to participate in exchange for extra course credit. Participants were asked to read either the No or Yes Boycott text-segments to compare the argument structure of the constituting texts. The results of the pilot study indicated that students who only read the Yes Boycott text-segment failed to identify either of the counterarguments put forward by the author in an argument verification task. Instead, the majority of students judged the statements as being in favor of boycotting. Based on the results of the pilot study, we inferred that the two text-segments were perceived as one-sided texts, and they were used in an unmodified form.
Second, a materials study followed the pilot study, and it was conducted to examine the relative equivalency, persuasiveness, and credibility of the two one-sided persuasive texts. In all, 251 undergraduate students who enrolled in educational psychology (98%) or statistics (2%) courses at a large mid-Atlantic university participated in this preliminary study for extra course credit. As in the first pilot study, we asked participants to read one of the two one-sided texts. The results indicated that both one-sided texts had a similar effect on students’ perceived knowledge and beliefs; participants also rated the credibility of the two one-sided texts similarly. Regardless of text condition, the text had a significant effect on students’ topic beliefs but not on their perceived knowledge about the issue. In fact, at postreading, participants reported stronger beliefs in alignment with the position advocated by the author whose text they read. Moreover, the fact that participants reacted similarly to the features of the text regardless of which text they read further supports the equivalence of the two texts. It is important to note that the purpose of these preliminary studies was to assess the perception of each text independent of the other; thus, the one-sided texts were not tested against each other in either the pilot study or the materials study, and in both studies, participants only read and reported on one of the two texts. The details of these preliminary studies are explained in greater detail in a former article by the authors (see Note 1; Andiliou et al., 2012).
In the sections that follow, we describe the tasks and measures that were administered in the present study and the coding and scoring procedures that we employed to assess the student-generated arguments.
Tasks
Position task
Both before and after participants had read the commentary, they responded to a position task designed to gauge their ability to argue their position on the issue of boycotting Chinese-made goods. Students were directed to take a position on the issue by circling either Yes or No in response to the overarching question: Should Americans boycott Chinese products as a way to encourage China to reduce its emissions?
Argument task
Once the students committed to a position on the issue (i.e., responding to the position task), both before and after they had read the commentary, they were asked to imagine that they were writing an editorial in the newspaper regarding their position on boycotting Chinese-made goods. Instead of writing the editorial, participants were asked to outline the information that would appear in the editorial. As in previous studies of student argumentation, we provided students with a minimal structure support (i.e., external representation) to facilitate their argumentation. We followed Nussbaum and Schraw’s (2007) approach in which an argument was broken into its basic elements to serve as an adjunct aid for students. Specifically, we asked respondents to fill in a structured outline of the arguments they would present in their editorial to support their position on the issue of boycotting Chinese-made goods (see Figure 1 in the supplementary online appendix).
The outline was presented in a way that prompted students to develop four main components of an argument, including: (a) a reason to justify the position, (b) evidence to support the reason, (c) reasons why others might disagree (i.e., counterarguments), and (d) reasons why others are wrong (i.e., rebuttal). In short, we provided necessary, but not sufficient, conditions for students to craft arguments, as we did not provide explicit instructions (i.e., teaching) on the characteristics of an effective argument. As such, changes in performance on the task would be reflective of the viability of an intratextual persuasive message as a catalyst for higherorder thinking.
Argument coding
Responses to the argument task were coded for quantity, quality, and content to determine changes in students’ argumentation from prereading to postreading.
Quantity
Each student was assigned a quantity score for (a) each of the four individual argument elements (i.e., reasons, evidence, counterarguments, and rebuttals) and for (b) the total sum number of argument elements generated. Participants were awarded 1 point for an accurate response (i.e., factually correct or understandable), 0.5 points for a partially accurate response, and 0 points when no response was provided for a given argument element. For example, the student-generated argument element, “Boycotts do not always work,” was coded as accurate. In contrast, the student-generated argument element, “It hinders us!” was coded as partially correct, because the reason was unclear as to either what “it” or “us” referred to. For each argument element (e.g., reason), the total number of points awarded for that respective element type across all of the arguments generated by the respondent were summed. For example, if a respondent generated three arguments that each contained an accurate reason, then each reason element would be awarded 1 point and the respondents’ composite score for reason would be a 3; this process was also used to calculate evidence, counterarguments, and rebuttals. The total number of all argument elements was summed for students’ overall argument score.
In some cases, participants provided multiple reasons for a given reason prompt. For a response to be coded as two or more separate reasons, there needed to be distinct clauses that individually offered a unique reason to support the participant’s position (e.g., “If we boycott their goods, [1] that brings our economy down and [2] our way of living would change”). The interrater agreement for the quantity coding was .94, with all disagreements resolved through discussion.
The above scoring method was used for all participants on both pretest and posttest arguments. The scores at pretest were assigned as described; however, the procedure for calculating posttest composite scores was modified slightly. Specifically, the posttest composite score for each argument element represented the sum of the total number of unique element points awarded across pretest and posttest. For example, if a given student provided three reasons at pretest and three additional unique reasons at posttest, then their posttest composite reason score would be 6. By comparison, if a student provided the same three reasons at pretest and posttest, then that respondents’ posttest composite reason score would be 3. Overall argument composite score also took into consideration the likelihood of duplicated arguments at post, and the method for calculating total argument score summed the total number of all argument elements at pre and all additional unique argument elements at post.
This method of calculating posttest scores enabled us to account for the total number of additional reasons the respondents generated from the task. It also allowed us to differentiate between participants who gained novel arguments in the postreading task from participants who simply repeated the same arguments from prereading. Without accounting for, and eliminating, duplicate responses in posttest scores, using a simple cumulative quantity count at posttest would not differentiate between the two types of response patterns above. Rather than just comparing total responses at pretest to total responses at posttest, additional rationale behind calculating a posttest score that removed duplicated argument elements recognized that it was unlikely that students forgot arguments posed prereading. Instead, it was highly likely that they gained new insights with which to form their postreading arguments in reading the intratextual persuasive message.
Quality
To evaluate the quality of the arguments that students generated, we first developed a scheme to assess the viability of the argument elements. Specifically, for a high-quality argument, the reason had to support the position adopted by the participant. Evidence had to be quantifiable, verifiable, or factual, and it had to directly support the reason(s). Furthermore, it was necessary that the counterargument actually countered the participants’ reason, and the rebuttal directly addressed the counterargument. For example, one participant took a No Boycott position and proposed the “Embargo on Cuba” as evidence to justify the reason that “Boycotts do not always work.” Because the evidence was factual and verifiable (i.e., appropriate) and it directly addressed the reason (i.e., relevant), we counted this piece of evidence as viable.
For the quality scoring of the arguments, two raters coded 10% of the responses to a criterion agreement level above 80%. The remainder of the responses were scored independently with agreement checks for each additional 10% of the remaining codings. The goal was to improve agreement as the coding proceeded. The final interrater agreement for the quality of arguments was 82%.
Before and after reading, the quality score for each argument was a proportion, calculated by dividing the number of viable argument elements by the total number of argument elements (i.e., reasons, evidence, counterarguments, and rebuttals) and averaged across the number of arguments that a participant generated. Similar scoring approaches have been used in the extant literature on argumentation (e.g., Ferretti, MacArthur, & Dowdy, 2000; Inch & Warnick, 2002; Kuhn, 1991; Nussbaum & Kardash, 2005; Page-Voth & Graham, 1999).
It is important to note that quality and quantity are independent methods for coding different constructs. When coding quantity, duplicated responses at posttest were removed from the composite score so as not to artificially inflate quantity for respondents who reported essentially the same argument at pre- and posttest. However, quality was coded for all argument elements, irrespective of whether the response was duplicate or unique at posttest. The decision to not discriminate between unique and duplicated responses for the quality code was intentional, because regardless of whether the students reported a unique or duplicated response at posttest, the quality of that argument may have increased, decreased, or remained the same.
Content
Two raters coded the content of the reasons through an iterative keyword and content analysis coding procedure. Six major content categories emerged that captured the content of the reasons. The categories were the following: economic, effectiveness, environmental, health, moral, and political. Two other categories captured nonvalid responses: (a) a duplicate category was created to capture reasons forwarded by participants that actually duplicated another reason they had provided, and (b) a nonsensical category included reasons that were not meaningful.
Descriptors and exemplars for each category were generated so as to make the coding more systematic. For example, responses for the environmental category included keywords such as global warming, greenhouse gas emissions, and “going green.” Two raters independently coded students’ prereading and postreading responses using the coding categories and met once a week for 7 weeks to resolve disagreements. Prior to discussion, the raters reached 87% agreement for category agreement, and the disagreements were resolved through discussion.
Measures
Perceived topic knowledge
Participants indicated their perceived knowledge about the topic by rating their knowledge of (a) the relation between economics and climate change, and (b) the relation of global warming and politics. Students provided ratings of their topic knowledge on a 10-point Likert-type scale ranging from Not Very Much (1) to Very Much (10), before and after reading the intratextual persuasive message. The sum of the participants’ ratings of the two statements comprised a composite score for pre- and postperceived knowledge, with a total possible score of 20. The internal consistency indices for the pre- and postreading perceived knowledge composite were acceptable (αpre = .79 and αpost = .80).
Topic beliefs
Students indicated their beliefs about the issue of boycotting Chinese-made goods by rating their level of agreement with two statements directly derived from the two constituent text-segments. Participants rated each statement on a 5-point Likert-type scale ranging from Strongly Disagree (1) to Strongly Agree (5). Each statement represented the primary claim of one of the authors. For example, the statement that represented the primary argument of the No Boycott text-segment was as follows: The best way for Americans to promote global climate change is to engage China on environmental issues. From the Yes Boycott text-segment, we derived the following statement: American trade limits are the best lever for pressuring China’s government to reduce greenhouse gas emissions.
A topic belief index was calculated by subtracting a participant’s score for the Yes Boycott belief statement from their score on the No Boycott belief statement. The topic belief index ranges from −4 to +4, with higher negative values indicating stronger beliefs against boycotting Chinese-made goods and higher positive values representing stronger beliefs in favor of boycotting.
Procedure
In a university classroom setting, undergraduate students were provided with a large envelope that included an implied consent form, prereading measures (i.e., perceived topic knowledge, topic beliefs, position task, and argument task), the newspaper commentary (i.e., intratextual persuasive message), a long-division intervening task, and postreading measures (i.e., perceived topic knowledge, topic beliefs, position task, and argument task). After the implied consent form was read aloud by the administrator, participants were instructed to read and respond to the materials in the order that they appeared and that they should not look back to previous materials during completion. Participants were instructed to read the materials carefully and were encouraged to use any technique, such as highlighting or underlining, which they would normally use when thoroughly reading a text. Participants were allowed to take as long as desired on any particular task, and length of time was not monitored. Although there was no time limit for any given task, overall participants spent approximately 40 min completing all tasks. Thirty-one counterbalanced conditions were designed to ensure that there were no effects based on the order of text-segment presentation or measure administration. All procedures associated with recruitment, materials, administration, and data use and storage were approved by the Office for Research Protections. Participants were treated in accordance with the ethical principles set forth by the American Psychological Association (2013).
Results and Discussion
The overarching purpose of the present study was to explore the extent to which intratextual persuasive messages serve as a catalyst for higherorder thinking among undergraduate students. To accomplish this purpose, we asked students to complete an argument task before and after reading an intratextual persuasive message. As mentioned previously, the intratextual persuasive message presented an overarching question followed by two text-segments written by two different authors forwarding contradictory positions to address the question. It was presumed that if the overall message promotes higherorder thinking, then such thinking will likely be manifested in increased quantity and quality of arguments, as well as in the content of the reasons that align more closely with the intratextual persuasive message.
Overall Changes From Prereading to Postreading
Our first research question addressed the extent to which students exhibited higherorder thinking as manifested in the quantity and quality of their arguments, as well as in the content of the reasons provided to support their position on the overarching questions (i.e., whether they support a boycott of Chinese-made goods). In our discussion, we first overview our results for quantity and quality and then turn to the results for content of the students’ reasons.
Quantity and quality
Means and standard deviations for the quantity of students’ overall arguments, argument elements (i.e., reasons, evidence, counterarguments, and rebuttals), and quality are provided in Table 1. Overall, after reading the intratextual message, there was a significant increase in the quantity of arguments and argument elements. To further explore these prereading to postreading changes, we submitted the data to an omnibus, repeated-measures MANOVA, with time as the within-subjects variable, and argument quantity, reasons, evidence, counterarguments, and rebuttals as the dependent variables. The results revealed a statistically significant main effect for time, F(4, 257) = 420.39, p < .0001, η2 = .87. There were statistically significant increases in students’ overall argument quantity, F(1, 260) = 1,638.76, p < .0001, η2 = .86; quantity of reasons, F(1, 260) = 1,348.59, p < .0001, η2 = .84; evidence, F(1, 260) = 1,227.93, p < .0001, η2 =.83; counterarguments, F(1, 260) = 344.40, p < .0001, η2 = .81; and rebuttals, F(1, 260) = 267.96, p < .0001, η2 = .82.
Quantity of Argument Components and Quality at Prereading and Postreading by Combined, Altered, and Maintained Position Groups.
Note. Combined n = 261; Altered n = 79; Maintained n = 182.
In contrast, students’ arguments were statistically significantly poorer in quality after reading the intratextual persuasive message, F(1, 260) = 4.80, p = .029, η2 = .02. These results suggest that the intratextual persuasive message did not appear to serve as a catalyst for higherorder thinking with respect to the quality of students’ arguments. In fact, the result was quite the opposite, that is, the quality of students’ arguments marginally decreased over time.
Content of reasons
To closely examine students’ reasons for the position they embraced relative to the central question of the intratextual persuasive message, we used an iterative keyword and content analysis (Krippendorff, 1980). As displayed in Table 2, this process resulted in six major categories of reasons students used to support their position on boycotting Chinese-made goods. Generally, some of the reason categories were employed more than others. For example, as shown in Table 2, at prereading, students most frequently cited reasons included economics (e.g., “We are dependent on Chinese products”), effectiveness (e.g., “This is the only way to get the message across”), environment (e.g., “Greenhouse gases are a major contributor to global warming”), or moral grounds (e.g., “Boycotting China will hurt the people more than reduce the emissions”). After reading the intratextual persuasive message, we observed descriptive changes in the categories of reasons most often used by students. For example, substantively fewer students supported their position using reasons related to economics (14% decrease) or the environment (10.4% decrease). Interestingly, economic-related and environment-related arguments were made independently by the authors of both text-segments within the persuasive message. In contrast, there were 18.4% more reasons related to the moral dimensions of boycotting Chinese-made goods at postreading than at prereading. More often than not, the moral grounds reasons cited by students after reading the message aligned with the position of the author of the No Boycott text-segment. An example of a moral reason posed at posttest was as follows: “America emits a significant amount of greenhouse gases—a boycott would highlight our hypocrisy.” Given that the moral grounds reason category was more directly aligned with the arguments in the No Boycott text-segment, we explored possible variations in reason content in relation to whether students modified their position on the central issue of the persuasive message.
Percentage of Prereading and Postreading Reason Content by Combined, Altered, and Maintained Position.
Changes in Argument Elements by Modification of Position
The second research question in this investigation explored the extent to which students’ higherorder thinking, as evidenced in changes in the quantity of arguments and argument elements, overall argument quality, and the content of reasons for their position from prereading to postreading, was moderated by changes in their position on the central issue of the persuasive message (i.e., boycotting Chinese-made goods).
Quantity and quality
Means and standard deviations for the quantity of students’ overall arguments, argument elements (i.e., reasons, evidence, counterarguments, and rebuttals), and quality by whether students altered (i.e., changed from one position at pre to the other position at post) or maintained (i.e., held the same position at pre and at post) their position are provided in Table 1. As was the case with prereading to postreading changes in argument quantity for the total sample, students exhibited increased argument quantity regardless of whether they chose to alter or maintain their position on boycotting. Generally, in both the altered and the maintained groups, students’ overall argument and argument element quantity increased after reading the intratextual persuasive message. Unlike the results for quantity, the decrease in postreading quality scores was only statistically significant for those who maintained their position.
To further explore these prereading to postreading changes, we conducted a repeated-measures MANOVA layered by position change group, with time as the within-subjects variable, and argument quantity, reasons, evidence, counterarguments, and rebuttals as dependent outcomes. As reflected in Table 1, students who altered their position from prereading to postreading exhibited statistically significant increases over time in the overall quantity of their arguments, F(1, 78) = 703.23, p < .0001, η2 = .90; quantity of reasons, F(1, 78) = 683.97, p < .0001, η2 = .90; evidence, F(1, 78) = 573.50, p < .0001, η2 = .88; counterarguments, F(1, 78) = 447.78, p < .0001, η2 = .85; and rebuttals, F(1, 78) = 414.00, p < .0001, η2 = .84. These differences mirrored outcomes for the overall group. Argument quantity over time was similar for those students in the Maintained position group. As noted in Table 1, students who maintained their position from prereading to postreading exhibited statistically significant increases in the overall quantity of their arguments, F(1, 181) = 1,007.40, p < .0001, η2 = .85; quantity of reasons, F(1, 181) = 800.66, p < .0001, η2 = .82; evidence, F(1, 181) = 731.06, p < .0001, η2 = .80; counterarguments, F(1, 181) = 668.38, p < .0001, η2 = .79; and rebuttals, F(1, 181) = 763.634, p < .0001, η2 = .81, again, mirroring outcomes for the overall group.
The overall quality of arguments significantly decreased from prereading to postreading for the Maintained position group, F(1, 181) = 4.93, p = .028, η2 = .03, but not for the Altered position group, F(1, 78) = 0.385, p = .537, η2 = .01. These results seem to suggest that students who altered their position engaged in relatively more higherorder processing as we have operationalized it here than did the students who maintained their position from prereading to postreading.
Content
We also descriptively explored the content of students’ reasons relative to whether they altered or maintained their position on boycotting. The percentages of students forwarding a particular category of reason by whether they altered or maintained their position from prereading to postreading are provided in Table 2.
The results by position change group are particularly more telling than the results for the overall sample. For example, the percentage of specific categories of reasons shifted dramatically for students who altered their position from prereading to postreading. In fact, substantive shifts were exhibited for five of the six major reason categories, including economic (14.4% decrease), effectiveness of boycotting (14% increase), environmental (30% decrease), moral grounds (29% increase), and political (5.8% increase). Clearly, those who altered their position felt that it was necessary to support their newly acquired position on the issue of boycotting with different, relevant, and valid reasons from the text, which is evidence of higherorder thinking.
By comparison, students in the Maintained position group showed only minimal shifts in the categories of reasons they offered to support their position. Specifically, approximately 14% of the reasons provided by students in the Maintained position group shifted from the economics category to the moral grounds category. We attribute this change to the fact that the Maintained position group is largely comprised of students maintaining a No Boycott position, and a central argument forwarded in the No Boycott text-segment centered on moral concerns (e.g., “A blanket boycott of Chinese goods would actually hurt the Chinese people more than the pollution regulation evading firms that have moved their manufacturing plans to China” or “. . . with the second largest greenhouse gas emitter—the United States—dragging its feet, China will not feel compelled to take action”). In essence, this change in the category of reasons indicates that readers, who maintained their negative position for a potential boycott of Chinese-made goods, may have searched the two text-segments for examples of familiar or relevant reasons and evidence to inform their written arguments without the intention to change their initial position (cf. Newell et al., 2011).
To further explore higherorder thinking as evidenced in differences in reason category change, we subdivided the Altered position group and the Maintained position group based on the nature of students’ position change from prereading to postreading. As can be seen in Table 3, this resulted in two groups who altered their initial position (i.e., No Boycott to Yes Boycott and Yes Boycott to No Boycott) and two groups who maintained their initial position (i.e., No Boycott to No Boycott and Yes Boycott to Yes Boycott). Across all position groups, the category of reasons shifted away from economics with percentage decreases ranging from a 28.3% decrease for the No-Yes group to a minimal decrease for the Yes-Yes group of only 4.9%. Another trend evidenced across all position groups is a substantive change in the percentage of reasons categorized as pertaining to moral grounds. Indeed, the Yes-No position group evidenced a 31% increase in moral reasons, indicating that the students incorporated the arguments of the No Boycott text-segment. Interestingly, however, the Yes-Yes position group exhibited a decrease in the number of reasons forwarded pertaining to moral grounds (5.1%) and instead increased the number of reasons regarding the effectiveness of boycotting (7%). The effectiveness argument echoed one of the lines of argument forwarded by the author of the Yes Boycott text-segment, so this change in reasoning seems to parallel the No-No group embracing the moral grounds reasoning from the No Boycott text-segment. Thus, even participants who maintained their position modified to some extent their argumentation by incorporating text-based reasons presented by the author who was a proponent of the same side (i.e., Yes or No Boycott).
Percentage of Prereading and Postreading Reason Content by Position Group.
One other trend that seems worthy of mention is that both groups within the Altered position group, regardless of their final position, substantively modified the nature of the reasons they provided across all categories (see Table 3). This trend lends further support that for some individuals, the intratextual persuasive message seemed to serve as a catalyst for qualitative changes in the content of the reasons that students provided. What differentiates the No-Yes group from the Yes-No group seems to be an emphasis on political ramifications by the former group, whereas the latter group was more concerned with the environment. Both groups were comparably concerned about the effectiveness of a boycott and the moral dimensions of a boycott. Interestingly, mirroring the authors of the text-segments, these students embraced the reason categories of effectiveness and moral grounds to a different degree. As a case in point, the Yes-No group felt a boycott would have limited effectiveness in promoting more environmentally friendly policies in China, whereas the No-Yes group suggested that a boycott was the only effective way to reduce environmental fallout. For example, one participant posed the reason that “[Greenhouse gases being emitted have a] harmful effect on environment,” at pretest to support a Yes position toward boycotting but altered to the No position after reading the text posing reasoning that “it simply will not work,” and, “[there are] other ways to go about this.” Thus, these students may have engaged more deeply with the intratextual message as they altered their position and embraced a new line of reasoning put forward by the author who aligned with the final position they adopted. They also incorporated additional reasons that were put forward by both authors.
Argument Quantity and Quality by Students’ Perceived Topic Knowledge and Beliefs
The final research question in this investigation concerns the extent to which students’ higherorder thinking, as evidenced in the overall argument and argument element quantity and quality, is differentially influenced by students’ perceived topic knowledge and beliefs. Previous research suggests that students with more moderate levels of perceived topic knowledge and beliefs are more likely to alter their position when reading a two-sided refutational text (Murphy & Alexander, 2004). This change usually aligns with the position forwarded by the author. In comparison, as students’ knowledge increases or their beliefs become more extreme, students are more able to resist change (Murphy & Alexander, 2004).
To explore this research question, we conducted a series of linear regressions with students’ prereading topic knowledge and beliefs as the predictor variables and changes in students’ overall argument quantity and quality as outcome variables. All results were nonsignificant, suggesting that prereading perceived knowledge and beliefs were not viable predictors of the extent to which students would alter the quantity and quality of their arguments after reading an intratextual persuasive message. It is likely that this lack of predictive value is attributable to the relatively small change in students’ quantity and quality scores from prereading to postreading.
Conclusions and Implications
Students frequently encounter novel and complex forms of text such as intratextual persuasive messages consisting of two or more text-segments with contrasting perspectives on a controversial issue. The present study examined the extent to which an intratextual persuasive message functions as a catalyst for improving students’ higherorder thinking as it is manifested through their argumentation. The results of this investigation suggest that the intratextual persuasive message fostered students’ higherorder thinking as it was manifested in an increase of the overall quantity of arguments and argument elements and in the content of students’ arguments. However, the intratextual message had limited effectiveness in improving the quality of the student-generated arguments for students who maintained their position. In fact, the overall quality of argumentation was somewhat weaker, on average, particularly for students who maintained their initial position. The intratextual persuasive message initiated argumentation as students were able to forward more arguments after reading, but it was not sufficient to promote high-quality “orchestrated” argumentation that is organized, elaborated, and supported by evidence or personal experience (Perloff, 2003). The argument task provided the structure for students’ written arguments by outlining the components of an argument to scaffold the development of their argument and organize its presentation. However, in the absence of instruction on the qualities of a good argument and exemplars of effective arguments, participants had to rely on their prior knowledge of argumentation or derive and transfer the argumentative schemata from reading the two text-segments. Based on the prereading argument quality score, participants’ argumentation skills were not well developed, probably due to limited prior experience with argumentation.
In our investigation, the intratextual message had a significant impact on the content of the reasons that students offered after reading. Overall, before reading, students primarily forwarded reasons associated with the economic consequences of a boycott and the potential effectiveness of an enforced boycott in promoting greener policies in China. However, after reading, students suggested more reasons centered on the moral dilemmas relevant to a potential boycott. Arguably, moral reasons represent a type of emotive argument, and readers generally find such emotive appeals more persuasive than rational appeals (Murphy, 2001). Moreover, the complexity of argumentation was differential based on students’ position. For participants who altered their position after reading, the change of position was accompanied by more extensive modification of their reasoning, which was evident in the shift of the reasons they forwarded at postreading. Respondents who altered their position suggested fewer economic and environmental reasons. Instead, they proposed more justifications based on the moral grounds relevant to boycotts and referred to the effectiveness and the political ramifications of boycotting. These findings suggest that, for students who altered their position, the intratextual message functioned as a catalyst for higherorder thinking, as those students were able to think critically about the type of reasons that would better support their new position and, consequently, modify substantially their reasoning.
The effectiveness of the intratextual message to promote higherorder thinking was limited for the majority of the participants who maintained their position on the controversial issue of boycotting Chinese-made goods and exhibited evidence of case-building. Buehl et al. (2001) have defined case-building as “the selective attention to arguments in a persuasive text that reinforce one’s previously held conceptions or beliefs about a particular topic or issue” (p. 275). Students who maintained their position embraced the main line of reasoning of the author that supported their initial and unmodified perspective by adopting the most central argument of the author to build a case for maintaining their position (Buehl et al., 2001).
Among students who maintained a No Boycott position, there was a shift to the moral reasons that reflected the author’s concerns about the impact of boycotts on the Chinese people rather than the multinational companies. Similarly, individuals who maintained a Yes Boycott position embraced the main line of reasoning of the Yes Boycott text-segment as there was an increase in the number of reasons associated with the effectiveness of boycotts in reducing greenhouse emissions with fewer economic and moral reasons. Thus, it is possible that participants who maintained their position might have engaged in “biased processing” (Chambliss, 1995, p. 804) by overlooking or downplaying reasons and evidence proposed by the author with a different perspective than their initial position. The statistically significant poorer quality of argumentation among students who maintained their previously held position represents possible evidence that these students did not engage fully with the intratextual persuasive message.
This finding echoes the results of our previous study in which we found that the intratextual persuasive message strengthened the preexisting topic beliefs of students who maintained their initial position on the issue of boycotts (Andiliou et al., 2012). It is possible that for the majority of the students who maintained their position, the intratextual persuasive message functioned similarly to a two-sided nonrefutational text, which research has indicated to have limited effectiveness in changing readers’ positions (Allen, 1991).
However, individuals who altered their position modified more substantively the content of the reasons they put forward to justify their position. Interestingly, individuals who embraced a final No Boycott position after reading centered their argument on moral reasons, which indicates that they incorporated the line of reasoning of the No Boycott text-segment, but they also cited more reasons associated with the effectiveness of the boycotts and the political ramifications of this approach presented in both text-segments. In contrast, individuals who changed to a final Yes Boycott position primarily forwarded reasons regarding the effectiveness of boycotts and the environmental damage, which reflect the main points raised in support of a potential boycott. They also cited more concerns about the moral dimensions of a boycott presented in the No Boycott text-segment after reading both segments. For students who altered their positions, our findings suggest that they engaged more deeply with the arguments offered by both perspectives, revised the reasons they provided to align with those put forward by the author whose position they adopted, and critically selected reasons that better justified their new position. In line with the findings of Murphy and Alexander (2004) and Nussbaum and Sinatra (2003), our results suggest that students who altered their position showed higher levels of conceptual engagement with the entire intratextual message, evidenced in the modification of the underlying reasoning that supported their newly adopted position. It is yet unclear the extent to which those who maintained their position also engaged in higherorder thinking.
Limitations and Directions for Future Research
Although the present study substantively expands what is known about higherorder thinking and text-based processing, several caveats must be acknowledged. One limitation of the study is the use of a single intratextual persuasive passage, which was considered to represent a novel and complex form of text that students now encounter more frequently in digital and paper sources. To draw more valid conclusions about the potential of intratextual persuasive passages to improve student argumentation, future research needs to replicate such experimental investigations of text effects on argumentation using other types of passages that have a similar structure in which two or more authors provide their perspective in response to an overarching controversial question.
In addition, an important limitation of this study is the use of college undergraduates as participants. Because participants were sampled from college classrooms, these findings may not generalize to younger students. However, it is important to note that these findings are informative, in that they reveal more about the argumentation capabilities of college students.
Finally, in this study we intentionally provided participants with an argument scaffold. Based on prior research, we believed this was the best approach to answer these particular research questions. However, future research should consider other methods for measuring higherorder thinking as it is manifested in argumentation, for example, written tasks (e.g., persuasive/argumentative/reflective essays, argument tasks, and argument diagrams), student discourse, and interviews.
Implications for Educational Practice
Several implications for educational practice can be gleaned from this study. First, the results of this study call for greater attention to college students’ reasoning skills as the quality of the argumentation was poor. College instructors need to set among their primary learning objectives not only the acquisition of core knowledge in a discipline but also to scaffold and develop students’ ability to integrate knowledge and ideas, and to use argumentation to reflect, research, and create knowledge in their discipline.
The higherorder thinking skill of argumentation facilitates learning, conceptual change, and problem solving (Jonassen & Kim, 2010), and thus, educational practitioners need to design, facilitate, and assess group and individual learning activities that provide students with opportunities to engage in argumentation and reflect on their ability to argue effectively. The findings of this study indicate that students who alter their position after reading an intratextual persuasive message substantively modify the nature of their reasons in alignment with the reasoning of the author whose position they adopted. Moreover, the quality of the argumentation did not improve; in fact, for students who maintained their position, the quality of argumentation was poorer after reading. Our study confirms the findings of others (Kuhn, Shaw, & Felton, 1997; Means & Voss, 1996; Reznitskaya et al., 2001), who pointed out the challenges that adolescents and young adults experience with tasks that require them to comprehend written arguments and generate their own arguments, that is, to exhibit higherorder thinking.
Based on these findings, we would also encourage educational practitioners to carefully weigh their expectations relative of a particular type or genre of text against how that type of text has performed in relevant empirical literature. As we suggested previously, students interact with the text, and those interactions have been documented as being mediated by the nature of the text and the characteristics of the learner. For good reason, an educator might assume that a persuasive text will promote higherorder thinking and encourage students to carefully weigh various perspectives and concomitant evidence. Our present study suggests that this assumption is not necessarily the case. In fact, complex texts that incorporate multiple perspectives seem to affect the number of arguments and the constituent argument elements that students can produce, but they do not improve the quality of argumentation. Simply put, these findings call on educators to do more than provide this type of text and assume that students will contrast and integrate within a text or across text-segments and will be able to engage in higherorder thinking while reading.
Indeed, educational practitioners who prioritize the development of higherorder thinking skills in their classrooms may need to facilitate that development more systematically. Some of the successful instructional strategies pertaining to teaching argumentation skills include explicit instruction targeting the nature of effective argumentation and modeling strategies for generating coherent arguments (Nussbaum & Kardash, 2005; Nussbaum & Schraw, 2007; Voss & Means, 1991; Wiley & Voss, 1999). Moreover, educators can provide students with visual representations such as outlines or graphic organizers to scaffold their argumentation or require them to construct argument diagrams to reflect on and monitor their argumentation process (Newell et al., 2011; Nussbaum, 2008). In addition, for students to be able to develop the tactics, skills, and strategies of argumentation, instructors may need to provide them with opportunities to engage in argument-based and inquiry-driven discourse (e.g., collaborative reasoning). The findings of our study indicate that relying solely on argumentative text minimally improves students’ argumentation. However, for practitioners who aim to develop effective argumentation, it is essential to integrate reading and writing argument instruction to facilitate transfer of argument schema from comprehension to writing and vice versa.
In conclusion, structuring learning activities around case studies, problem scenarios, or persuasive texts about ethical or controversial issues are authentic and meaningful ways to engage students in higherorder thinking that will require them to reflect on their beliefs and draw on their relevant topic knowledge to generate argumentation. Employing instructional strategies such as the aforementioned may better equip learners with the skills they need to more effectively navigate the complex texts they encounter.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Notes
Author Biographies
). She is a fellow of American Psychological Association (APA) and American Educational Research Association (AERA). She is a past vice president of Division C of AERA and received the Richard E. Snow Early Career Achievement award from APA. She is the incoming editor of Review of Educational Research and serves on several editorial boards. She has authored or coauthored numerous publications in such outlets as Journal of Educational Psychology, Contemporary Educational Psychology, and Educational Researcher.