Abstract
In this article the author addresses the history of reliability and validity in qualitative research as this method of inquiry has progressed through various paradigms. The importance of the concepts of reliability and validity in research findings is traced from the traditional era, where there was only a modest distinction between qualitative and quantitative researchers involving their definitions of research reliability and validity, through the current era, where some researchers question the need to be restricted in their research by attempting to control for or account for the reliability and validity of their research findings. The author rejects a strict need for reliability and validity as traditionally defined in quantitative research and outlines a less restrictive approach to ensuring reliability and validity in qualitative research.
Introduction
Denzin and Lincoln (1998) identified the present as the fifth moment for qualitative research. They predicted that “more action-, activist-oriented research is on the horizon, as are more social criticism and social critique. “The search for grand narratives will be replaced by more local, small-scale theories fitted to specific problems and specific situations” (p. 22). If Denzin and Lincoln are correct in predicting the future pathway of qualitative research, they also are correct in their assumption that again qualitative researchers will be burdened with redefining and justifying the reliability and validity of their research results.
Although Denzin and Lincoln (1998) did not elaborate on the specific nature of the social criticisms and critiques, it is suggested that they are concerned with allegations that qualitative research will be identified as undisciplined, sloppy research comprising subjective observations. Further, it will be conducted by individuals who have personal motives or are chasing the current popular theme. Finally, as few individuals outside the field of qualitative research make distinctions between the different methodological styles within the qualitative paradigm, the label of undisciplined, sloppy research might spread across the qualitative research field instead of remaining with those who embrace fifth moment practices. Lincoln and Guba (1985) identified similar criticisms from quantitative and qualitative positivist researchers as the constructivists and naturalists attempted to redefine reliability and validity.
In this paper I intended to discuss some of the threats to reliability and validity found in qualitative research, and the various measures employed in the past to reduce these threats. I will begin by tracing the history of qualitative research to acquaint the reader with four paradigms qualitative research has progressed through and how each has affected how reliability and validity in qualitative research have been defined and measured. Reviewing the history of qualitative research also is deemed necessary as the methodology and terminology used in various older paradigms to control for reliability and validity are still found in current research and literature. As Creswell and Miller (2000) noted,
Writing about validity in qualitative inquiry is challenging on many levels. Multiple perspectives about it flood the pages of books (e.g., Lincoln & Guba, 1985; Maxwell, 1996; Merriam, 1998; Schwandt, 1997) and articles and chapters (e.g. Altheide & Johnson, 1994; Lather, 1993; Maxwell, 1992). In these texts, readers are treated to a confusing array of terms for validity, including authenticity, goodness, verisimilitude, adequacy, trustworthiness, plausibility, validity, validation, and credibility. Various authors have constructed diverse typologies of validity (e.g., Maxwell's five types, 1992; Lather's four frames, 1993; and Schwandt's four positions, 1997). It is of little wonder that Donmoyer (1996), who wrote an editorial on validity in the Educational Researcher, commented on the diverse perspectives of validity by contrasting Miles and Huberman's (1994) “traditional conception of validity” with Lather's (1993) “ironic validity.” Novice researchers, in particular, can become increasingly perplexed in attempting to understand the notion of validity in qualitative inquiry. (p. 124)
The importance of ensuring the validity of qualitative research is found throughout the literature that espouses qualitative methods (Creswell & Miller, 2000; Denzin & Lincoln, 1998; Kvale, 1996; Lincoln & Guba, 1985). Although the majority of today's qualitative researchers agree about the need for truthfulness in their research, they do not all agree with their quantitative counterparts about the need for reliability and validity as defined in quantitative research (Wolcott, 2005). In reviewing Denzin and Lincoln's five moments of qualitative research, it is clear that during the traditional period positivist and postpositivist researchers did agree, and still do agree, about the definition and need for reliability and validity in research regardless of whether the method was qualitative, quantitative, or combined.
Qualitative research in the 20th century
Traditional and modernist eras
During the traditional period (1900–1945), qualitative research mirrored that of quantitative research regarding reliability and validity. Both paradigms were concerned with offering valid, reliable, and objective interpretations of their findings. Although the traditional period ended in 1945, the traditional qualitative researcher survived and is currently labeled as a positivist bound by internal and external validity; guided by the logical-deductive, scientific, or grounded theory; and still producing scientific reports (Denzin & Lincoln, 1998).
The modernist phase (1945–1970) moved qualitative research even closer to quantitative research as researchers experimented with different experimental designs encompassing internal and external validity as well as causal narratives and quasi-statistics. “This was the golden age or rigorous qualitative analysis, bracketed in sociology by Boy in White (Becker et al., 1961) at one end and The Discovery of Grounded Theory (Glaser & Strauss, 1967) at the other” (Denzin & Lincoln, 1998, p. 17).
As qualitative research moved through both the traditional and modernist phases, the researchers supporting these methods of inquiry shared the beliefs that qualitative research could be used both to predict future behavior and to identify causal variables. Traditionalist and modernist adhered to the ideology that reasoning is governed by basic physical laws and rules, which could be identified either deductively through the use of quantitative research methods or inductively using qualitative research methods. In addition, they believed that the source of genuine knowledge was empirical research and logical analysis; and similar to their quantitative counterparts of the time, they believed that any knowledge claim must be capable of verification (Schwandt, 2007).
Reliability in quantitative research has focused on the concept of consistency, which primarily concentrated on instrumentation and outcome (Shadish, Cook, & Campbell, 2002). Instrumentation issues primarily evolve around survey instrument reliable: Was it consistently and accurately (validity) capturing the variables it was designed to measure? Test/retests of instruments, pilot studies, factor analysis (designed to identify unitary constructs), and data reduction (Cronbach's alpha) are just a few of the methods usually reported in quantitative studies to suggest to the information consumer that the survey instrument was reliable. In addition to instrumentation reliability, outcome reliability is assessed. In assessing only outcome measures, if a program is found to have little or no impact on the outcome variable, the program often is rejected. The program rejection becomes a reliability issue as the treatment implementation was not assessed (process evaluation).
From the traditional and modernist perspectives, reliability in qualitative research also is synonymous with consistency. From these perspectives, qualitative research is considered reliable if the research findings can be replicated by another researcher. Schwandt (2007) noted, “Traditionally, social scientists assume that while not all repeatable and replicable observations and accounts are necessarily valid, all valid accounts are (at least in principle) replicable” (p. 262). To enhance reliability, traditional researchers precisely document their field notes and decision points permitting future researchers to replicate the study. This audit trail included how the researcher analyzed transcriptions, interrater checks on coding, categorization procedures, detailed information about decision making, and variations in observations (Schwandt, 2007).
The rigid standards for research reliability set forth by traditional and modern qualitative researchers were maintained when assessing the validity (truthfulness) of the research findings. Quantitative researchers use the concepts of internal validity, external validity, construct validity, and statistical conclusion validity to assess the truthfulness of their findings. Using surveys and inferential statistics, quantitative researchers sample a proportion of a population, with the findings from the sample being inferred as qualities of the entire population. Through the use of various experimental or quasi-experimental research designs to control for internal validity and construct validity, the research could be replicated using the same survey instrument or similar survey instruments using different locations, times, and groups to assess the external validity of the findings. As the quantitative researcher was not in direct contact with the respondent, researcher bias (reflexivity) and reactivity could be cautiously dismissed. For the traditional and modernist qualitative researchers, reflexivity and reactivity could not be dismissed because often the researcher is the data collection instrument; nor could other perceived threats to the validity of their research be dismissed (i.e., inferences from small samples, nonprobability sampling, single source data collection, etc.). To address plausible threats to validity (descriptive, interpretative, and theoretical), traditional and modernist researchers developed various checks and balances to enhance the validity of their findings (i.e., member checking, triangulation, collaboration, etc.). Through the incorporation of various reliability and validity checks, both qualitative and quantitative research findings could be assessed for reliability and validity. (Validity threats and a validity checklist, along with a description of various validity checks, are discussed later.)
Blurred genres
From 1970 to 1986 qualitative research passed through its third developmental stage, know as the blurred genres period. During this period “the golden age of the social sciences was over, and a new age of blurred, interpretive genres was upon us. The essay as an art form was replacing the scientific article” (Denzin & Lincoln, 1998, p. 19). Also during this period qualitative researchers as a whole were split about what was the meaning of reliability and validity and how these two concepts should guide and influence their research designs. Positivists and postpositivists remained with the scientific model, relying on internal and external validity, reliability, and objectivity based on the traditional definitions found in quantitative research. Constructivists and naturalists moved away from the strict scientific definitions of reliability and validity, arguing that by restricting qualitative research to these confining definitions, researchers were unable to accurately report all the data they had collected (Denzin & Lincoln, 1998; Wolcott, 2005).
Constructivists and naturalists replaced traditional internal validity checks, which focused on construct and accuracy of the measurement tool(s), treatment and selection of the sample, conceptualization and operationalization of variables, and the development of the research design, with credibility. Credibility, which also focuses on internal validity, emphasizes the truthfulness of what the researcher reports. The researcher was responsible for enforcing and explaining to the research consumer not only what was being observed but also why what was being observed would have occurred naturally. Unlike traditional qualitative researchers, who emphasized internal validity in the initial design, constructivists and naturalists emphasized internal validity both prior to and during the research. To ensure internal validity, the researchers employed numerous validity checks throughout the course of the research (Creswell & Miller, 2000; Denzin & Lincoln, 1998; Lincoln & Guba, 1985; Maxwell, 1996).
Constructivists and naturalists also found fault with external validity as traditionally defined. Qualitative traditionalists and quantitative researchers define external validity as
whether a relationship observed in a specific population, at a specific time, in a specific place would also be observed in other populations, at other times, in other places. External validity is concerned with generalizability from a relationship observed in one setting to the same relationship in other settings. Replication enhances external validity. (Maxfield & Babbie, 2001, p. 422)
In short, external validity checks were attempts to ensure that research results are transferable not only to the population the sample was extracted from but that the results also would hold true across various people, times, and settings. Constructivists and naturalists observed that external validity was better defined as transferability instead of generalizability. Further, transferability was best accomplished by providing a thick, rich description of the research findings and permitting research consumers to draw their own inferences about research transferability to different groups, circumstances, and events. In applying previous research to current situations, applicability proof lies with the person making the new application, not the original researcher (Creswell & Miller, 2000; Lincoln & Guba, 1985).
The debate questioning the quality and value of the constructivists and naturalists' research was intensified by their refusal to accept the traditional definitions of reliability and validity, as characterized by quantitative researchers and embraced by the positivist and postpositivist qualitative researchers. Reliability refers to whether a particular research technique will yield the same results if applied repeatedly to the same object (Babbie, 1997) or “that quality-of-measurement standard where the same data would have been collected each time in repeated observations of the same phenomenon” (Maxfield & Babbie, 2002, p. 426).
Often, when an example is given to clarify the meaning of reliability, bathroom scales are presented and shown to be reliable by always giving the same weight for an individual as he or she steps on the scales several times. It is then noted that although the scales might be reliable, they are not necessarily valid unless the scales are displaying the individual's correct weight. “The conventional definition of reliability are those of stability, consistency, and predictability” (Lincoln & Guba, 1985, p. 298). Replication of the stability, consistency, and predictability of a measure or instrument constitutes reliability. Constructivists and naturalists convincingly argue that this is an unrealistic standard for the social sciences as the environment always is changing and that dependability should be the standard. Lincoln and Guba (1985) noted,
But replicability depends, again, upon the assumption of naive realism. There must be something tangible and unchanging “out there” that can serve as a benchmark if the idea of replication is to make sense. If the thing “out there” is ephemeral and changing, noted instabilities cannot be simply charged off to the inquiry procedure; they are at least as much a function of what is being studied as the process of studying. (p. 299)
Constructivists and naturalists would argue that the bathroom scale example of reliability is too sterile an example to be an accurate illustration of the need to substitute dependability for reliability. Given the example of a sharpshooter, with an extremely accurate weapon, who continually places all of his or her shots in a 2.5 centimeter circle at 500 meters, it is posited that qualitative and quantitative researchers would agree that both the weapon and the shooter display reliability and validity. After repeated tests to verify the reliability and validity of the shooter and the weapon, the quantitative researchers send the instrument out into the field, confident of both its consistency and accuracy. Constructivists and naturalists also would field the instrument, but throughout the study they would monitor its consistency and accuracy. Through prolonged observation, the researchers would discover that on days when it was windy or raining, the instrument (shooter and weapon) appeared to lose reliability and thus validity. Further investigation would disclose that neither the shooter nor weapon had lost reliability or validity, but that the wind was pushing the bullet to the right or left, and the rain was forcing the bullet downward, both of which caused the bullet to miss its intended mark. Qualitative researchers could identify, describe, and compensate for these findings, thus strengthening the reliability and validity of their research findings.
Finally, objectivity, which is found in the traditional model, was replaced by what the constructivists and naturalists defined as confirmability (Creswell & Miller, 2000; Hoepfl, 1997; Lincoln & Guba, 1985). The traditional meaning of objectivity is that if a large number of people report experiencing the same thing, it is objective, and if only a single person experiences it, then it is subjective (Lincoln & Guba, 1985). In using the traditional definition of objectivity, it was virtually impossible for constructivist and naturalist researchers to claim objectivity. In substituting confirmability for objectivity, the researcher's integrity and character are no longer in question because the emphasis is placed on the collected data and whether the data are capable of being confirmed by the research consumer.
The constructivists' substitution of confirmability for objectivity, dependability for reliability, and generalizability for external validity, arguably, was less favorably received by traditional and modernist qualitative researchers than by quantitative researchers. Schwandt (2007) suggested that traditional and modernist qualitative researchers, similar to quantitative researchers, argue that without reliability, there can be no validity. Findings based on dependability could plausibly affect interpretation validity. Although constructivists argued that the validity of their research suffered from the strict technical standards set forth by quantitative research, traditional and modernist qualitative researchers argued that these standards could and should be maintained to ensure the accuracy of qualitative research findings.
Crisis of representation
In the mid-1980s researchers sought new models of truth and methods for qualitative research. The paradigm for qualitative research again had changed. Qualitative researchers began to focus on feminist, ethnical, Marxist, and cultural perspectives (Denzin & Lincoln, 1998). With these perspectives came the need to again redefine the meaning of reliability and validity in qualitative research. Research was being dispersed in various interpretive forms, with no consensus between the different paradigms within qualitative research (positivists, postpositivists, naturalists, constructivists, and the orientational inquirists) and the internal paradigms of the crisis of representation period (feminists, culturalists, Marxists, etc.) about the standard for reliability and validity.
Researchers operating from the orientational inquiry perspective, or what Creswell and Miller (2000) called the critical perspective, control for validity by reporting their findings and stating their biases.
As a challenge and critique of the modern state, the critical perspective holds that researchers should uncover hidden assumptions about how narrative accounts are constructed, read, and interpreted. What governs our perspective about narratives is our historical situatedness of inquiry, a situatedness based on social, political, cultural, economic, ethnic, and gender antecedents of the studied situations. The implication for validity of this perspective is that validity is called into question, its assumptions interrogated and challenged, and the researchers need to be reflexive and disclose what they bring to the narrative. (pp. 126–127)
Critical social science “rejects the idea of disinterested social science and emphasizes attending to the cultural and historical conditions on which the theorist's own intellectual activity depends” (Schwandt, 2007, p. 54). Schwandt has suggested that many critical social scientists minimize or even discount the concepts of reliability in qualitative studies, arguing that replication of fieldwork is not plausible by another investigator. Regarding validity, Schwandt noted,
One additional, even stronger objection to truth and validity comes from radical postmodernists who hold that every idea of truth as essential to knowledge or as a goal of science is a modernist, Enlightenment value associated with orders, rules, logic, rationality, and reason, all of which are considered suspect, at best, and oppressive, at worst. (p. 310)
From a critical social science perspective, the individual conducting the research is responsible to report what actions were taken throughout the research to ensure the research was valid. Further, the researcher must disclose any biases and other relevant factors he or she brought to the research that could have affected the findings. The final determinations about whether the findings are valid rely on the individual consumers of the research, based on the information provided by the researcher. The repudiation of the traditional concepts and even the expanded concepts of reliability and validity offered by constructivists makes critical social science research distinctly different from that of traditionalists, modernists, and constructivists.
Threats to reliability
The definition of reliability in qualitative research differs between positivists (traditionalists and modernists), constructionists, and the critical researchers, but there is concurrence in the need for trustworthiness, accuracy, and dependability of research findings. Wolcott (2005) noted, “It is difficult to escape the suggestion that if our work is not reliable, then it must be unreliable” (p. 158). Although Wolcott has identified reliability as irrelevant in field research as it distracts from the research findings by forcing the researcher to focus on the research process, many qualitative researchers do not share his beliefs about reliability (Creswell, 1994; Fetterman, 1989; Guba & Lincoln, 1981, 1989).
As the qualitative researcher often is perceived as the research instrument, he or she must ensure that the information he or she reports/records is accurate and not oversimplified or misinterpreted. If multiple observers are being used, they must report/record similar observations in the same manner (Guba & Lincoln, 1981, 1989). Creswell (1994) has suggested that qualitative researchers should identify their biases and values, report how the sample was selected, and report any preconceived assumptions they might have possessed to assist the reader in determining the reliability of the research and to enhance replication. Several other methods to enhance the reliability of qualitative research have been identified.
Reliability measures
As stated previously, reliability refers to whether a particular research technique will yield the same results if applied repeatedly to the same object (Babbie, 1997). Arguably, this definition for reliability is applied more easily to quantitative research than to qualitative research, based on differences in the model designs.
The use of standardized question sets and large sample populations, and the gap between the researcher and the respondent all serve to strengthen the reliability of quantitative research. Although these options seldom are available to the qualitative researcher, there are several different strategies that can be employed to increase the reliability of qualitative research.
Qualitative researchers can enhance reliability by ensuring research worker reliability, variations in observations, and the use of various data collection techniques such as the test-retest method and split-half method. These four methods of enhancing the reliability of qualitative research will be discussed.
Research worker reliability
If more than one researcher is working on the project, it is imperative that all be trained to record events, collect data, and conduct interviews in an identical manner. The research workers need to be familiar with the environment they will be working in as it could affect the responses and actions of those being observed. A White man making observations or interviewing minority, inner city residents would be less likely to get a true assessment of the situation than would a minority interviewer who grew up in the inner city and could relate more clearly with the respondents. Conversely, the researcher who grew up in the inner city might be more likely to miss discrepant data because of a preconceived notion about the environment.
In addressing research worker reliability, the following areas must be addressed (Kvale, 1996):
Analysis methods: Are interviews interpreted the same by different researchers?
Answer reliability: Did the researcher ask the same question in several ways?
Coder reliability: Are the interviewers asking the same thing in an unbiased manner?
Critical checking: Are all researchers asking critical questions to test the interviewee's story?
Follow-up questions: Are all researchers using follow-up questions to ensure the collection of thick, rich data?
Leading questions: Are interviewers avoiding leading questions that may solicit a desired response, but not necessarily an accurate response?
Transcription: Are interviews and observations being transcribed correctly and accurately?
Check-coding can be used to ensure that interviewers and researchers transcribing the interviews are doing so accurately. Check-coding can be completed by having interviewers separately code the same interviews and then have them come together to compare and discuss the results. Check-coding for transcribers can be completed by having them transcribe the same audio- or videotape and then conducting a document comparison on the computer to determine the difference in transcriptions. The two transcriptions can then be rectified through comparison. Adherence to these practices will enhance the reliability of the research (Kvale, 1996).
Variation in observations
To enhance the reliability of observations, researchers should vary the time and place that the observations occur. Changing the time and place of the observations will assist in establishing reliability by showing the same occurrence regardless of whether it is day or night, or winter or summer (Denzin & Lincoln, 1998). Changing the times and places that observations are made is similar to the test-retest method.
Test-retest method
As noted, a test-retest method can be used to ensure that the same observations are found by varying the times and places the observations are made. The test-retest method also can be used to ensure that previously received information gathered from a respondent was accurate. Once information has been collected from a respondent, asking him or her about it at later dates can reconfirm the accuracy of the data. For example, if a respondent had elaborated on a gang fight in which he or she had stated that two individuals were injured, at a later period you could interject that information into the conversation to see if he or she can accurately recall what was stated previously.
Split-half method
The split-half method is used to ensure reliability by soliciting several responses from a respondent to the same question asked in various manners. The responses to the various questions should be the same or very similar. A correctional educator might answer that he or she does not have any problems in his or her classroom that are caused by students displaying low self-control. Later, on a question addressing student responses, the correctional educator might relate that he or she hates it when the students just shout out the answer without being called on and that this is a regular occurrence. The two answers do not match and would warrant further inquiry and possibly classroom observation.
It is important to remember that having a reliable measure or increasing the reliability of a measure does not guarantee validity, credibility, or transferability. Similar to the procedures intended to increase the reliability of qualitative research, there also are proven methods to enhance the validity of a study (Creswell & Miller, 2000; Maxwell, 1996; Moilanen, 2000).
Qualitative research validity
Threats to validity
Maxwell (1996) identified five threats to the validity of qualitative research. These include how observations are described and interpreted, and how the data might be consciously or accidentally manipulated to fit a specific theory. He also noted that researcher bias (inherit reflexivity) and even the researcher's presence (reactivity) can affect what is observed. Maxwell offered several procedures that can be implemented to strengthen the validity of qualitative research. Other researchers also have endorsed the procedures recommended by Maxwell but with slight variations in how the procedure is applied or what a particular researcher named a procedure (Creswell & Miller, 2000; Kvale, 1996; Lincoln & Guba, 1985).
Descriptive validity
What an individual fails to record while collecting data often is as important as what is collected. Researchers, especially those new to qualitative research, must record interviews accurately and completely. The researcher must ensure that the words recorded are those of the individual being observed and not a shortened form written down by the observer. Tape and video recordings of interviews can help validate descriptive data but cannot eliminate all of the threats. The researcher is still responsible for accurately describing the physical setting of the location, something not possible for a transcriber listening to an audiotape. The researcher also must describe the environment and actions outside the lens of video recorders to ensure all the causes of “what happened” can be captured and analyzed. Maxwell (1996) noted that failure to accurately collect and interpret descriptive data will lead to invalid interpretations, which will result in invalid conclusions.
Interpretation validity
To accurately interpret what has transpired, the researcher must capture the observation as interpreted by the person being studied. “The main threat to valid interpretation is imposing one's own framework or meaning, rather than understanding the perspective of the people studied and the meanings they attach to their words and actions” (Maxwell, 1996, p. 89). To avoid compromising interpretation validity, researchers should use open-ended questions that permit the respondent to elaborate on answers. Questions should not be misleading or directional in an attempt to solicit any response other than the one the respondent would have naturally issued. Researchers should permit respondents to elaborate on information and attempt to understand the observation from the respondent's perspective.
Theory validity
At the onset of an inquiry, the researcher often has a particular theory or perspective that he or she believes the data will support. Researchers must ensure that they do not force the data to fit a certain theory, nor can they ignore data that does not fit the theory (discrepant data). Researchers must present all data even if it does not support their hypotheses.
Researcher bias
Every researcher will possess some sort of bias. The bias need not be racial, ethnical, gender related, or cultural. The bias could be simply supporting one theory over another or failing to interview certain types of offenders. Researchers must identify and highlight their biases to ensure they do not influence the research results. Researchers must explain in their proposal how they will deal with their individual biases to ensure they do not affect the conduct and conclusions of their research (Maxwell, 1996).
Reactivity
How much of what you are observing is caused by you being there? Researchers can affect both the setting and the individuals being observed. Eliminating reactivity is virtually impossible, but the researcher must be aware of it and how it affects what is being observed. Interviewees often are reacting to the interviewer and not the situation being observed (Maxwell, 1996). Interviewees might lie to interviewers to make themselves seem more important, less important, or tougher. Those being studied may stage events for the benefit of the researcher. Researchers must remain conscious of how their presence is affecting the setting and the individuals being observed, and how this could affect the research results.
Validity checklist (credibility)
Overview
To enhance the validity of qualitative research, researchers recommend the use of validity checklists (Creswell & Miller, 2000; Maxwell, 1996; Lincoln & Guba, 1985; Whittemore, Chase, & Mandle, 2001). A validity checklist assists the researcher in establishing techniques that will be used for the duration of the research to strengthen validity issues. Although validity is checked throughout the course of the research, techniques for demonstrating validity should be presented in the research proposal (Maxwell, 1996) and should address areas such as design consideration, data generation, analysis, and presentation (Whittemore et al., 2001). It would be virtually impossible to address all of the validity checks recommended to enhance qualitative research. The discussion of various validity checks will be restricted to those most often used by constructivists and naturalists.
Triangulation
The concept of triangulation involves locating an unknown point from two or more known points. The more known points that are used, the more likely the unknown location will be identified. In qualitative research, by using interviews, theory, previous research literature, personal observations, and other data, findings can be compared to determine the validity of a certain theme or category. In using a multitude of sources to explain an event, the findings become more valid than explaining an event from a single incident or observation (Creswell & Miller, 2000; Maxwell, 1996).
Negative cases, discrepant data, or disconfirming evidence
In using triangulation, researchers attempt to confirm certain themes and categories. After the themes or categories are determined, researchers must then search for data that would disprove the established themes or does not fit into one of the categories. Creswell and Miller (2000) noted,
In practice, the search for disconfirming evidence is a difficult process because researchers have the proclivity to find confirming rather than disconfirming evidence. Further, the disconfirming evidence should not outweigh the confirming evidence. As evidence for the validity of a narrative account, however, this search for disconfirming evidence provides further support of the account's credibility because reality, according to constructivists, is multiple and complex. (p. 127)
Supporting and discrepant data need to be rigorously examined to determine if the themes or categories support it. If the themes or categories cannot support the data, they need to be modified (Maxwell, 1996). Regardless if the themes and categories are modified (Maxwell) or the prevalent data are reported (Creswell & Miller, 2000), the researcher must use the research narrative to alert her or his audience of the discrepant data.
Bias or researcher reflexivity
The researcher must describe to the consumer of the research any assumptions, beliefs, values, or biases the researcher possesses that could have affected the study. He or she must then identify how these assumptions, beliefs, values, and biases were suspended or controlled for during the research (Creswell & Miller, 2000).
Member checking
“The member check, whereby data, analytic categories, interpretations, and conclusions are tested with members of those stake-holding groups from whom the data were originally collected, is the most crucial technique for establishing credibility” (Lincoln & Guba, 1985, p. 314). Member checking is a continuous process. It provides the respondent with both an immediate and continuous opportunity to correct errors and misinterpretations of what was stated or observed. It provides the respondent with the opportunity to volunteer additional information and to summarize information. Finally, it reinforces the data by having the participant confirm what was said and observed (Lincoln & Guba, 1985).
Prolonged engagement in the field
As discussed previously, the researcher can affect both the setting and the individuals being observed. Eliminating reactivity is virtually impossible, but the researcher must be aware of it and how it affects what is being observed. Through prolonged engagement in the field, the researcher becomes more a part of the environment and less of a novelty. In becoming part of the environment, the researcher enhances the opportunity to observe the environment and participants as they really are in daily life. The researcher learns the norms, language, and habits of those being studied and can better predict and interpret the meaning of events. The researcher also can build trust that can lead to identifying different sources for information and who has access to certain information, both of which would enhance the research and the triangulation of data.
Collaboration
The researcher should collaborate with the participants to build their view into the study. For research about low self-control in correctional classrooms, correctional educators might be extremely helpful in assisting the researcher in identifying questions that would solicit the appropriate response from other correctional educators. Correctional educators also could assist with data analysis, such as determining instructional style. Researchers with limited experience might be dependent on participants to assist in determining which questions are appropriate, but researchers also must use caution to ensure participant bias is not introduced into the research (Creswell & Miller, 2000).
Audit trail
The audit trail provides clear documentation of all research decisions and activities. The audit trail is established by reviewing memos, logs, journals, field notes, computer files, and any other data pertaining to the research. If available, an external auditor should be used to review the study to determine its trustworthiness. An auditor will review the details of the research and the decision-making process. Similar to a financial audit, the audit trail ensures there is no creative accounting occurring in the research. The audit will review record accuracy, justify log entries, and spot-check various entries to ensure they actually occurred. The audit trail is conducted in a rigorous manner as it often is the only item that will persuade quantitative researchers that the research is valid (Creswell & Miller, 2000; Maxwell, 1996; see also Lincoln & Guba, 1985, for an audit trail matrix).
Thick, rich description
The researcher must describe the research setting, the participants, and the themes in depth. Unlike quantitative research, the qualitative research cannot just report the facts. The qualitative researcher must present the entire picture, thus transporting the reader into the environment, setting, and situation. The researcher also must capture the reader's imagination by not only detailing the physical appearance of the participants, but also by capturing their emotions, feelings, and experiences (Creswell & Miller, 2000).
Feedback or peer debriefing
A peer familiar with the research or the phenomenon involved should review the data, decision matrixes, and other documentation to question the methods and interpretations. The peer must be removed from the study but familiar with the research. Peer review should be ongoing throughout the research to ensure credibility and avoid problems that would be difficult to correct at later points in the study. The feedback provided by the peer is intended to keep the researcher honest.
The peer reviewer should evaluate all areas of the research to include decision-making, methodology, legal and ethical issues, and other matters pertaining to the research. The peer reviewer also can be used by the researcher as a sounding board for new ideas and could present the chance for the researcher to clear her/his mind of questions about past and future decisions. The peer should be equivalent to the researcher in status as recommendations from a superior might be deemed more as orders or criticism, and input from subordinates could possibly be ignored (Lincoln & Guba, 1985).
Qualitative researchers have numerous methods available to enhance the evidence of reliability and validity. When used in combination, the methods identified to enhance the evidence of reliability and validity in qualitative research are extremely effective.
Keep in mind that these strategies are effective only if you actually use them. Simply invoking them as magical spells won't drive the validity threats away. Nor will putting them in your proposal as boilerplate convince most reviewers; you will need to demonstrate that you have thought through how you can effectively use them in your own study. (Maxwell, 1996, p. 92)
Conclusion
In conclusion, if conducted within the standards set forth for good qualitative research, using the validity standards set by positivists or constructivists and naturalists, qualitative research has been proven to be as truthful as quantitative research. The problem is that not all qualitative researchers believe it is necessary to adhere to these or any standards of reliability and validity. Wolcott (2005) noted that “we need to recognize the circumstances that render reliability essentially irrelevant as a central concern in field work” (p. 159). He argued further that validity is not a relevant criteria measure in qualitative research as qualitative research offers an illustration of what occurred.
It is too early to predict what impact, if any, the fifth moment will have on qualitative research, although it appears that it will be very similar to that of the orientional inquiry. Research using orientational inquiry (feminist, ethnic, Marxist, cultural) often is conducted from a postmodernism position. “This position doubts all criteria and privileges none, although those who work within it favor criteria like those adopted by some poststructuralists… . Such criteria would flow from the qualitative project, stressing subjectivity, emotionality, feeling, and other antifoundational factors” (Denzin & Lincoln, 1998, p. 277).
The fear is that again all qualitative researchers will be on the defensive about the truthfulness of their research, especially if the research appears subjective and openly discounts the need for reliability and validity. The future support for qualitative research might depend on how the postmodernist position handles issues of reliability and validity since qualitative researchers seems to be grouped as a whole regardless of which paradigm one supports.