Abstract
The author of Interrogating Ethnography explores the ways social scientists have responded to his critiques and offers practices he believes might strengthen the web of evidence.
Most scholarly disciplines expect and invite fact-checking (including replication and falsification), but attempts at outside verification are infrequent in ethnography, as I learned writing Interrogating Ethnography: Why Evidence Matters. Many ethnographers take care to authenticate their work—including the use of documentation alongside video and audio recording—but reinvestigation by third parties is rare and sometimes unwelcome.
Many of the responses to Interrogating Ethnography have been favorable, for which I am grateful; some have been mixed, which is always expected; and a few have been bafflingly resistant to the idea of fact-checking. This leads to a curious phenomenon in which ethnographers’ factual assertions are vigorously defended, even if those claims are seriously questionable or demonstrably wrong. Writing as an ethnography enthusiast, my goal here is to explore this phenomenon, and to offer some hopefully useful correctives.
The premise of Interrogating Ethnography is that accuracy could be improved by adopting more rigorous approaches to the evaluation of facts. In the book, I urged ethnographers to distinguish more carefully between observed or verifiable events and the accounts of their research participants, which may include impressions, inferences, and beliefs. I advised greater skepticism of informants’ stories, increased use of documentary sources, and less reliance on generalizations and composites. As an organizing principle, I suggested that some concepts in the law of evidence could be helpful by analogy, especially the hearsay rule’s distinction—often blurred in ethnography—between statements offered for their truth value and those establishing the declarant’s “state of mind.” In other words, ethnographers should be careful about distinguishing between “what I saw” (perception) and “what I was told” (hearsay).
To state the legal standard simply— offered only as a starting point—hearsay evidence is always admissible as proof of a declarant’s opinions, motivation, or beliefs. Second-hand accounts are considered less dependable, however, to prove the occurrence of the described events. Thus, although I have never proposed that ethnographers adhere to strict legal rules, I did suggest that they should be more wary of vouching for their sources’ stories.
The practice of institutional skepticism, or adversary testing, is familiar to scientists, journalists, historians, and lawyers, but it appears relatively uncommon among ethnographers, who seem overly willing to accept one another’s truth claims. Nonetheless, some ethnographic narratives raise questions that can be tested against documentary or circumstantial evidence.
Attorneys and other fact-seeking professionals look to such documentary and circumstantial evidence for three reasons: First, documents don’t tend to change. Unlike the accounts of witnesses or research participants, documents are not subject to the vagaries of recollection or wishful thinking. Second, documents are usually (though not always) created before there is an issue in controversy, meaning that they are likely to be relatively objective. Although documents are not always accurate or unbiased, they do have certain advantages over human memory and should be consulted if available. Despite its bad reputation among lay people, circumstantial evidence can actually provide a neutral context for evaluating certain claims. To take a simple example, the presence of ice on a pond provides valuable circumstantial evidence of a below-freezing temperature, and can be more reliable than the impressions of a witness, which might be influenced by confounding factors such as wind or clothing.
The best ethnographers establish a web of evidence, supplementing first-hand observations and the accounts of research subjects with documentation and circumstantial evidence.
Documentation
Documentary evidence can raise (and sometimes answer) questions about events that anchor even prominent ethnographies. In Off the Books, Sudhir Venkatesh tells the story of “Babycake Jackson,” murdered while sleeping in an abandoned building because he refused to act as a lookout for a Chicago drug dealer. According to Venkatesh, Babycake was shot 22 times, and the killing occurred in the “Maquis Park” neighborhood (which he reveals as the area around 47th Street and King Drive) in February 2002. Venkatesh attributes this information to Babycake’s brother “Bill,” who is said to have been at the scene. Venkatesh does not claim to have observed the murder or seen the body, and it is not clear whether he spoke directly to Bill or got the story from an intermediary.
The best ethnographers establish a web of evidence, supplementing first-hand observations with documentation and circumstantial evidence.
It turns out, however, that there was no such murder in Chicago in February 2002. The Chicago police often fail to solve crimes, but they do a good job of keeping track of dead bodies, and the official homicide records include only one killing that month in an abandoned building—the strangulation of a woman several miles from Maquis Park. There are several possible explanations for the discrepancy. Venkatesh may have jumbled the facts to preserve anonymity, although he makes no such disclosure in Off the Books. Or he might have been misinformed by his source, which, is the central problem with relying on hearsay. But in any case, it appears that no fellow ethnographer, reviewer, or editor ever thought to check Venkatesh’s story against public records.
There are at least seven discrete facts in the Babycake narrative: the fact of the murder, its date, its location, its reasons, its perpetrators, the manner of death, and the number of shots. Any or all of these could have been exaggerated, misrepresented, or invented by Venkatesh’s informant. Consider the claim that Babycake was shot 22 times. If Bill was indeed an eyewitness, he would have been far too traumatized to keep track of the shots—as we know from eyewitness accounts of other shootings—which could have been fired in under a minute. Accurate information about the number of rounds could only be provided by the shooter (who was not interviewed by Venkatesh) or the medical examiner (who has no records of any such killing). We might therefore call this a faux detail. Although it provides the appearance of certainty, it has no credible source.
Venkatesh uses the grim particulars of the Babycake story to illustrate the impact of the murder on the Maquis Park community, explaining that neighborhood people feared that the violence might spiral out of control. But that point could have been made more simply, just by saying that a small-time lookout was murdered when he stopped helping dealers—which is the most that Venkatesh could reliably know about the incident. Instead, Venkatesh opts for dramatization, enhancing the account with details that range from invented (the location), to unsubstantiated (22 shots), to hearsay (everything else). The elaboration adds flair at the cost of facticity. The sensationalism of the narrative makes it more engaging, but only if we choose not to question its accuracy.
When a reporter finally did question Venkatesh (in an article about Interrogating Ethnography), he stood by every aspect of his story. “The events and incidents described in the book are real,” he said, without addressing the dependability of the hearsay on which he evidently relied.
The Venkatesh vignette is hardly unique. Similar accounts can be found in other ethnographies, in which faux details— based on hearsay or composites—cannot readily be distinguished from real, verifiable ones. Nor is Venkatesh alone in discounting rigorous documentation. Writing in Social Science Space, the British sociologist Robert Dingwall criticized my reference to documents, as opposed to informants’ statements, as “invalidating the best method by reference to the second-best method.”
Circumstantial Evidence
As we have seen, documentation can directly controvert the stated facts in an ethnography. Circumstantial evidence plays a different role, raising questions about reliability without necessarily resolving them. In $2 a Day, Kathryn Edin and Luke Shaefer tell the story of a group of poor middle school students from the Mississippi Delta who took a 2007 class trip to Washington, D.C. Although the children traveled by airplane, Edin and Shaefer tell readers that some had never even heard of elevators: “[S]ome of them didn’t believe that the box behind the doors could actually transport them from one floor to another. They honestly thought it was some sort of joke that the teachers were playing on them.”
I found it difficult to believe that “some” American adolescents were completely unaware of the existence of elevators in the 21st century. Because Edin and Shaefer had not been on the trip, it was impossible for me to fact-check the unnamed informants for their second- or third-hand anecdote. As an exercise in exploring circumstantial evidence, I therefore consulted a group of Mississippi educators, none of whom believed the story. In addition, I looked at a Mississippi school curriculum and a list of popular television shows, both of which confirmed the virtual unavoidability of elevators in young people’s consciousness.
It turned out that other ethnographers were more accepting of the story, to the point of refashioning it to heighten its plausibility. Peter Moskos said in one symposium that he found it “entirely believable” because “a lot of the people that I know in my ethnography in Baltimore have never been in an elevator after they were born and left the hospital.”
But Edin and Shaefer did not say merely that the children had never been in an elevator. They wrote that the students could not even believe such things existed in the age of television, airplanes, and space travel. Moskos allowed that the students must have seen elevators on television, but added “there are lots of things on TV you’re not supposed to believe”—thus suggesting that Mississippi sixth-graders would put elevators in the same category as extra-terrestrials, witches, and zombies. Were these kids really shocked to discover that they could get to the second floor of a building in an elevator? Or did Edin and Shaefer’s unnamed informant exaggerate or misunderstand the students’ excited reaction to their first elevator ride?
Edin and Shaefer’s $2 a Day is a closely documented account of extreme poverty in America, in which the elevator story is but one paragraph. My exploration of this narrative is not a critique of this excellent book, but rather a demonstration of how circumstantial evidence can be brought to bear on questionable claims. A rigorous review of the story as written—rather than as reimagined—would at least weigh the circumstantial evidence before coming to a conclusion about the experiences and reactions of Mississippi teenagers.
Eyewitnesses and Their Uncertainties
Ethnographers often express a preference for eyewitness accounts, explaining that they provide a voice for marginalized or discredited populations and allow people to describe their own lives and to speak for themselves. These concerns are valid and important, yet they risk eliminating the distance between researcher and source, thus undermining the much-needed critical evaluation of a subject’s story. When the accuracy or reliability of sources is questioned, a common response is something like “they were there and you were not,” as though to say that eyewitnesses cannot be mistaken, inaccurate, or deceptive—a point few sociologists would seem willing to concede if pressed.
Journalists recognize that all sources are potentially misleading, often for personal reasons that may not be immediately obvious to reporters or researchers. In the New York Times, Hannah Beech wrote of interviewing Rohingya refugees at a camp in Bangladesh. She gathered “a notebook filled with the kind of quotes that pull at heartstrings,” only to learn later that “little of it was true.” The people in the camp had suffered greatly at the hands of Myanmar’s security forces, and yet they repeatedly provided Beech with “false narratives” and “embellished tales” that seemed calculated to increase their access to relief supplies.
Beech was sympathetic to her Rohingya informants: “Such strategies are a natural survival tactic,” she wrote. “Who wouldn’t do the same to feed a family?” Nonetheless, she felt professionally obliged to employ her “reporter’s necessary skepticism” to search out inconsistencies and get the story right. The resulting article was aptly titled, “The Rohingya Suffer Real Horrors. So Why Are Some of Their Stories Untrue?”
Lawyers and journalists are not the only professionals who test eyewitness accounts against disconfirming and circumstantial evidence. Historians routinely rely on surrounding context to reconstruct past events, as I learned in my earlier work on antebellum abolitionists.
One of the troubling issues in the historiography of abolitionism concerns John Brown’s “Pottawatomie Massacre” in “Bleeding Kansas.” On the night of May 24, 1856, Brown and a handful of followers murdered five pro-slavery settlers near Pottawatomie Creek. Tensions had been high in Kansas for years, involving armed combat between pro- and anti-slavery militias and the sacking of Lawrence, an abolitionist stronghold, by “Border Ruffians” from Missouri. Brown’s vigilante executions, in which the victims were hacked to pieces with broadswords, took the violence to a new level. Why did he do it?
For many decades, historians relied on the account of Brown’s son Salmon, who participated in the killings. According to Salmon, his father and his band had been angered by the sack of Lawrence, but their rage turned deadly when they learned that Massachusetts Senator Charles Sumner, while sitting at his desk on the Senate floor, had been beaten nearly to death by South Carolina Congressman Preston Brooks. “At that blow, the men went crazy—crazy,” explained Salmon. “It seemed to be the finishing, decisive touch.” The attack was launched the following night.
Salmon’s story was accepted without question until the biographer Jules Abel attempted to verify it in the early 1970s. To his surprise, Abel discovered that the telegraph lines in 1856 did not extend beyond St. Louis, almost 300 miles from Brown’s encampment. Sumner’s beating had occurred on May 22, and the news could not possibly have reached Brown’s men on the day before the massacre. The eyewitness account—including the vivid detail that a rider had arrived with “the message hidden in his boot”—must have come from “Salmon’s imagination,” Abel wrote. Circumstantial evidence had proven a primary source’s story to be impossible. (No railroad reached Kansas until 1859.)
Contemporary researchers also find remarkable surprises in written records, occasionally requiring reassessment of their own observations. The anthropologist Katherine Verdery frequently visited Romania from 1973-1988, gathering information for her work on political, economic, and ethnic change. She spent much of her time in the village of Bintini, where she found the people “unfailingly cheerful and hospitable,” leading her to “gratefully thank all those who… welcomed me with affection as their friend.” Her professional contacts, too, greeted her with “tolerance, friendship, advice, and assistance far beyond my expectations.” Whatever difficulties Verdery encountered in her research, she attributed to her own “lack of foresight [or] bureaucratic bungling,” rather than “any government wish to conceal data.” In all, she enjoyed “hospitality from both government officials and villagers,” and she was impressed “that there had been no intimidating inquiry” after her departure.
Contemporary researchers also find remarkable surprises in written records, occasionally requiring reassessment of their own observations.
Decades later, Verdery obtained her secret police file from the Romanian state archive. Reading the nearly 3,000-page dossier, she learned that things had not been as they appeared: much had been concealed from her by the government, and over 70 of her most trusted subjects had informed on her to the secret police. Most shockingly, she learned that she had almost been arrested as a spy, and she had been fortunate to leave Romania when she did. Much of the information in Verdery’s folder was misleading or unreliable, as police reports can often be, but the totality of the documentation still led Verdery to reassess many aspects of her research. Far from disdaining the documentation in favor of her earlier assessments, Verdery admirably came to new understandings about the meaning of her field work.
Facts and Beliefs
It may sometimes be difficult to distinguish between actual facts and an informant’s subjective beliefs, as is illustrated by the following true story reported by the Detroit Free Press (all names are real).
In October 2006, Ginnah Muhammad arrived in a Hamtramck, MI, courtroom, expecting to defend herself against a claim by Enterprise-Rent-a-Car. A conservative Muslim, she wore a full veil (niqab) exposing only her eyes. Before Muhammad could testify, Judge Paul Paruk told her that she would have to bare her face, saying, “unless you take that off, I can’t see your face, and I can’t tell whether you’re telling me the truth or not.” Muhammad declined, explaining that removing the niqab would violate her religion. Paruk then barred Muhammad’s testimony and entered judgment against her for $2,750.
If there had been an ethnographer in the courtroom, the resulting narrative might have been framed in either of two ways. It could be a story about the clash between a witness’s faithful adherence to her religion and the judge’s legitimate need to see her face. In that version, Paruk’s assertion would be included for its truth value, meaning that an accurate verdict truly depended on Muhammad’s removal of her niqab. Alternatively, it could be a story about a judge’s unreasonable belief the he had to see every witness’s face and the unnecessary burden that put on Muhammad’s religious freedom. In this version, Paruk’s demand to see Muhammad’s face would remain important, but mostly as a reflection of his attitude toward religious minorities, rather than as an explanation of necessary court processes.
As it happens, Paruk was empirically wrong, as research could have disclosed at the time. It is mostly a myth that anyone, including judges, can determine credibility from facial cues. According to studies by Paul Ekman and others, even supposedly trained professionals experience high error rates when it comes to recognizing falsehoods. In one well-known set of tests, police officers, psychotherapists, judges, trial lawyers, customs agents, and CIA agents all failed to identify liars at a rate better than about 50%. Most of the commonly accepted indicators of deception—blinking, looking down, frowning—are really signs of discomfort or nervousness.
It would be necessary, in any account of the Muhammad case—ethnographic or otherwise—to report the judge’s statement that he relies on facial expressions to decide cases. It would be a mistake, however, to accept his claim at face value, rather than describe it as an apparently sincere but nonetheless misleading belief that could have severe consequences for religious litigants.
Fact-Checking for Ethnographers
The ultimate issue here is whether, and to what extent, facts and fact-checking should matter as much for ethnographers as for historians, journalists, and lawyers. The best ethnographers, perhaps needless to say, take painstaking measures to achieve accuracy, but there are others who seem to minimize the importance of factual specifics in favor of the big picture.
In an interview with the Chronicle of Higher Education, MIT’s John Van Maanen declared, “If ethnography was concerned only with verification and, quote, facts, it would be a kind of dismal endeavor.” Although he did not explain the difference between facts and “quote, facts,” Van Maanen continued, “We wouldn’t get the narrative. We wouldn’t get the story. We wouldn’t get the feel for the culture that is being explored.”
This is a recurring theme among ethnographers. According to Moskos, for example, ethnography is “about the narrative,” so long as “it does reflect a greater truth.” Harvard’s Christopher Winship explained to Slate the distinction ethnographers make between specific facts and general truths: “If you told a sociologist they got a particular fact wrong, they’d say, ‘Well, that doesn’t matter—what’s important is that it’s true in a bigger sense.’”
It should be clear that I reject this paradigm, which is more appropriate to oral history or memoir (or fiction) than it is to social science. It is certainly valid to tell a story based on the subjective beliefs of an ethnographer’s research subjects—what lawyers would call their state of mind. But when it comes to actual truth claims, reasonable care should be taken to verify that reported events occurred as described. An ethnographer studying the current White House would surely report President Trump’s insistence that his 2016 inauguration was the best attended in history, but she should also consult the National Park Service’s aerial photographs to confirm or negate the claim. There is no contradiction between providing a “feel for the culture” and obtaining accurate data.
When encountering questionable or implausible stories, ethnographers must avoid the temptation—based on a greater truth or a bigger picture—to imagine ways in which they might still be true. As the psychologist Daniel Effron has shown, it is not uncommon for people presented with even explicit falsehoods to indulge “counterfactual thoughts” that make the information “feel more or less truthful,” especially when the misinformation is “aligned with participants’ political preferences.” Moreover, neither prior knowledge nor subject-matter expertise provides complete protection from “illusory truth.”
Effron’s prime examples come from politics, as when President Trump retweeted a phony video purporting to show a Muslim migrant committing an assault. Rather than acknowledge the falsehood, White House press secretary Sarah Huckabee Sanders recast the situation as representative of a bigger truth: “Whether it’s a real video, the threat is real.” No social scientist would fall to Huckabee Sanders’s level of political manipulation, but the threat of greater truthism is ever-present. Dingwall, for example, said he had “no problem in imagining” a counterfactual in order to shore up the dubious scenario described by an ethnographer. And Moskos re-envisioned the Mississippi teenagers’ reported disbelief in the existence of elevators as a much more plausible story of seeing “an elevator for the first time.”
Ethnographers are at their best when they make frank distinctions between actual occurrences and subjective impressions, whether their informants’ or their own. For those who adhere closely to the “greater truth” or “bigger picture” school, let me suggest that their books and articles should be accompanied by prominent, explicit, and repeated disclaimers. If the reported facts are not to be taken literally, it is an author’s obligation to say so early and often. And for those—doubtless the majority—who care about fact-checking, well, keep up the good work.