Abstract
Syed Ali on Interrogating Ethnography
Interrogating Ethnography: Why Evidence Matters by Steven Lubet Oxford University Press, 2017 216 pages
Steven Lubet is a lawyer and historian who is fascinated by ethnography. He thinks ethnographies are great, but asks how we can know whether they accurately portray reality. Because we (as readers and fellow researchers) generally don’t have access to the ethnographer’s field notes and interviews the way quantitative researchers more commonly make their data and codes public, the truth basis of ethnographies is basically “trust us.” That doesn’t cut it for this lawyer. The book takes an adversarial approach, putting ethnographies “on trial” so as “to assess the use of evidence in ethnography—in terms of sources, collection, presentation, and dependability—by comparing it to the standards that have been developed to determine the reliability of evidence in law practice.”
I’ll cut to the chase: This book is an essential critique of the most public-facing product sociology has to offer. It should be required in every sociology (and anthropology—I’m looking at you!) graduate methods class, and probably in undergrad methods classes, too. It’s a fast, easy read, and it lays bare the issues in an admirably clear way that earns it the Contexts’ Seal of Approval™.
Lubet got started after reading Alice Goffman’s On the Run: Fugitive Life in an American City, for which she conducted a six-year ethnographic study of a poor Black community in West Philadelphia. While that book got heaps of praise, it also garnered a lot of criticism (including from my Contexts co-editor, Philip Cohen). Lubet’s problem was that he found many of the author’s claims implausible, at best. At least a dozen assertions were inconsistent with his experience as a defense lawyer in Chicago. Could Philly be that different? Lubet got curious, and he checked newspapers and public records, then interviewed knowledgeable public defenders and defense lawyers, police officers, prosecutors, hospital staff, university employees, city officials, and neighborhood residents. His conclusion? Many of Goffman’s descriptions of events ranged from improbable to impossible.
How could such inaccuracies have been printed by the University of Chicago Press, notorious for its rigorous editing? (And not just them: Goffman’s work has appeared in many high-profile publications, and no one questioned her work’s veracity.) Lubet quotes Christopher Jencks’s review from the New York Review of Books: “This is a world with which few readers of this journal are likely to have had much contact. I certainly haven’t despite having spent a lifetime writing about social policy.” Social scientists, liberal though they generally may be, tend to have no first-hand knowledge about the lives of the people Goffman wrote about, so they took it on faith that Goffman’s accounts had to be accurate. This is the key point—we assume what ethnographers tell us is factually correct. So was Goffman’s work an aberration in ethnography, or the norm?
Driven by that question, Lubet’s book is an extended exploration of sociologists’ ethnographies of U.S. cities. His reading list is wide-ranging in theme and in time, from W.E.B. Du Bois’s The Philadelphia Negro (1899) up to Crook County (2016) by Nicole Gonzalez Van Cleve and Evicted (2016) by Matthew Desmond.
Lubet found that most books held up pretty well under scrutiny, though he singles out Evicted for special praise, noting that “Desmond hired an independent fact-checker who corroborated episodes in the book by conducting additional interviews, locating public documents, and reviewing Desmond’s field notes.” (Wow. That’s amazing, though probably way more expensive than most ethnographers can afford.) For Lubet, On the Run and Evicted stand at opposite ends of his consideration of evidence in ethnography, with pretty much everyone else falling in between.
While he has a lot of criticism for Goffman (though he also finds great virtues in her book) and much praise for Desmond (who benefitted greatly from foundation grants and his position as a Harvard professor), Lubet spends more time talking about the books and projects people are conducting in between these extremes, suggesting ways to strengthen and support academic findings.
Ultimately Lubet’s concern is with accuracy and reliability, and his book is a journey toward strengthening both within ethnography. He starts by discussing American trial evidence—specifically testimony, hearsay, opinion, and documentation—to shed light on the use of sources and informants in field work. He then applies these ideas to unreliability, credulity, selectivity, rumor, anonymity, and criminality—problems that can plague ethnographers—and closes with some recommendations.
Lubet lays bare a lot of assumptions ethnographers work with, including the value judgments that make their way into our writings. “In real trials, implicit value judgments are often excluded as ‘characterizations.’ There is no such objection in ethnography, nor should there be, but it would be better if authors would make a point of explaining what they mean.”
We also don’t rely on documentation (the way Desmond admirably does), which is too bad because documents usually don’t stretch the truth the way humans sometimes can, consciously or not. Lubet gives an example from Kathryn Edin and Luke Schaefer’s $2.00 a Day in which a main character is summarily fired after eight years because her cash drawer is $10 short. The details are crucial to arguments in the book, but is the story true? We don’t know whether the protagonist stretched the truth, because the authors didn’t fact-check this important recollection. There are documents that could ascertain the truth—like employee files at the particular (anonymized) store, tax returns, and the like. Lubet’s antennae were raised, he writes, because eight years is a really long time to work at one place in retail, and summarily firing a long-time employee seemed capricious. So Lubet asked a journalist who specializes in retail; she told him that a reliable employee would likely “have the goodwill of their supervisors who also know how hard it is to find and retain good staff who work well with others, know the merchandise, etc. In that instance, I doubt they’d fire someone that quickly, as other factors would come into their decision-making.” Of course it’s possible that the woman was telling the truth about the length of her employment and the cause and manner of her dismissal and that it was all relayed accurately to Edin and Schaefer. But then again, maybe not. We don’t know for sure, because the authors took it on faith.
The truth basis of ethnographies is basically “trust us.” That doesn’t cut it for lawyer and historian Steven Lubet. Thus, this book is an essential critique of the most public-facing product sociology has to offer.
At the root of many of these problems ethnographies have is the default tendency/standard practice of anonymizing people and places, as well as changing personal characteristics, altering facts, and rearranging or eliding time sequences. Lubet writes, “When all identities are thoroughly masked, it becomes nearly impossible to separate reliable informants from storytellers and rumor-spreaders, much less to determine whether the author has omitted or distorted essential information.” Many researchers believe that this is required by university institutional review boards (IRBs), but that’s actually not the case, as Alexandra Murphy and Colin Jerolmack pointed out in their essay for a Contexts viewpoints feature called “How To Do Ethnography Right” (Spring 2016).
Echoing another point made by Murphy and Jerolmack, Lubet notes that the practice of using pseudonyms or omitting names effectively undermines replicability. The inability to do re-interviews or to re-investigate frustrated Jerolmack in his study of fracking in Pennsylvania. Another researcher had studied the same area pre-fracking, and information about the place and the people could have served as a great resource to help show change over time, but everything had been anonymized. This means that successor ethnographers, like Jerolmack, can’t incorporate prior research into longitudinal studies or question earlier accounts. Sociologist Christopher Winship has put it bluntly: “It makes it really hard to verify—you don’t even know if the people exist,” adding, “the discipline thinks it’s fine and that’s probably totally wrong.”
In his last chapter, Lubet addresses criminality. He starts with a sort of roll-call of famous lawbreakers in the ethnographic tradition—William Foot Whyte hanging out at an underground gambling joint and engaging in voter fraud; Clifford Geertz observing cockfights in Bali; Howard Becker explaining how to “get high”; and perhaps most controversially, Laud Humphreys posing as a “watchqueen” for men’s anonymous and clandestine sexual encounters in public bathrooms. (Humphreys was also in murky ethical waters for writing down license plate numbers and tracking down these men to interview them under the pretense of a community health study about their lives and families.) He then turns to episodes in Sudhir Venkatesh’s Gang Leader for a Day and Randol Contreras’s The Stickup Kids, showing how they walked a fine line between legal and illegal. Venkatesh said that, after being ordered by his department at the University of Chicago to consult a lawyer, he learned, “[I]f I became aware of a plan to physically harm somebody, I was obliged to tell the police. Meaning I could no longer watch the gang plan a drive-by shooting, although I could speak with them about drive-bys in the abstract.” Lubet, in discussing how Venkatesh and Contreras came close to, and sometimes crossed the line of criminality, observes that, “It turns out that avoiding crime is not so easy when studying it up close.” This seems obvious, and at the same time profound. When you’re in the thick of it and events are moving fast, it must be hard to take stock and step back. Sometimes Venkatesh failed, like when he took part in a beating in a stairwell.
The ethnographer asks readers and other researchers to trust them, even though they’re building a foundation whose soundness we cannot test.
And then Lubet goes back to Goffman. Following the murder of her friend “Chuck,” she drove “Mike” around looking for Chuck’s killer. She writes that Mike kept “his hand on his Glock as he directed [her] around the area.” After the book was published and criticism mounted, Goffman wrote on her website that this was a mourning ritual meant “to satisfy the feelings of anger and pain” after Chuck’s murder, not a scouting mission for a retaliatory murder. But Lubet’s not having any of it: “These events, as Goffman recounts them in On the Run, constitute a conspiracy to commit murder under the laws of Pennsylvania and virtually every other state. In her own words, she agreed to assist in the commission of a crime, and she engaged in multiple ‘overt acts’ in furtherance of the scheme. Thus, she committed a felony, even though the potential victim was never located. According to one former Philadelphia prosecutor, to whom I provided the relevant passages, ‘She’s flat out confessed to conspiring to commit murder and could be charged and convicted based on this account right now.’”
Ethnographies are strange. Unlike most journalism and other forms of research, they suffer from a lack of replicability and fact-checking. Lubet implores ethnographers to do better by engaging in practices to strengthen the perceived validity of their work. They should fact- and citation-check. If there are no court or newspaper documents, they should ask other experts whether their results are plausible (and include that information in their subsequent accounts). Changing standard practice to naming actual locations and subjects (with anonymity as the fall-back rather than the default practice) would allow for revisits and re-interviews. And don’t make composites of people or change minor details; they are necessarily inaccurate. (I would add that they constitute white lies that undermine ethnographers’ arguments. If you’re fibbing about this, what else are you fibbing about?) Finally, he writes, ethnographers should allow others to examine their field notes. Ok, it’s not practical that ethnographers post the full suite of their raw notes to SocArXiv, but they should certainly allow other researchers access to that data.
Basically, ethnographic practice is kind of a black box. The output is the book (and, often, a Contexts article), and we’re impressed. But we have no idea how it was made. The ethnographer asks readers and other researchers to trust them, even though they’re building a foundation whose soundness we cannot test. Lubet says instead that ethnographers should trust their readers by making transparency their first commitment. I agree.