Abstract
Introduction
The NYC Department of Health and Mental Hygiene (Health Department) is an anti-racist, equity-focused agency (NYC DOHMH, 2023a). We value examination of our own practices. Accordingly, we read the paper, “Health Inspector Ratings of Asian Restaurants during the Early COVID-19 Pandemic,” with concern (Cherng et al., 2022). The paper claims that our workforce discriminated against Asian restaurants during January and February of 2020. Upon learning of the publication of this paper, we searched for evidence of bias against Asian restaurants in our inspectional data but have been unable to corroborate this paper’s findings. Moreover, we call attention to the authors’ use of an incomplete dataset for their analysis, which we describe below. The problematic nature of the source data leads us to believe that the results presented by Cherng et al. are invalid, and we reject the conclusions drawn from them.
How to reconcile authors’ recommendations with their practice?
The authors’ concluding statement: ‘Researchers should … partner with public agencies to … utilize data to inform practice,’ is in direct conflict with how they approached their research. The Health Department regularly provides complete restaurant inspections datasets to researchers and academic institutions. We would have welcomed a collaborative effort to investigate the concerns about bias that the authors raised. However, the authors did not once reach out for assistance or in partnership. They did not request an official, complete research dataset, nor did they ask for guidance on how our complex inspectional program works. If they had done so, the errors they committed could have been avoided. For example, their references to “citation counts” throughout the paper are implausible. The paper describes an average of more than 10 citations per restaurant inspection, whereas the actual average is 3 to 4. The authors inaccurately conclude that our inspectional staff do not receive anti-bias and anti-racism training, because ‘[their] inspection of public facing documents found no mention of any type of training that is required of health inspectors.’ But the first result of a Google search for “NYC health department anti racism training” describes the programs we have in place to combat structural, institutional and interpersonal racism in agency operations and policies (NYC DOHMH, 2023a).
Why the paper’s data source is invalid
Cherng et al. used publicly available “OpenData” for their research. One of the downsides of government data transparency movements is that we cannot control how private individuals interpret or share our data. However, if researchers choose to use online data for research, it is incumbent upon them to vet and understand the associated challenges and caveats.
Accompanying documentation of the NYC restaurant inspections dataset on the OpenData platform clearly states that it cannot be used for historical analysis (NYC DOHMH, 2023b). There are two reasons.
First, OpenData only provides inspectional data for currently active restaurants. Restaurants that have closed are removed on a nightly basis. This means that data from previous years over-represent restaurants that have remained in business up to the date of the data pull – and those more established restaurants perform better on average. In contrast, more recent data over-represent newer, less established restaurants that do not perform as well on average. The impact of excluding historical data from restaurants that have closed can be large. Among restaurants that were active on January 1, 2017, 35% were out of business by February 29, 2020. More importantly, patterns of restaurant openings and closings are not random. For example, 8.7% of Asian restaurants active on February 29, 2020, opened on or after September 1, 2019, compared with 4.7% of American restaurants. This is a classic example of confounding by restaurant age: the newness of a restaurant is associated with both being an Asian restaurant in more recent months of data (i.e., January and February of 2020, which can be considered the “exposure” in this analysis) and poorer performance (the outcome).
Second, OpenData is updated in real-time as restaurants contest violations at an administrative tribunal. Every restaurant has the right to contest violations that are cited. If a violation is dismissed by an administrative judge, it is removed from the dataset. Adjudication can go on for months, so the data on OpenData are an unpredictable mix of pre- and post-adjudicated violations at any given time. Such heterogeneous data cannot answer a research question about inspector behavior because the data also reflect the behavior of administrative judges.
Further compounding our concerns, the source data cited by Cherng et al. (see Note 2 in their article) is not the official NYC restaurant inspections dataset on OpenData. Instead, they cite a “view” that was created by an anonymous member of the public (Anonymous, 2023). This view excludes over half the rows from the full dataset. The excluded data are primarily records from “initial” inspections that did not result in an A grade.
The Health Department conducts at least one initial inspection of every restaurant annually. If the restaurant does not earn an A grade (<14 points), it is reinspected about 30 days later (NYC DOHMH, 2023c). A little less than half of restaurants do not earn an A on initial inspection. An assessment of discrimination during the inspection process cannot be considered valid if initial inspections are excluded from analysis, because it is those initial inspections that result in the most punitive actions. Restaurants that do not receive an A on their initial inspection are fined; inspected again 30 days later (with more fines if they again fail to achieve an A grade); and put on a more frequent inspection schedule. It is difficult to predict the impact of excluding almost half the data from initial inspections from an analysis of discrimination, but it is all but certain such an exclusion would generate uninterpretable patterns in an analysis of scores.
Presentation of results from a complete dataset
aViolations are limited to those that are scored. An analysis that included unscored violations generated the same patterns.
bInspections are defined as a unique date of visit.
cRestaurants self-identify cuisine type on their permit application. This analysis categorized cuisine as “Asian” (Asian/Asian Fusion, Chinese, Chinese/Cuban, Chinese/Japanese, Filipino, Japanese, Korean, Southeast Asian, Thai), “American” (American, Californian, Hamburgers, New American), Caribbean, Italian, “Latin” (Brazilian, Chilean, Latin American, Peruvian), Mexican (including Tex-Mex), and all other.
We note that restaurants that self-identify as Asian cuisine do not perform as well as restaurants with self-identified American cuisine type on average; we see the same pattern across other cuisine types that are more likely to be small and independently owned (vs franchises or chains). It is well-established that chain restaurants perform better on average during inspections, making it a potential confounder in an analysis that uses restaurant characteristics to predict inspection results (Brown et al., 2014; Leinwand et al., 2017). We are troubled by the lack of control for chain restaurant status in the authors’ predictive models. The likelihood of uncontrolled confounding is high, as only 2% of Asian restaurants are chains or part of a franchise, compared with 14% of American restaurants (as of January 1, 2020).
Conclusion
For the reasons stated above, we believe the findings of Cherng et al. are erroneous. We further emphasize that the authors’ claims diminish and contradict our agency’s extensive investment in and dedication to combating racism, and we reject the unfounded accusations of discrimination in this paper.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Data availability statement
The data given in this article are available upon request from the New York City Department of Health and Mental Hygiene at 
.