An Assessment of the Effectiveness of Multiple Hypothesis Testing for Geographical Anomaly Detection

Abstract

The practice of multiple significance testing is reviewed, and an alternative to the frequently used Bonferroni correction is considered. Rather than controlling the family-wise error rate (FWER)—the probability of a false positive in any of the significance tests—this alternative due to Benjamini and Hochberg controls the false discovery rate (FDR). This is the proportion of tests reporting a significant result that are actually ‘false alarms’. The methods (and some variants) are demonstrated on a procedure to detect clusters of full-time unpaid carers based on UK census data, and are also assessed using simulation. Simulation results show that the FDR-based corrections are typically more powerful than FWER-based ones, and also that the degree of conservatism in FWER-based procedures is quite extreme, to the extent that the standard Bonferroni procedure intended to constrain the FWER to be below 0.05 actually has a FWER of around 6 × 10⁻⁵. We conclude that in situations where one is scanning for anomalies, the extreme conservatism of FWER-based approaches results in a lack of power, and that FDR-based approaches are more appropriate.