Do Employers “Walk the Talk” After All? An Illustration of Methods for Assessing Signals in Underpowered Designs

Abstract

This article presents two alternative methods to null hypothesis significance testing (NHST) for improving inferences from underpowered research designs. Post hoc design analysis (PHDA) assesses whether an NHST analysis generating null findings might otherwise have had sufficient power to detect effects of plausible magnitudes. Bayesian analysis with default priors offers advantages over NHST for assessing null findings and detecting signals in underpowered data. Both methods are illustrated by application to Pager and Quillian’s influential study on attitude-behavior correspondence. PHDA results suggest the original study lacked sufficient power to detect strong associations between employers’ attitudes and behaviors. Bayesian analysis confirms strong attitude–behavior associations cannot be ruled out given the data. Together, these results question a frequently cited conclusion about attitude–behavior incongruence in survey vignettes. Overall, the examples illustrate how these analytical tools can be useful for describing uncertainty surrounding estimates and for improving substantive and theoretical debates across sociology.