Judging a book by its cover: comparing visual grammar analysis of book covers with an eye-tracking study on viewer perceptions

Abstract

This study represents an initial effort to integrate Visual Grammar analysis of book covers with an eye-tracking study of 65 Chinese viewers’ perceptions of 10 covers. While Visual Grammar usefully systematizes meaning-making resources in multimodal texts, there remains a lack of empirical evidence to support the accuracy of semiotic analysis in predicting reader responses. Drawing on eye-tracking and supplementary questionnaire data, this study tests the predictive capabilities of nine semiotic predictors, based on the two compositional systems of position and salience, for viewers’ visual processing and perceptions of book covers which combine several semiotic modes to inform and engage potential buyers. It reveals that font style and a centre-margin processing sequence are the most reliable predictors, followed by a top–bottom sequence. However, no dominant left–right or right–left sequence emerges on covers featuring vertical orthography. Furthermore, the accuracy of these predictions is moderated by the hierarchies created by position and salience, and readers’ top-down assessments. These findings reflect a complex interplay between bottom-up (compositional variables) and top-down factors (goal and attraction) in guiding readers, highlighting the need to complement semiotic analysis with more reception and experimental studies in visual communication.