Rating Product Satisfaction by Watching Others Using Video

Abstract

During the COVID-19 pandemic, usability practitioners and researchers had to find new approaches to product testing. In-person, contact-intensive product testing became a safety concern, resulting in the need for more remote testing practices. An underexplored and promising method to capture subjective usability measures is Watching Others Using Video, wherein users rate a product after watching videos of others using it. This can have broad application, but previous research found this method yields positively biased usability ratings relative to post-use ratings. This study explored potential factors (e.g., success, error, and failure) that may impact how users perceive satisfaction using this method. To do so, participants were shown videos with different product interactions while systematically varying the factors of interest. Additionally, the effects of the number of errors and error recovery versus failure were explored. Participants watched different videos of the following products being used and rated them using the After-Scenario Questionnaire (ASQ): a website, electric can opener, digital timer. Results found inflated satisfaction ratings across products, however the effect did not reach statistical significance for the website. There also was no observable effect of increasing errors or showing failures. This may be attributed to poor error detection or negligible error severities. Further research is needed for Watching Others Using Video to be accurately implemented as a viable testing method.