Abstract
This exploratory study inquires into the validity and reliability of dedicated mobile phone diary applications. We developed Watchy, a dedicated mobile viewing diary application, and compared users’ compliance and usage patterns with those of users of the paper viewing diaries. Participants received paper diaries or installed mobile diary apps, with or without daily reminders, to document their viewings over a 4-day period. Documentation was more extensive in the smartphone app with reminder group compared to the paper diary group. Reminders increased documentation rates. Extent of documentation decreased as the experiment progressed for mobile app users. Findings suggest that mobile viewing diaries are an important tool for viewing studies, yet their use requires careful planning.
Introduction
The new media era offers new possibilities for media content consumption, and changes the consumption habits of many audiences. This is true for all types of genres but is especially true for the news genre, which is characterized by speed and immediacy (Lee and Lee, 2015; Voorveld and Viswanathan, 2015). Many findings point to changes in news consumption among today’s audiences, including increased consumption of news content through new media platforms such as smartphones (Westlund and Färdigh, 2015), consumption of news content originating from social networks rather than dedicated news sites (Gottfried and Shearer, 2017), or performing second screen actions parallel to the consumption of news content (Lowenstein-Barkai and Lev-On, 2018; Gil de Zúñiga, Garcia Perdomo, and McGregor, 2015; Vaccari et al., 2015).
The rapid changes described above also require the development of new methods to keep pace with technological development and enable researchers and professionals to characterize the contemporary media ecology in a valid and reliable manner. Thus, if in the past paper-and-pen questions or written diaries were sufficient to map media consumption habits, today it is possible to harness the ubiquity of smartphones for developing customized measurement applications.
In recent years, there has been a growing interest in network-based measurement methodologies and their potential advantages (see, for example, Joorabchi et al., 2013; Zhang et al., 2017). However, the number of comparative studies—those that compare both old and new techniques at the same time and examine differences in the outcomes they provide—is limited (Ohme et al., 2016). Moreover, in many of the studies that measure media consumption, the choice of the technology to be used to collect the diary data is either arbitrary or not made explicit (Hoplamazian et al., 2018: 134).
The present study aims at contributing to the emerging field of measuring contemporary media consumption habits through a comparative experiment in which a dedicated smartphone application was developed and its use was compared to that of traditional paper-and-pen diaries. The comparison of the two techniques provides an opportunity to examine whether and to what extent digital devices may be an alternative to traditional methods of measuring media consumption.
The paper diary method
Measuring the consumption of media content is almost as old as the media itself. From the beginning of the development of the commercial media, content producers were required to estimate the extent of their consumption in order to know what type of content to invest. In the television market ratings, data were also needed to build the broadcasting schedule and to know where to put each program (Carpentier et al., 2013; Napoli, 2011).
The paper diary method was developed in the 1940s and is one of the oldest methods for measuring media usage. Despite the technological advances that have triggered the development of digital methods to measure viewing in advanced digital ways, the paper diary method is still prevalent in major TV markets in the United States (where it is supposed to be replaced in 2018 by digital tools), Canada, and East Asia due to its relatively low cost (Carey, 2016; Fisher and Gershuny, 2013). The method is based on self-reporting: Diaries are sent to the homes of sampled representatives of the population, who are directed to record the television content they view. The details recorded in the diaries typically include the name of the program, the time and channel on which it was aired, and demographic information of the family member who watched the program. After several days, the diaries are collected and their data are coded (Webster et al., 2013).
The paper viewing diary method is not free of problems, resulting in low internal and external validity (Alaszewski, 2006; Li, 2014). First, it is argued that it may lack external validity. Studies comparing the completion patterns of paper diaries and electronic devices such as the People Meter showed that the level of cooperation of young people with the paper diaries was significantly lower in comparison with adults (Shimp and Andrews, 2013). Furthermore, since the method requires literacy, it may exclude illiterate populations, which may be more prevalent among minority groups than in the majority population (Webster et al., 2013).
Second, even if the sample who record their viewing habits in paper diaries is representative of the entire population, the method suffers from an inherent weakness regarding its internal validity: The method relies on self-reports, and lacks any method to confirm the veracity of the reports. If a person noted in his diary that he was watching a specific content at a specific time, there is no way to confirm this report (Carey, 2016). Various data show that a percentage of sampled groups fail to follow the guidelines for recording a real-time log, and complete their diary at a more convenient time. Others complete their diaries on weekends and thus fail to record significant contents consumed earlier in the week (Feinberg et al., 2013). The fact that some participants record their viewing in retrospect and rely on their recollection generates concerns that the method prioritizes programs on major television networks, which are more salient in media consumers’ recollection (Napoli, 2011). Other data demonstrate a “diary fatigue” effect, as participants’ weekly compliance is high during the first half of the week, and gradually diminishes toward the weekend (Webster et al., 2013).
Third, the usage of paper documentation also reduces the method’s internal validity, because in cases where the diary is not available, subjects cannot document their viewing. This may be especially relevant for out-of-home viewing (Lekakos, 2007).
Still, the paper diary method has prevailed in major TV markets in the United States, Canada, and East Asia as a complementary method to the electronic methods because of its relatively low costs (Fisher and Gershuny, 2013). In the United States, for example, about two million viewing diaries are still collected annually during November, February, May, and July, which are called “sweeps” rating periods (Nielsen, 2014). The method is the only basis for pricing advertisements in local markets (Napoli, 2005).
Over the years, more sophisticated measurement technologies have been developed, such as the People Meter and the Portable People Meter. The main advantage of these methods is their lack of reliance on respondents’ self-reports. Once a respondent is identified (when he or she presses a special identification button on the TV remote or a pager-like device), the content that he or she views is recorded automatically. A more recent extension of the viewing diary method, which enjoys most of its advantages but does not suffer from most of its weaknesses, is based on mobile applications.
Mobile diary applications
Mobile diary applications have been in use for several years in other fields including psychology and medicine, for documenting patterns of behavior (such as leisure activities and eating habits) and sometimes as tools to help understand the emotions and motivations associated with such behaviors (Henderson et al., 2005). The broad distribution of smartphones and the fact that they are carried on or close to users’ bodies throughout most of the day (Klasnja and Pratt, 2012) make them invaluable platforms for self-reported measurements and calls have increased to use their advantages in behavioral studies as alternative technologies to collect self-reports (Eveland et al., 2011).
Mobile applications can also send users reminders to complete their diary, track reporting in real-time, and even filter out retroactive reporting. Another advantage of the mobile diary method is its unrestricted physical space requirements compared to paper diaries. Data are even returned and coded online, in a transparent manner, unlike paper diaries that suffer from logistical obstacles related to their dispatch to users and subsequent collection from users to research teams (Alaszewski, 2006); Furthermore, and due to the time that elapses until all diaries are obtained and coded, data publication may be delayed (Napoli, 2011).
Most of the specialized applications using mobile phones in which the participants report on their activities in real time have been developed as part of Ecological Momentary Intervention research, and they include active/intervened features (such as notifications). For example, Reid et al. (2009) developed a mobile phone program to track young people’s experiences of mood, stress, and coping behaviors in real time. Lane et al. (2011) developed BeWell, a mobile application that automatically monitors a person’s physical activity, social interaction, and sleep patterns; summarizes the effect of the monitored behavior on well-being; and provides feedback that enables users to effectively manage these aspects of their health.
Studies have also recorded higher compliance rates in recording tasks involving mobile phone applications compared to other methods. For example, Carter et al. (2013) collected outcomes of a self-monitoring weight-management intervention delivered by a smartphone app and compared them to outcomes using website and paper diaries. The results indicated that participants’ compliance with dietary self-monitoring was statistically significantly higher in the smartphone group than in the website and paper diary groups. The compliance effect of mobile-phone-based diary applications increases when technology-based reminders are used. Tapia (2004) showed that reminders are useful both as a memory aid in fixed study designs and as a means of sampling participants’ experiences according to a variable (random or simply unknown to participants) schedule within prespecified time blocks. Palermo et al. (2004), who compared two groups of children with headaches or juvenile idiopathic arthritis who completed pain diaries, found a higher level of compliance in the group that used electronic diaries that included reminders, compared to the group that completed paper diaries without reminders.
The mobile diary method is not, however, without weaknesses, including the high demands imposed on users who are required to download and install a mobile app (compared to receiving a diary by mail), especially among technologically illiterate populations; as well as potential technical problems resulting from researchers’ need to support multiple operating systems. Nonetheless, its advantages seem to outweigh its disadvantages. Table 1 compares paper and mobile diaries on several key elements.
Comparison of paper and mobile diaries.
Although, to the best of our knowledge, no mobile phone application exists for measuring video content viewing, we assume that the advantages of mobile diaries may also be realized in applications for reporting on television content viewing. A mobile diary application might also overcome the validity and reliability issues related to paper viewing diaries. To explore the potential of mobile viewing diary apps end, we developed Watchy, a dedicated mobile viewing diary application, and compared users’ compliance and usage patterns with those of users of the paper viewing diaries. We, furthermore, tested the internal validity of this new diary tool for documenting viewing patterns on mobile phones. The case study we use comprises the viewing patterns of news videos. A “news video” was defined as a video whose content deals with news and current events, such as news reports, news releases, news flash videos, political commentary programs, and so on. News is one of the most consumed genres in Israel, and the Israeli public consumes news content at a very high rate. According to the Israeli TV Rating Committee, in January 2013, news was the second most popular category of videos among Internet users (Mann & Lev-On, 2014), and in 2014, the country’s three major news programs had a combined average rating of approximately 40% (Walzer, 2015).
Research aims
Given the technological changes described above on the one hand, and the increasing consumption of video content on the other, the aim of this study is to contribute to the evolving field of evaluating digital measurement methodologies. We want to examine how modern media audiences respond to application-based viewing logs compared to traditional paper-and-pen diaries, whether their fatigue rate is faster/slower, and how features such as reminders affect their fill volumes. By doing so, we want to examine whether and to what extent digital technologies are more useful for measuring media consumption than traditional platforms.
Research questions and hypotheses
RQ1. What is the extent of recording of viewings in mobile diaries compared to paper diaries?
H1. Participants who use mobile diaries (with reminders) to record their viewing will report more viewings than participants who use a paper diary (based on findings of Carter et al., 2013)
RQ2. What effect do diary reminders have?
H2. Participants in the mobile diary with reminders group will report more viewings than participants using mobile diaries with no reminders (in line with findings of Palermo et al., 2004; see also Iida et al., 2012).
RQ3. Does participants’ scope of recording decline over the length of the experiment?
H3. Scope of recording will decrease as the week advances (Webster et al., 2013).
RQ4. Does the extent of information recorded in mobile diary apps differ by age, gender, education, or income?
H4. Age is the only variable that has been frequently found to predict differences in recording patterns (Broderick et al., 2003; Iida et al., 2012; Shimp and Andrews, 2013), where older people record information in diaries more frequently than younger individuals. Only one study we know (Morren et al., 2009) found that the number of reports increased with education. Therefore, the hypothesis addresses age only and predicts that number of reports will be higher in older participants compared with younger participants.
Methodology
The study compares the patterns of recording news content using mobile and paper viewing diaries. A dedicated Android-based mobile application was developed and tested on a variety of Android platforms and was found to work on generation “Galaxy 2” phones or higher. A paper diary was also designed using a traditional format.
The study population comprises all smartphone owners in Israel. It is estimated that two prominent types of platforms that are used almost equally used, iOS/iPhone and Android, dominate about 90% of the market (Mann & Lev-On, 2014). Smartphones also constitute a major and almost exclusive source of access to the Internet via mobile (Mann & Lev-On, 2014). After verifying that no significant sociodemographic differences (gender, age, education, and income) were found between Android users and iPhone users, a decision was made to develop the dedicated application for this study for Android smartphones only, for reasons of savings in development, testing, and maintenance costs.
To construct the sample, we used the services of a commercial polling institute. Participants were divided randomly into three groups, while ensuring quotas of age and gender, with the aim of getting at least 100 subjects who filled the diaries they received throughout the entire 4 days that the experiment lasted. Two of the three groups were asked to download a mobile application. Members of the third group received paper viewing diaries. Members of the paper viewing diary group were also smartphone owners, a criterion designed to rule out the possibility that differences in report patterns (if any) might be attributed to the different behaviors of individuals who do or do not own smartphones.
However, as typically happens, the number of participants who completed the experiment is lower than the original number of people who expressed willingness to participate in the study, for several reasons: (1) not everyone who expressed willingness to participate in the study actually downloaded and installed the application as instructed; (2) not everyone who downloaded the app registered on the opening screen as requested; (3) several individuals who registered apparently changed their minds and did not turn on the app in real time; and finally (4) several paper viewing diaries were lost in the mail. Consequently, our final sample comprises data from 356 smartphone owners in Israel who completed the study.
Study design
Participants were divided into three experimental conditions.
Group 1: mobile diary plus reminder
Participants in this group were instructed to complete the viewing diary via a mobile application. They received two reminders a day. One hundred ninety individuals in this group received a link with an invitation to participate in this condition of the experiment, of whom 122 completed the experiment (n = 122), a completion rate of 64.2%.
Group 2: mobile diary no reminder
Participants in this group used the mobile diary application to record their viewings and received no daily reminders to do so, although they received text messages at the beginning and the conclusion of the experiment. The 209 individuals in this group received a link with an invitation to participate in this condition of the experiment, of whom 105 completed the experiment (n = 105), a 50.2% completion rate.
Group 3: paper diary
Participants in this group received paper diaries. Of the 192 paper diaries that were distributed, 129 completed diaries were received by the investigators (n = 129), a 67.2% completion rate.
Screenshots of Watchy. From left to right, the screens allow users to record (1) the time the video news content was viewed, (2) the duration of viewing, (3) the platform, (4) the website, if viewed online, (5) whether users were involved in other communicative acts while viewing, and (6) the communicative acts in which users were involved during viewing.
Five days before the beginning of the experiment, participants in the first two groups received a message from the polling institute to download the (free) app from Google Play, using a link sent to their smartphone. After installation, participants were instructed to open the app and register, using the email they frequently use, to allow the system to recognize them every time they report their viewing. They were also instructed to enter biographic information (age, gender, and average volume of news consumption).
The 192 participants in the paper diary experimental condition received paper viewing diaries by express mail 1 week before the start of the experiment. They also received a stamped envelope, with the return address of a mailbox that was rented for the purpose of this study. In line with standard paper diary study protocols, the investigators contacted a sample of this group 2 days later to confirm that the diaries had been received and that participants were willing to proceed with the study.
The following Sunday, we sent SMS messages to all members of the three groups to remind them that the experiment would begin the following day. Participants were given the email address and phone number of a contact person in the study team, whom they could contact for assistance or information. The experiment was conducted over 4 days, from Monday morning at 9:00 a.m. to Friday morning at 9:00 a.m. These specific days were selected because people have been shown to have different viewing patterns on weekends, and religious people do not watch video content, write, or type over the weekend. Sunday was devoted to the logistical needs of the research and other preparations (in Israel, Saturday is a rest day and the working week starts on Sunday).
All three groups received identical instructions: They were instructed to use the mobile app or paper diary to document every occasion on which they consume news-related video content. Instructions included a definition of news-related video content. To allow the investigators to distinguish absence of viewing from failure to use the viewing diary, participants were also told that if they consumed no news-related video content on any specific day, they should record this fact in their mobile or paper diary.
Participants were instructed to report any viewing of news video content immediately after viewing, including the time, duration, platform, and whether the participant was concurrently engaged in another communication operation and if so what kind. Participants’ viewing records were saved to a database, and every record included the user’s details and the actual time of documentation.
On the morning of the first day of the experiment, participants received a reminder that the experiment had begun and would last from Monday morning to Friday morning. At 8:00 a.m. on Thursday, Groups 1 and 2 received a reminder that the experiment would end at 9:00 a.m. Group 3 (paper diaries) received the same reminder in addition to instructions to mail the paper diaries by mail.
In addition to the reminders at the beginning and the conclusion of the experiment, Group 1 received reminders twice a day on each day of the experiment, at 8:00 a.m. and 8:00 p.m. 8:00 a.m. is the time that represents the beginning of the day and 8:00 p.m. was chosen because major news programs typically start at this time. The wording of the reminder was, “This is a reminder from the Watchy app: If you have not yet reported your viewing yet, please do so now.”
All participants were equally compensated for their participation by the polling institute. Compensation was standard and not based on the extent of reporting to prevent participants from inflating their viewing documentation. No unusual newsworthy events occurred in Israel during the week of the experiment, and, therefore, this period can be considered a representative period of normal daily news consumption.
Findings
Number of recorded viewings
As shown in Table 2, participants in the mobile diary plus reminder group recorded a greater number of viewings than either of the other two groups: Participants in the mobile diary plus reminder group recorded an average of 2.88 views per day compared to an average of 2.48 views per day reported by the paper diary group, and an average of 2.32 views per day by the mobile diary no reminder group.
Recorded viewing by group.
p < 0.05.
We used a one-tailed analysis of variance (ANOVA) test to compare the extent of recorded viewing in the three groups. Analysis of the findings reveals a significant difference in the extent of recording performed by the three groups. Scheffe post hoc analyses indicate that the number of viewing records was significantly greater in the mobile diary plus reminder group, than in the mobile diary no reminder group. The number of viewing records was significantly greater in the mobile diary plus reminder group, than in the paper diary group.
These findings confirm hypothesis H1, which predicted that mobile diary group members would provide more extensive viewing records compared to members of the paper diary group.
These findings also confirm H2, which predicted that the number of recorded viewings would be significantly greater in the mobile diary plus reminder group than in the mobile diary no reminder group. Participants in the mobile-diary-no-reminder group received no reminders since the beginning of the experiment, while participants in the mobile-diary-plus-reminder group received a total of nine reminders over the days of the experiment.
We also examined whether the number of recorded viewings declined over time. To compare all groups, an ANOVA test for repeated measures was conducted. We found differences in the number of recorded viewings by day, F(3, 1071) = 10.69, p < 0.01. Results of a follow-up Bonferroni test indicated that these differences can be attributed mainly to differences between the first day (average of 2.74 recorded viewings) and each of the other days experiment. Excluding the “sunset effect” on the final day of the experiment, which has been reported in previous studies, these findings partially support H3.
Comparing the total number of recorded viewings in each group, striking differences were found between the paper diary group and the two mobile diary groups (see Table 3). While the majority of participants in the two mobile diary groups provided a total of 4–9 recorded viewings, the majority of paper diary participants provided 10–12 recorded viewings, the maximum number, or almost maximum, of views they could enter into the paper diary. Recall that members of the paper diary group were limited to 12 reports throughout the experiment, while members of the mobile diary groups were not limited in the number of viewings they could record.
Distribution of recorded viewings by group.
Age and gender effects
Next, we examined age and gender effects on the extent of recorded viewings. Findings of t-tests showed no effects of gender on extent of recording. To analyze the effect of age, the sample was divided into two groups—individuals between ages 18 and 37, and individuals between ages 38 and 85. The cutoff age (37) was selected in line with Prenski’s (2001) definition of “digital natives”—people born on or after 1980, who tend to be “native speakers” of the digital language of computers, video games and the Internet (p. 2).
Comparing recorded viewings by age group, we found that younger participants recorded fewer viewings per day compared with older participants, on all 4 days of the experiment. Findings of a two-way ANOVA indicate that while older participants recorded a greater number of viewings than younger participants in all three study groups, no age × group interaction effect was found. These findings confirm hypothesis H4, concerning the effect of age on diary recording (Table 4).
Daily recorded viewings by age.
p < 0.00, *p < 0.05.
Recorded viewings by time of day
Finally, we examined participants’ viewing records by time, using a division into time strips conventionally used in the television market (morning, afternoon, pre-prime, prime-time, and post-prime-time). Table 5 demonstrates that most viewing reports throughout the entire experiment were made between 6:00 a.m. and 1:00 p.m. for all groups.
Distribution of viewing reports throughout the day, by group.
Discussion
This exploratory study examined the patterns of documenting video content viewing using a mobile application and paper diaries. For this purpose, a mobile application was developed to document viewing, and participants were asked to record the video content they watched over a 4-day period, using a paper diary or a mobile application. Participants who used the mobile app either received or did not receive reminders during the experiment to record their viewings.
The most interesting finding is related to differences between the groups in terms of extent of documentation. A significantly higher average daily documentation rate was found for participants who used the mobile app with reminder, compared to participants who used paper diaries, in spite of the fact that paper diaries were not regulated by the experimenters, and allowed participants to document viewings retroactively, contrary to the instructions. It should be noted that these findings align with findings of other viewing diary studies, such as studies in the field of health and lifestyle, which found a significantly greater extent of documentation by users of mobile apps compared with users of the paper diaries (Stone et al., 2002).
Furthermore, reminders were effective: Participants in the mobile app with reminder documented their viewings more extensively than the participants in the mobile app with no reminder group. This finding points to the relative utility of application-based logs compared to written diaries and reinforces previous studies that examined the effectiveness of reminders (e.g. Christensen et al., 2015; Svensson et al., 2012). Theoretically, reminders can also be sent to fill in written diaries, but the fact that the reminder is sent to the same device where viewing is reported may increase response rates.
Still, the documentation patterns in the paper diaries group differed significantly from the patterns in the two mobile app groups: In the paper diaries group, a greater proportion completed the maximum number of views (three times a day for the 4 days of the experiment or a total of 12 viewings), compared to a much more balanced distribution among the mobile app groups. All these data support the trajectory of a significant degree of retroactive documentation in the paper diaries group, where participants retroactively documented viewings to please the experimenter (although they were not asked to do so).
As suggested by the hypotheses, this study showed that the extent of documentation declined as the experiment progressed. Documentation patterns within days were similar across all groups: The most popular hours for the documented viewing were 6:00 a.m.–1:00 p.m. for all groups.
Another interesting finding relates to the significant difference in the extent of documentation by younger and older people. Because we found no interaction effect between age and tools (paper diaries/mobile app) on extent of documentation, we can see there is no technologically driven bias. Instead, these differences apparently stem from generational differences, which arise from, among other things, the availability of free time, as found in previous studies (Broderick et al., 2003; Iida et al., 2012; Shimp and Andrews, 2013).
The findings suggest, then, that the mobile diaries are an important tool for time management studies in the social sciences and other fields, but their use requires thoughtful planning. Note the very high attrition rates of participants in the sample who were asked to download the mobile app, enter their information, and activate it. But despite issues concerning application downloading and registration, rates of response are encouraging, and documentation rates surpass those of paper diary users.
Future research should continue to validate this and similar tools, for example by examining how changes in app features, such as the ability to record audio messages or links to social networks (such as notices to contacts when viewing is documented), affect the extent of documentation. It would be interesting to compare cellular and paper diaries in other genres, such as sports and lifestyle.
Conclusion
In an age of new media, mobile applications allow ongoing, readily available and simple documentation of human behaviors. The study points to the benefits of using mobile applications, compared to paper logs, for documenting media-related behaviors. The findings demonstrate that the ease of use and mobile accessibility of the cell phone, as well as the possibility of generating reminders, make it a useful tool for documenting media exposures.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was supported by the Institute for the Study of New Media, Society and Politics in Ariel University. We thank Moriya Yaffe, Galit Madar, and especially Inbal Laks-Freund, for their assistance in planning the experiment and preparing the article for publication.
