Two-Step Ballot Verification: Mitigating the Impact of the Hawthorne Effect on Vote-Flipping Studies

Abstract

This paper investigates the effects of the Hawthorne effect on voter behavior in ballot verification studies, specifically in the context of using ballot marking devices (BMDs) to print paper ballots. Previous studies suggested an insufficient number of voters verify the printed ballot from the BMD. This study introduces a two-step verification process aimed at reducing the Hawthorne effect’s impact, which suggests that individuals alter their behavior due to the awareness of being observed. The methodology involves direct questioning about participants’ awareness of manipulated votes and a subsequent inquiry to identify the specific contest where a vote was flipped. The findings indicate that when directly asked, a higher percentage of participants acknowledged noticing vote discrepancies, illustrating the potential influence of the Hawthorne effect in previous research methodologies. The paper recommends a vote flipping study protocol to account for the effect and to ensure the accuracy of vote flipping studies.

Introduction

Elections are critical to our democracy and must be conducted with integrity and careful attention to security. They are the cornerstone of representative government, ensuring that the will of the people is accurately reflected in the leadership and policies that guide our nations. The advent and integration of technology in elections have brought about significant changes, aimed at improving efficiency, accessibility, and security. However, these technological advancements also introduce new challenges and vulnerabilities that must be rigorously examined and addressed.

One of the notable advancements in election technology is the introduction of ballot marking devices (BMDs). These devices allow voters to make their selections electronically, which are then printed onto a paper ballot that serves as the official record. BMDs offer numerous advantages for enhancing accessibility for voters with disabilities (Morak, 2023). They enable voters to adjust font sizes, choose different languages, and utilize auditory assistance, thereby promoting greater independence and privacy for voters with disabilities.

Despite these benefits, the security of BMDs has come under scrutiny (Appel et al., 2020). The potential for malicious manipulation of these devices poses a significant threat to the integrity of elections. The study “Can Voters Detect Malicious Manipulation of Ballot Marking Devices?” highlights this concern by exploring whether voters can reliably detect if their electronically marked ballots have been altered before being printed (Bernhard et al., 2020). The findings suggest that many voters fail to notice discrepancies, raising questions about the reliability of BMDs in ensuring accurate vote recording.

This issue of security is not isolated. It is part of a broader concern about the vulnerability of election technologies to tampering and fraud. The transition to BMDs, while increasing accessibility, also necessitates rigorous security measures to prevent and detect vote manipulation. As of 2024, statistics indicate that 25.9% of registered voters live in jurisdictions where BMDs will be used exclusively for in-person voting on Election Day (The Verifier, 2024). This marks a significant increase from previous election cycles, highlighting the growing reliance on these devices and the corresponding need for enhanced security protocols.

The phenomenon of vote flipping, where a voter’s selection is altered to a different candidate or option, is a critical area of study in this context. Previous research has suggested that voters often fail to verify their printed ballots, which can result in undetected vote flipping (Bernhard et al., 2020; Kortum et al., 2021). This paper builds on such research by investigating the Hawthorne effect in voter behavior during ballot verification studies. The Hawthorne effect, where individuals modify their behavior because they know they are being observed, can skew the results of studies on voter verification practices (McCarney et al., 2007).

This study introduces a two-step verification process designed to mitigate the Hawthorne effect, providing a more accurate assessment of how voters detect manipulated votes. By directly questioning participants about their awareness of vote discrepancies and subsequently asking them to identify specific instances of vote flipping, the study reveals that direct inquiry can significantly increase the accuracy of detection of manipulated votes. The findings underscore the importance of developing robust protocols for vote flipping studies to ensure the reliability and security of election outcomes.

Related Works

The issue of voters detecting malicious manipulation of Ballot Marking Devices (BMDs) has been explored extensively in recent research. Bernhard et al. conducted a significant study in a realistic voting setting to investigate if and how voters notice errors introduced by a modified BMD (Bernhard et al., 2020). In their experiment, they introduced deliberate errors by flipping one of the voter’s choices on the paper ballot, printing a different candidate than the one selected by the voter. The study revealed that while voters have the potential to detect such anomalies, only a small percentage actively did so. Specifically, only 6.6% of the voters noticed and alerted the experimenters to the incorrect ballot, highlighting a significant gap in voter verification.

Building on Bernhard et al. (2020) work, Kortum et al. (2021) investigated the factors influencing voters’ behavior in verifying their BMD-generated ballots. Their study achieved a higher detection rate of 17.6%. Importantly, among those voters who chose to verify their ballots, 76% detected the errors. This study emphasized the potential for higher detection rates if voters are encouraged and motivated to verify their ballots. Kortum et al. (2021) also discussed possible underestimation in their detection rates but did not delve deeply into this aspect. Their recommendations focused on methods to increase voter engagement in the verification process, such as voter education and system design improvements.

Gilbert et al. (2021) also examined a voter’s ability to detect and identify anomalies with BMDs on a transparent voting machine (TVM); refer to Figures 1 and 2. This innovative system features a glass touch screen that sits in front of a printer that displays the physical paper ballot and allows voters to see the printer mark their selections in real-time as they make their choices on the touch screen. These studies collectively highlight the challenges and potential solutions for improving voter verification of BMD-generated ballots. Bernhard et al. (2020) demonstrated the low natural detection rate without intervention. Kortum et al. (2021) showed that with proper encouragement and education, detection rates could be significantly improved. Finally, Gilbert et al. (2021) introduced a transparent approach that significantly improved voter recognition of ballot anomalies while discovering the impact of the Hawthorne effect. Together, these works provide a comprehensive understanding of the current state and future directions for ensuring the integrity and reliability of electronic voting systems.

Figure 1.

Transparent voting machine: candidate selection interface as studied by Gilbert et al. (2021).

Figure 2.

Transparent voting machine: candidate verification interface as studied by Gilbert et al. (2021).

Hawthorne Effect in Deceptive Studies

During the initial transparent voting machine (TVM) study, Gilbert et al. (2021) informed participants that the study aimed to determine if there were changes to voter sentiment between the 2018 and 2020 elections. The true purpose was to observe if participants would notice if their votes were flipped using the transparent voting machine. The results showed that 36% of participants noticed when the flip occurred on the ballot and vocalized the error when they noticed it (Gilbert et al., 2021). Many of the other participants stated that the study went well and they didn’t have any issues. However, throughout the pilot study, the lead investigator noticed a physical reaction, such as a face twitch or a head tilt, in some participants when a vote was flipped. When asked how the study went, participants continued the sentiment that it went well and did not mention the flipped vote. These reactions led the investigator to ask, “Did you notice the flipped vote?” When asked directly, participants admitted to noticing the flipped vote. To ensure that the participant did not simply give an answer they felt the investigator was looking for, the investigator asked the participant if they could identify the flipped vote. A total of 41% of participants noticed the flip but did not say anything and correctly identified the flipped vote.

Therefore, 77% of the participants in the study noticed the flip and correctly identified the flip. If the experimenter did not prompt the voter with the question about their vote being flipped, the results would have yielded only 36% noticed. When asked why they did not speak up initially, a common response was that the participants did not say anything because they knew it was a study. This is considered the Hawthorne Effect, which technically says that when individuals know they are being studied, they will overachieve or over-perform (McCarney et al., 2007). However, this is generally taken in the context of studying physical activity. For example, if you are studying throwing a ball, participants will likely try to throw it as hard or far as possible. However, in a deceptive study where the user does not know the true intent, the Hawthorne effect would be at play by changing the participants’ behavior, which is unknown to the investigative team, as observed in Gilbert et al. (2021).

We found that a key factor of the Hawthorne effect was overlooked in the prior studies (Bernhard et al., 2020; Kortum et al., 2021). The researchers never accounted for the Hawthorne effect as seen in Gilbert et al. (2021); therefore, it is unknown if participants noticed their votes were flipped and simply didn’t say anything. Furthermore, because these were deceptive studies, participants could not over-perform, as defined by the Hawthorne effect, because they weren’t aware that the investigators were studying vote flips. Therefore, the Hawthorne effect in vote-flipping studies isn’t about over-performing; it’s about ignoring the flipped votes because it’s not a real election. The implication is that although the prior studies were conducted properly, their findings may be missing data as there is no way to know how many participants experienced the Hawthorne effect because they were not asked directly.

Conclusion

Vote-flipping studies are essential for protecting the integrity of future elections. By examining these issues, researchers can improve election security, ensure that every vote is counted accurately, and empower individuals to know that their participation makes a difference. These studies enhance voter confidence, identify system vulnerabilities, inform policymaking, drive technological improvements, and raise public awareness about the importance of secure and reliable voting processes. With this in mind, vote-flipping studies are encouraged. They should continue as they open opportunities for identifying and addressing vulnerabilities, ensuring that the voting process remains fair, secure, and reflective of the people’s will.

Exposing deception in vote-flipping studies is important because it ensures the accuracy and reliability of research findings, which are critical for developing effective measures to protect election integrity. Revealing the true intent of vote-flipping studies helps mitigate the Hawthorne effect in deceptive studies. By maintaining transparency, researchers can gather more precise data, leading to more accurate assessments of voting systems and the development of robust strategies to prevent vote tampering. In future studies, if deception is used, researchers must reveal the deception to the participant at an appropriate time. Therefore, we present a protocol. The recommended protocol for conducting vote-flipping studies involves gathering specific participant responses to flipped votes. After ballots have been surrendered, researchers should ask participants who do not speak up when their vote was flipped if they noticed one or more of their votes was flipped. If the participant responds “yes,” they should be asked which vote was altered and should identify the flip. This will allow the researchers to categorize each voter experiment result into one of the following categories as seen in Gilbert et al. (2021):

1. The participant noticed the vote flip and reported it without any prompting.

2. The participant did not speak up, but when prompted, said they noticed and then correctly identified the flip.

3. The participant did not speak up, but when prompted said they noticed, but could not identify the flip.

4. The participant did not speak up and said they did not notice, however, they did identify the flip.

5. The participant did not speak up and said they did not notice and could not identify the flip.

This detailed protocol could help researchers understand participant awareness and responses more accurately, which are crucial for assessing the effectiveness of voting systems and identifying potential vulnerabilities more accurately with respect to mitigating potential Hawthorne effects.

Implementing a protocol to examine participant responses to vote-flipping scenarios could mitigate the impact of the Hawthorne effect and improve future studies. Researchers can obtain more insight into human interactions with voting systems by directly asking if participants noticed and could identify flipped votes. This helps to ensure that the findings accurately reflect real-world behaviors rather than altered responses due to participants’ awareness of being observed. By melding methodological rigor with technological innovation, the recommended protocol aims to advance election integrity research while ensuring the reliability of findings.