Many Labs 5: Registered Replication of LoBue and DeLoache (2008)

Abstract

Across three studies, LoBue and DeLoache (2008) provided evidence suggesting that both young children and adults exhibit enhanced visual detection of evolutionarily relevant threat stimuli (as compared with nonthreatening stimuli). A replication of their Experiment 3, conducted by Cramblet Alvarez and Pipitone (2015) as part of the Reproducibility Project: Psychology (RP:P), demonstrated trends similar to those of the original study, but the effect sizes were smaller and not statistically significant. There were, however, some methodological differences (e.g., screen size) and sampling differences (the age of recruited children) between the original study and the RP:P replication study. Additionally, LoBue and DeLoache expressed concern over the choice of stimuli used in the RP:P replication. We sought to explore the possible moderating effects of these factors by conducting two new replications—one using the protocol from the RP:P and the other using a revised protocol. We collected data at four sites, three in Serbia and one in the United States (total N = 553). Overall, participants were not significantly faster at detecting threatening stimuli. Thus, results were not supportive of the hypothesis that visual detection of evolutionarily relevant threat stimuli is enhanced in young children. The effect from the RP:P protocol (d = −0.10, 95% confidence interval = [−1.02, 0.82]) was similar to the effect from the revised protocol (d = −0.09, 95% confidence interval = [−0.33, 0.15]), and the results from both the RP:P and the revised protocols were more similar to those found by Cramblet Alvarez and Pipitone than to those found by LoBue and DeLoache.

Many people develop fears of animals, and snakes are a particularly common target of these fears (Fredrikson, Annas, Fischer, & Wik, 1996). These fears may be, at least in part, shaped by a “prepared learning” process (Öhman & Mineka, 2001). According to the prepared-learning hypothesis, humans possess a fear module, a neural system that is selectively sensitive to evolutionarily relevant threatening stimuli. This neural system endows humans with an innate ability to detect threat-relevant stimuli (e.g., snakes) faster than nonthreatening stimuli (e.g., frogs or caterpillars). Thus, even young children, who have little or no experience with an evolutionarily relevant threatening object, should exhibit this ability.

In a series of three experiments, LoBue and DeLoache (2008) provided evidence suggesting that visual detection of evolutionarily relevant threat stimuli is enhanced not only in adults but also in young children. In all three experiments, threat-relevant stimuli (snakes) were detected more rapidly than nonthreatening stimuli (flowers, frogs, or caterpillars) by parents and their children. In Experiment 1, both adults and young children detected snakes faster than flowers. In Experiments 2 and 3, LoBue and DeLoache used nonthreatening comparison stimuli that were increasingly similar to snakes to provide a more stringent test of their hypothesis. Additionally, they recruited only 3-year-olds for the sample of young children because of their limited experience with snakes. In the second experiment, frogs were used as nonthreatening distractor stimuli because they resemble snakes in texture, color, and animacy. Finally, in the third experiment, caterpillars were used as the nonthreatening stimuli. Caterpillars possess the most salient physical characteristic of a snake (i.e., the elongated shape) and would presumably be more difficult to differentiate from a snake than frogs are. Thus, the third experiment provided the most stringent test of enhanced detection of fear-relevant stimuli.

A replication of Experiment 3, conducted by Cramblet Alvarez and Pipitone (2015) as part of the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015), demonstrated trends similar to those of the original study (i.e., a tendency of both children and adults to react faster to fear-relevant stimuli than to nonthreatening stimuli), but the effect sizes were smaller and statistically nonsignificant. There were, however, a few important differences between the original and replication studies that may account for the lack of replication. First, Cramblet Alvarez and Pipitone used tablets instead of larger touch-screen computer monitors (as in the original studies), and they reported that some children encountered difficulties using the tablets (i.e., they had difficulties selecting the stimulus they wanted to select). Second, the software for stimulus presentation was Web based and dependent on Internet speed, which was inconsistent. In addition, Cramblet Alvarez and Pipitone recruited children ages 3 to 5, in contrast to the homogeneous group of 3-year-olds used in the original experiment.

However, the feature of the replication study that may have played the largest role with regard to the failure to replicate the original effects may have been the selection of which of the three studies to replicate. As LoBue (2015) noted in her commentary on Cramblet Alvarez and Pipitone’s replication study, subsequent studies (LoBue, 2014; LoBue & DeLoache, 2011) have demonstrated that shape is an important factor in the detection of stimuli with snakelike shapes (e.g., detection of coiled wires or simple curvilinear lines is quicker than detection of noncoiled or straight lines). Thus, in the opinion of the original authors, Experiment 3 would have presented a stringent test of the prepared-learning hypothesis and might have been the most difficult to replicate. In the current research, we sought to examine the moderating effect of these features by testing the prepared-learning hypothesis using both the RP:P protocol developed by Cramblet Alvarez and Pipitone and a revised protocol.

Based on feedback from the authors of the original study regarding (a) the importance of the physical features of the stimuli in the detection of the focal effect, (b) the age of children recruited, and (c) technical issues such as screen size and software dependent on Internet speed, we created a revised protocol. The major difference between the revised protocol and the RP:P protocol used by Cramblet Alvarez and Pipitone (2015) was the choice of the control stimuli (i.e., the revised protocol used frogs, as used in Experiment 2 of the original research, instead of caterpillars, as used in the RP:P protocol). In addition, the sample for the revised protocol consisted of only 3-year-olds along with their parents (instead of 3- to 5-year-olds, as in the RP:P protocol). Finally, the study was implemented with Internet-independent software (so it could be run offline, thus avoiding problems associated with inconsistent Internet speeds) ¹ and on a larger screen, similar to those used in the original studies.

In addition to conducting a study following the procedures from the revised protocol, we conducted a separate study (using a separate sample of children and parents) using the protocol developed by Cramblet Alvarez and Pipitone (2015) for their RP:P replication. This allowed us to examine the replicability of enhanced visual detection of snakes in both protocols, to compare findings across protocols, and to identify important moderators.

Disclosures

Preregistration

The design and confirmatory analyses were preregistered on the Open Science Framework (https://osf.io/rj9hv).

Data, materials, and online resources

The registered results-blind manuscript, data set, materials, and script are available at https://osf.io/9g7pw/. Supplement 1 on the Open Science Framework (https://osf.io/wx952) contains model comparisons of the aggregated RP:P and revised protocols.

Reporting

We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.

Ethical approval

The institutional review board of the Faculty of Sport and Physical Education, University of Belgrade, approved the study in Belgrade. The ethics committee of the Department of Psychology, University of Novi Sad, approved the study in Novi Sad, and the ethics committee of the Department of Psychology, University of Niš, approved the study in Niš. The institutional review board of Loyola Marymount University approved the study in the United States. The data were collected in accordance with the 2013 Declaration of Helsinki.

Method

Power analysis and sampling

We replicated the power analysis from Cramblet Alvarez and Pipitone (2015), using the effect size for the main effect (enhanced detection of snakes compared with nonthreatening stimuli) from the original study (LoBue & DeLoache, 2008, Experiment 3; r² = .23) and α = .05. This analysis indicated that we would need 25 participants to achieve 95% power. We sought to recruit this many children and their parents for each protocol at each collection site (50 pairs per site across the two protocols). To maximize the power to detect the focal replication effect within each protocol, we planned to collect data at a minimum of three collection sites. We were able to recruit participants at preschool sites in three cities in Serbia and one city in the United States.

Participants were recruited for each protocol according to the age of the children (3 years of age for the revised protocol and 3 to 5 years of age for the RP:P protocol). Because of this restriction, participants were not randomly assigned to protocols. However, within each protocol, participants were randomly assigned to either the experimental condition (threatening stimuli as the target) or the control condition (nonthreatening stimuli as the target).

All parents provided informed consent and voluntarily consented to participate in the study together with their child. All children received a finger puppet, bubbles, or a coloring pad as compensation for their time. The parents in Serbia were not compensated for their participation, and those in the United States received a $15 gift card. After the testing procedure, the parents were individually debriefed.

RP:P-protocol sample

This sample consisted of 287 parents and children. Of these, 229 were recruited from seven preschool facilities in Serbia (Belgrade: n = 57, Niš: n = 104, Novi Sad: n = 68; 55.01% female; 100% Caucasian). The average age of children from Serbia was 52.72 months (SD = 10.69). The Serbian parent sample reported an average age of 38.05 years (SD = 7.94).

In the United States, respondents were recruited from two preschool facilities in Los Angeles (n = 58; 67.24% female; 58.6% White, 1.7% Black, 8.6% Asian American, 17.2% Latino or Hispanic, 10.3% multiracial, 3.4% other). The average age of U.S. children was 47.19 months (SD = 7.12). The U.S. parent sample reported an average age of 38.61 years (SD = 5.48).

Revised-protocol sample

This sample consisted of 266 parents and children. Of these, 209 were recruited from seven preschool facilities in Serbia (Belgrade: n = 63, Niš: n = 80, Novi Sad: n = 66; 56.46% female; 100% Caucasian). The average age of children from Serbia was 40.08 months (SD = 4.56). The Serbian parent sample reported an average age of 36.47 years (SD = 5.04).

In the United States, respondents were recruited via advertisements posted on a listserv for local mothers in Los Angeles, as well as advertisements at a local preschool (N = 57; 68.42% female; 56.1% White, 3.5% Black, 17.5% Asian American, 7.0% Latino or Hispanic, 7.1% multiracial, 8.8% other). The average age of U.S. children was 41.09 months (SD = 3.64). The U.S. parent sample reported an average age of 39.37 years (SD = 4.83).

Materials

For each stimulus category, we used 24 photographs, the same as in the original study by LoBue and DeLoache (2008). On a given trial, 9 of these photographs were displayed in a 3 × 3 matrix. Each matrix contained 1 target picture from one category and 8 distractor pictures from another category. The stimulus categories were snakes and frogs (for the revised protocol) and snakes and caterpillars (for the RP:P protocol). None of the animals were depicted in a threatening pose, and all animals were brightly colored. All snakes were depicted coiled on the ground or in trees, whereas caterpillars were in an elongated position. Each of the 24 pictures from the target category appeared as a target once (i.e., different targets appeared in each of the nine positions in the matrix two or three times). Distractor pictures appeared multiple times, but different distractors were presented approximately the same number of times across trials. More detail about which pictures were assigned to be targets and which were assigned to be distractors is provided later. For presenting the stimuli and recording the errors and reaction times, we used Matrix ² software. All materials are available at https://osf.io/tuwa8/.

In all Serbian sites, for the revised protocol, an MSI Wind Top AE2220 Full-HD color touch-screen all-in-one computer was used to present each picture matrix on a 54.9-cm (22-in.) screen. LoBue and DeLoache (2008) used a 61-cm (24-in.) screen. However, they were consulted and agreed that the 2-in. difference in screen size would likely not make a difference. For the RP:P protocol, in all Serbian sites, an HP Elitebook 2760P touch-screen laptop was used to present each picture matrix on a 30.5-cm (12-in.) ³ screen. In the United States, the data for the revised protocol were collected using an HP EliteOne 800 computer with a 55.9-cm (22-in.) Wacom Cintiq HD touch-screen, and the data for the RP:P protocol were collected using a Lenovo ThinkPad Helix 36984SU Ultrabook touch-screen tablet with a screen size of 29.5 cm (11.6 in.).

In the original study (LoBue & DeLoache, 2008), parents were asked to report on the “snake experience” (p. 286) of their child and to indicate whether their child had ever seen a live snake. Seventy-seven percent of children reportedly had experience with snakes. In the replication study done by Cramblet Alvarez and Pipitone (2015), only 16% of the children reportedly had experience with snakes. This dramatic difference could be due to differing definitions of snake experience In the replication study, visiting the zoo and seeing a snake qualified as a snake experience. To define snake experience more precisely, we added a question to the revised protocol asking parents to specify where the child had seen the snake (i.e., whether the child had seen the snake in the zoo).

Finally, Cramblet Alvarez and Pipitone (2015) added “unknown” as a response option for a question regarding the child’s fear of snakes. Because it was the parents who assessed the children’s fear, and because it seemed likely that they might not always be able to provide a “yes” or “no” answer (especially for children with no previous experience with snakes), we kept this change from the original study in our revised protocol.

Procedure

Participants in Serbia completed the study in the preschool, in a separate and quiet room, using individual computers. In the United States, the respondents were tested in a university research lab for the revised protocol and in a separate room in the preschool for the RP:P protocol. Both parents and children were asked to visually detect the target stimulus among eight distractor stimuli on each of 24 trials and to provide a response on a touch screen. Reaction times were measured. Participants in the experimental condition were instructed to find one threat-relevant target (snake) among eight threat-irrelevant distractors (frogs in the revised protocol and caterpillars in the RP:P protocol). Those in the control condition were instructed to find one threat-irrelevant target (frogs in the revised protocol and caterpillars in the RP:P protocol) among eight threat-relevant distractors (snakes).

For each child-parent pair, the child was tested first. Prior to the start of the experiment, the experimenter engaged in small talk with the child to relax him or her and to facilitate further interaction. The experimenter was seated next to the child, alongside the monitor, and instructed the child throughout the procedure. The child was seated in front of the touch-screen monitor, approximately 40 cm from the base of the screen, and told to place his or her hands on handprint cutouts located on the table immediately in front of the computer. After each trial, the child had to place his or her hands back on the handprints. This ensured that each child began each trial from the same starting position, to maximize the quality of the latency data.

First, a set of seven practice trials was conducted to teach the child how to use the touch screen. The pictures in the practice trials were randomly selected from the original set of 24 pictures. On the first two trials, a single picture appeared on the screen, and the child was asked to touch it. In the first trial, the picture was from the target category, and in the second trial, the picture was from the distractor category. On the next two trials, 1 target and 1 distractor picture were displayed, and the task was to touch the target picture. Finally, three practice trials consisted of a 9-picture matrix, and the child was asked to find the target (e.g., snake) among the distractors (e.g., frogs or caterpillars), as quickly as possible, and to touch the target before returning his or her hands to the handprint cutouts.

Following the practice trials, the child completed a series of 24 trials. In each trial, a different picture matrix, containing one target and eight distractors, was presented. Between trials, a large smiley face appeared on the screen, and the experimenter pressed the face when it was judged that the child was looking at the screen. This initiated the next trial and was done to ensure that the child was focused on the task and was looking at the screen before each trial. Latency was recorded starting from the moment when the matrix appeared on the screen until the child touched one of the pictures on the screen.

One parent of each child participated in the study and was tested after the child had completed the task. The parent was assigned to the same experimental condition as the child. The experimental procedure was the same, except there was no smiley face between the trials. After completing the experiment, the parent provided demographic information and responded to a short questionnaire assessing the following: (a) the child’s snake experience (i.e., whether the child had some experience with snakes and whether the child had seen a snake in a zoo) and (b) whether the child or parent had a fear of snakes. The parent was not present while the child was tested and was blind to the hypothesis of the study.

As already noted, children and their parents were tested either in preschools (but in a separate and quiet room) or in a research lab. For pragmatic purposes, we allowed data-collection sites to differ in this respect, as there was no reason to believe that the testing site should have any effect on study outcomes.

Data preparation

As in both the original study and the RP:P replication, only reaction times for correct responses were analyzed. To match the sample criteria from the original article (LoBue & DeLoache, 2008), we did not plan any exclusions. ⁴ However, we excluded 11 participants for whom previous experience with snakes was not reported on the questionnaire (i.e., the parent did not answer the question either positively or negatively), as this was one of the factors in our design.

Analytic strategy

LoBue and DeLoache (2008) and Cramblet Alvarez and Pipitone (2015) analyzed response latency using a 2 (target stimulus: snake vs. distractor) × 2 (participant age: children vs. adults) × 2 (snake experience: experience with snakes vs. no experience with snakes) analysis of variance (ANOVA). All factors were between participants. In this study, we built a mixed-effects model to predict latency from target stimulus, participant age, and snake experience, as well as their interactions (all treated as fixed effects), plus the three-way interaction nested within site (treated as a random effect). To test for moderation by protocol (RP:P vs. revised protocol), we added the interaction of protocol with this model as a fixed effect.

In the main text, we report the results for the sample of respondents who provided all critical measures (see note 4 for information regarding exploratory analyses on the sample with outliers excluded).

Results

Confirmatory analyses: focal replication effect

Following LoBue and DeLoache (2008), we hypothesized that fear-relevant stimuli would be visually detected faster than nonthreatening stimuli. Average reaction times for target stimuli were 5.11 s (SD = 3.93) for children and 2.43 s (SD = 2.81) for parents. Average reaction times for control stimuli were 5.14 s (SD = 3.19) for children and 2.98 s (SD = 3.36) for parents.

We first sought to create a model that best approximated the model used by LoBue and DeLoache (2008). As in the original study, this model included target stimulus, participant age, and snake experience, as well as all interactions as fixed effects (Model 1). Because the data were collected at multiple sites (unlike in the original study), we also included the three-way interaction, nested within site, as a random effect. The model failed to converge, so we respecified the model by removing random effects, beginning with prior experience with snakes (the factor least relevant for the focal hypothesis), followed by participant age and target stimulus. The model that did converge included all fixed effects, but only one random effect: the random intercept nested within site.

Following our planned analysis strategy, to examine the focal replication effect of target stimulus, we compared a model including the main effect of target stimulus (and its interactions with the other fixed effects) with a model without that main effect (and without its interactions with the other fixed effects). The removal of the focal effect from the model did not significantly reduce model fit, χ²(4, N = 545) = 3.443, p = .49, pseudo-R² = .01.

Confirmatory analyses: moderation by protocol

Participants in our study completed either the RP:P or the revised protocol. It was possible that the effect was much weaker in one protocol compared with the other, which would reduce our ability to detect an effect in the overall sample. To test for this possibility, we included protocol (along with all its interactions) as a fixed effect in the previously tested model including the main effect of target stimulus. We then conducted a likelihood ratio test to test whether the replication model was significantly improved with the addition of the protocol variable. The addition of this variable did significantly improve the model, χ²(8, N = 545) = 28.819, p < .001, pseudo-R² = .05.

Exploratory analyses: isolating the focal replication effect

The focal effect of interest in this study was the main effect of target stimulus on reaction times. In our planned analyses (just presented), this effect (and its interaction with protocol) was combined with other interaction terms in all model comparisons. Therefore, the tests failed to adequately isolate and estimate the focal replication effect. This error in our planned analyses was discovered after we began data analysis. Here, we present a series of alternative tests that better isolated the focal replication effect.

First, we tested the significance of the main effect of target stimulus in our final confirmatory model. In this model, the main effect of target stimulus did not significantly predict reaction time, b = −0.078, SE = 0.146, p = .59. In addition, the estimate of the variance explained by the random effect of site was 1.639, and the residual variance was 10.003, which indicated that the differences in individual reaction times could not be explained by the site variable.

Second, we conducted a series of model comparisons in an attempt to isolate the effect of target stimulus (see Table 1 for a summary of the models). We first created a model that included participant age, snake experience, and their interaction as fixed effects (Model 3). We compared this model with one that also included the main effect of target stimulus (Model 4). This comparison differed from our planned analyses in that it allowed us to isolate the effect of target stimulus (the focal replication effect) independent of its interactions with the other fixed effects (participant age and snake experience). The addition of target stimulus did not significantly improve the model, χ²(1, N = 545) = 1.063, p = .30, pseudo-R² < .01.

Table 1.

Overview of the Tested Models

Model	Fixed effects	Random effects
Testing the focal effect
Model 1	Target Stimulus * Participant Age * Snake Experience	Target stimulus * Participant Age * Snake Experience|Site
Model 2	Target Stimulus * Participant Age * Snake Experience	1|Site
Model 3	Participant Age * Snake Experience	1|Site
Model 4	Target stimulus + Participant Age * Snake Experience	1|Site
Testing moderation by protocol
Model 5	Target Stimulus * Participant Age * Snake Experience * Protocol	1|Site
Model 6	Target Stimulus * Participant Age * Snake Experience + protocol	1|Site
Model 7	Target Stimulus * Participant Age * Snake Experience + protocol + Target Stimulus * Protocol	1|Site
Model 8	Target Stimulus * Participant Age * Snake Experience + protocol + Participant Age * Protocol	1|Site
Testing for country differences
Model 9	Target Stimulus * Participant Age * Snake Experience + country	1|Site
Model 10	Target Stimulus * Participant Age * Snake Experience + country + Target Stimulus * Country	1|Site

Note: Asterisks (*) connecting variables indicates that both the variables’ main effects and their interaction were included in the model. All other listed variables were included as main effects only.

Exploratory analyses: isolating the interaction of the focal effect and protocol

In our planned analyses, we compared the final confirmatory model (Model 2 in Table 1) with a model that included protocol (and all its interactions) as fixed effects (Model 5). These additions did reliably improve the model, χ²(8, N = 545) = 28.819, p < .001, pseudo-R² = .05. However, this result in itself does not imply that the focal effect of target stimulus was moderated by protocol. Rather, it indicates that the addition of this interaction, along with seven other interactions and the main effect of protocol improved the model. To isolate only the interaction of protocol with target stimulus, we created two additional models (Models 6 and 7). In Model 6, we added only the fixed effect of protocol to the final confirmatory model (Model 2), and in Model 7, we included both the fixed effect of protocol and the Target Stimulus × Protocol interaction. A likelihood ratio test revealed that adding the interaction term did not significantly improve the model (i.e., Model 7 did not fit the data better than Model 6), χ²(1, N = 545) = 0.564, p = .45, pseudo-R² < .001. That is, protocol did not significantly moderate the focal replication effect of target stimulus.

To further explore the effect of protocol on the focal replication effect of target stimulus, we reran the final confirmatory model (Model 2) separately for each protocol, which allowed us to estimate the effect of target stimulus within each protocol. Average reaction times for the RP:P protocol were 4.38 s (SD = 4.68) for target stimuli and 4.83 s (SD = 4.42) for control stimuli, d = −0.10, 95% confidence interval (CI) = [−1.02, 0.82]). Average reaction times for the revised protocol were 3.13 s (SD = 1.81) for target stimuli and 3.28 s (SD = 1.67) for control stimuli, d = −0.09, 95% CI = [−0.33, 0.15]).

Target stimulus did not predict reaction time in either the RP:P protocol (b = −0.139, SE = 0.24, p = .56) or the revised protocol (b = −0.097, SE = 0.08, p = .21). Again, we additionally explored the main effect of target stimulus by comparing the model including only participant age, snake experience, and their interaction (Model 3) with models additionally including either the main effect of target stimulus only (Model 4) or the main effect of target stimulus along with all its interactions (Model 2). For the RP:P protocol, Model 4 was not better than Model 3, χ²(1, N = 286) = 0.723, p = .40, pseudo-R² < .01, and neither was Model 2, χ²(4, N = 286) = 3.243, p = .52, pseudo-R² = .02. In other words, neither the main effect of target stimulus nor the joint effects of target stimulus and its interactions were significant. The same pattern of results was obtained for the revised protocol—Model 4 was no better than Model 3, χ²(1, N = 259) = 0.902, p = .44, pseudo-R² = .01, and neither was Model 2, χ²(4, N = 259) = 1.647, p = .80, pseudo-R² < .01. ⁵

Exploratory analyses: moderation by participant age

One of the main differences between the RP:P and revised protocols, besides the choice of stimuli, was the age of the children (following the original authors’ recommendations, only 3-year-olds were recruited for the revised protocol, as opposed to 3- to 5-year-olds in the RP:P protocol). To examine the impact that this might have had on task performance, we examined the interaction of participant age (children vs. adults) and protocol. We tested this by comparing a model that included all main effects and their interactions, as well as the main effect of protocol (Model 6), with a model including all main effects and their interactions, the main effect of protocol, and the interaction of protocol and participant age (Model 8). However, a likelihood ratio test revealed that adding the Participant Age × Protocol interaction term did not improve the model, χ²(1, N = 545) = 1.430, p = .23, pseudo-R² < .01. ⁶

Exploratory analyses: additional moderation analyses and exclusion of outliers

In addition, we examined differences in results between the Serbian and U.S. samples. We were interested in exploring a potential interaction between target stimulus and country (i.e., whether the main effect of target stimulus was equal in the two countries or not). We did this by comparing a model including target stimulus, participant age, and snake experience with all their interactions and the main effect of country (Model 9) with a model in which we also included the interaction of country and target stimulus (Model 10). The latter model did not explain the data better, χ²(1, N = 545) = 0.678, p = .41, pseudo-R² < .01; in other words, adding the interaction did not improve model fit. Participants in the U.S. sample responded equally quickly to snakes and control stimuli (M = 2.99 s, SD = 1.67, for snakes and M = 2.8 s, SD = 1.39, for control stimuli), and so did participants in the Serbian sample (M = 3.99 s, SD = 4.0, for snakes and M = 4.42 s, SD = 3.75, for control stimuli). The interaction effect became marginally significant when outliers were excluded from the sample; there was a trend for the effect of target stimulus to be more pronounced in Serbian participants compared with those from the United States.

We also explored the focal effect of target stimulus separately for each data-collection site. The results, along with those of the previous studies, are displayed in Figure 1 and Table 2. Overall, the focal effect was not significant at any of the sites, and while the revised protocol did show slightly larger effect sizes, they were not significantly larger compared with the RP:P protocol.

Fig. 1.

Focal replication effect (in Pearson’s r units) across labs and protocols. The error bars indicate 95% confidence intervals. The aggregate estimates for the protocols in the current study were calculated using a random-effects meta-analysis. RP:P = Reproducibility Project: Psychology.

Table 2.

Focal Effect of Target Stimulus, Separately for the Previous Studies and Each Protocol and Data-Collection Site in the Current Study

Study, collection site, or protocol	N	r	95% CI
Previous studies
Original study (LoBue & DeLoache, 2008)	48	.48	[.26, .70]
RP:P replication (Cramblet Alvarez & Pipitone, 2015)	48	.18	[−.10, .46]
RP:P protocol
University of Belgrade	57	0	[−.26, .26]
University of Niš	104	.14	[−.04, .33]
Loyola Marymount University	58	−.15	[−.40, .11]
University of Novi Sad	68	−.04	[−.28, .20]
Revised protocol
University of Belgrade	63	.08	[−.18, .34]
University of Niš	80	0	[−.22, .22]
Loyola Marymount University	57	.02	[−.25, .28]
University of Novi Sad	66	.06	[−.18, .30]
All sites in the current project
RP:P protocol	287	.01	[−.19, .21]
Revised protocol	266	.04	[−.02, .10]
Protocols combined	553	.02	[−.05, .10]

Note: Effect sizes are scored such that positive effects are directionally consistent with the original study. CI = confidence interval; RP:P = Reproducibility Project: Psychology.

Finally, as part of exploratory follow-up analyses, we tested all of the described models and their comparisons when participants detected as outliers were excluded. Overall, the results were similar, though some analyses revealed marginally significant patterns of results. However, even in these cases, the differences in effect sizes between analyses including and excluding outliers were minimal (see Supplement 1 at https://osf.io/wx952/).

Discussion

LoBue and DeLoache (2008) tested the hypothesis that humans have an innate tendency to visually detect threat-relevant stimuli. They found evidence of enhanced visual detection of threatening stimuli in both adults and young children. In their experiments, snakes were detected faster than several nonthreatening categories of stimuli (i.e., flowers, frogs, and caterpillars). The current replication study tested whether the focal effect could be observed using a protocol from a previous replication attempt conducted as part of the RP:P and using a revised, peer-reviewed protocol.

Although the participants in the current study were overall slightly faster at detecting threatening stimuli than nonthreatening stimuli, the differences in reaction time were not significant (i.e., the results did not support the prepared-learning hypothesis). This finding could be interpreted in several ways. It is possible that the prepared-learning hypothesis is not reliable or at least not adequately understood. It is also possible that snakes are not a sufficiently relevant stimulus for examining this hypothesis. Finally, it is possible that the tested protocols do not enable valid measurement of young children’s and adults’ sensitivity to detect evolutionarily relevant threat stimuli. Future studies should explore these possibilities by using a wide range of both experimental stimuli and procedures.

Regarding the research protocol, another important question our study addressed was whether the type of protocol is a significant moderator of the focal effect size. The original study (LoBue & DeLoache, 2008) supported the prepared-learning hypothesis, whereas a previous replication study (Cramblet Alvarez & Pipitone, 2015) within the RP:P project did not confirm this finding. Therefore, in our study, we collected data using two protocols: one taken from Cramblet Alvarez and Pipitone’s (2015) RP:P study and another that was revised following the recommendations of the original authors, so as to be as similar to the original protocol as possible. However, our analyses did not indicate that the focal effect was moderated by protocol. Our evidence suggests that the previously noted deviations from the original protocol in Cramblet Alvarez and Pipitone’s study were not the primary cause for the lack of an effect. Rather, it would seem that the true population effect itself is smaller than initially suggested. As were participants in the RP:P replication project, participants in our study were slightly faster to detect threatening stimuli, but the effect sizes were noticeably smaller than those observed by LoBue and DeLoache and were not statistically significant.

As part of exploratory analyses, we examined differences between the Serbian and U.S. samples and investigated the focal effect for each collection site separately. Our results showed no significant differences in effect sizes between the Serbian and U.S. samples. Additionally, the focal effect was not significant for either protocol at any of the collection sites. This confirms our overall result that participants were not faster in responding to threatening compared with nonthreatening stimuli.

Finally, as did Cramblet Alvarez and Pipitone (2015), we observed a surprisingly high number of long latencies and errors in responding. This pattern was consistent across collection sites. Because time-based tasks are very sensitive measures and the number of errors can highly influence results, taking special care of potential outliers is recommended (Ratcliff, 1993). Therefore, we conducted exploratory analyses in which we tested differences between collection sites using only a subset of participants who were not identified as outliers on the basis of their average reaction time or their proportion of errors. This analysis revealed weak, but significant (albeit sometimes only marginally significant) differences between sites in the focal effect. Nevertheless, the combined results from Cramblet Alvarez and Pipitone and our replications do not provide strong empirical support for the prepared-learning hypothesis.

Limitations

A possible limitation to our study is that, because of the specificity of the required sample and additional technical requirements, only four laboratories were recruited to conduct replication studies. However, the sample size per site was larger than that of either the original or the RP:P study, and the overall sample size exceeded the one required to detect the original effect size.

Conclusion

The current crowdsourced replication project was an attempt to replicate a finding from LoBue and DeLoache (2008), Experiment 3, as well as to test for moderation of this effect by protocol (RP:P vs. revised). Although we observed some trends in the expected direction—the participants were faster to detect threatening, compared with nonthreatening, stimuli—the effect size was considerably smaller than in the original study and did not reach statistical significance. We did not find evidence for moderation of the effect of target stimulus by protocol. It is possible that the small effect observed in the study was due to highly sensitivity to error rates, as there was a weak effect when outliers were excluded in a follow-up analysis. Overall, our findings do not support the hypothesis of enhanced visual detection of evolutionarily relevant threat stimuli in young children.

Supplemental Material

Lazarevic_Rev_Open_Practices_Disclosure – Supplemental material for Many Labs 5: Registered Replication of LoBue and DeLoache (2008)

Supplemental material, Lazarevic_Rev_Open_Practices_Disclosure for Many Labs 5: Registered Replication of LoBue and DeLoache (2008) by Ljiljana B. Lazarević, Danka Purić, Iris Žeželj, Radomir Belopavlović, Bojana Bodroža, Marija V. Čolić, Charles R. Ebersole, Máire Ford, Ana Orlić, Ivana Pedović, Boban Petrović, Ani N. Shabazian and Darko Stojilović in Advances in Methods and Practices in Psychological Science