A simulation was conducted to assess the effect of technical variance on the statistical power of web experiments measuring response times. The results of the simulation showed that technical variance reduced the statistical power and the accuracy of the effect size estimate by a negligible magnitude. This finding therefore suggests that researchers’ preconceptions concerning the unsuitability of web experiments for conducting research using response time as a dependent measure are misguided.
BirnbaumM. H. (2001). Introduction to behavioral research on the Internet. Upper Saddle River, NJ: Prentice Hall.
2.
BirnbaumM. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832.
3.
BoxG. E. P.CoxD. R. (1964). An analysis of transformations (with discussion). Journal of the Royal Statistical Society B, 26, 211–246.
4.
BrandA. (2004). A web experiment based enquiry into the verbal overshadowing effect (PhD thesis). Cardiff: Cardiff University.
5.
BrandA.BradleyM. T.BestL. A.StociaG. (2011). Multiple trials may yield exaggerated effect size estimates. The Journal of General Psychology, 138, 1–11.
6.
CohenJ. (1977). Statistical power analysis for the behavioral sciences(revised edition). New York: Academic Press
7.
CorleyM.ScheepersC. (2002). Syntactic priming in English sentence production: Categorical and latency evidence from an Internet-based study. Psychonomic Bulletin and Review, 9, 126–131.
8.
DamianM. F. (2010). Does variability in human performance outweigh imprecision in response devices such as computer keyboards?. Behavior Research Methods, 42, 205–211.
9.
GoslingS. D.VazireS.SrivastavaS.JohnO. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93–104.
10.
HeathcoteA.PopielS. J.MewhortD. J. K. (1991). Analysis of response time distributions: An example using the Stroop task. Psychological Bulletin, 109, 340–347.
11.
HechtH.OeskerM.KaiserA.CivelekH.SteckerT. (1999). A perception experiment with time-critical graphics animation on the World-Wide Web. Behavior Research Methods, Instruments and Computers, 31, 439–445.
12.
HewsonC. (2003). Conducting research on the Internet. The Psychologist, 16, 290–293.
13.
HohleR. H. (1965). Inferred components of reaction times as functions of foreperiod duration. Journal of Experimental Psychology, 69, 382–386.
14.
KellerF.GunasekharanS.MayoN.CorleyM. (2009). Timing accuracy of web experiments: A case study using the WebExp software package. Behavior Research Methods, 41, 1–12.
15.
KelleyK. (2007). Methods for the behavioral, educational, and social sciences: An R package. Behavior Research Methods, 39, 979–984.
16.
LuceR. D. (1986). Response times: Their role in inferring elementary mental organization. New York, NY: Oxford University Press.
17.
McGillW. J. (1963). Stochastic latency mechanisms. In LuceR. D.BushR. R. (Eds.), Handbook of mathematical psychology (pp. 309–360). Vol. 1, New York, NY: Wiley.
18.
McGrawK. O.TewM. D.WilliamsJ. E. (2000). The integrity of web-delivered experiments: Can you trust the data?. Psychological Science, 11, 502–506.
19.
MuschJ.ReipsU. D. (2000). A brief history of web experimenting. In BirnbaumM. H. (Ed.), Psychological experiments on the Internet (pp. 61–87). San Diego, CA: Academic Press.
20.
NosekB. A.BanajiM. R.GreenwaldA. G. (2002). Harvesting implicit group attitudes and beliefs from a demonstration website. Group Dynamics, 6, 101–115.
21.
PlantR. R.HammondN.WhitehouseT. (2003). How choice of mouse may affect response timing in psychological studies. Behavior Research Methods, 35, 276–284.
22.
PlantR. R.TurnerG. (2009). Millisecond precision psychological research in a word of commodity computers: New hardware, new problems?. Behavior Research Methods, 41, 598–614.
23.
R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved fromhttp://www.r-project.org
24.
ReimersS.MaylorE. A. (2005). Task switching across the life span: Effects of age on general and specific switch costs. Developmental Psychology, 41, 661–671.
25.
ReimersS.StewartN. (2007). Adobe Flash as a medium for online experimentation: A test of RT measurement capabilities. Behavior Research Methods, 39, 365–370.
26.
ReipsU. D. (2000). The web experiment method: Advantages, disadvantages and solutions. In BirnbaumM. H. (Ed.), Psychological experiments on the Internet (pp. 89–117). SanDiego, CA: Academic Press.
27.
ReipsU. D. (2002). Standards for Internet-based experimenting. Experimental Psychology, 49, 243–256.
28.
StasinopoulosD. M.RigbyR. A. (2007). Generalized additive models for location scale and shape (GAMLSS) in R. Journal of Statistical Software, 23, 1–46.
UlrichR.GirayM. (1989). Time resolution of clocks: Effects on reaction time measurement: Good news for bad clocks. British Journal of Mathematical & Statistical Psychology, 42, 1–12.
31.
WeinbergerJ.WestenD. (2008). RATS, we should have used Clinton: Subliminal priming in political campaigns. Political Psychology, 29, 631–651.
