More than 50 years ago, the idea of demand characteristics was introduced by Martin Orne in a widely cited American Psychologist article. Through the 1960s and the mid-1970s, numerous studies were conducted investigating the role of demand characteristics in a variety of research areas. Demand characteristics faded from researchers’ attention in the late 1970s, relegated to brief descriptions in research methods textbooks. The present article traces the origins of and battles fought over demand characteristics during its heyday. Evidence is provided that suggests demand characteristics experienced a rebirth in the 1980s and it remains a widely referenced idea up to today. Demand characteristics reflect perennial concerns about the difficulties of and limitations to doing research with humans, concerns that often surface in the periodic crises that confront psychology. The types of problems that animated the crisis of confidence associated with demand characteristics in the 1970s form one dimension of the current replication crisis. Reinterpretation of this current replication crisis and a new direction for experimental research with human subjects are derived from this review of demand characteristics.
AdairJ. G. (1973). The human subject: The social psychology of the psychological experiment. Boston, MA: Little, Brown and Company.
2.
AdairJ. G. (1977). Review of “The Human Subject in the Psychological Laboratory.”Canadian Psychological Review, 20, 109–111. 10.1037/h0081501
3.
AdairJ. G. (1978). The combined probability of 345 studies: Only half the story?Behavioral and Brain Sciences, 3, 386. 10.1017/S0140525X00075518
4.
AdairJ. G. (1982). Meaning of the situation to subjects. American Psychologist, 37, 1406–1408. 10.1037/0003-066X.37.12.1406
5.
AdairJ. G. (1984). The Hawthorne effect: A reconsideration of the methodological artifact. Journal of Applied Psychology, 69, 334–345. 10.1037/0021-9010.69.2.334
6.
AdairJ. G. (1991). Social cognition, artifact, and the passing of the so-called crisis in social psychology. Canadian Psychology, 32, 445–450. 10.1037/h0079016
7.
AdairJ. G., & FentonD. P. (1971). Subject's attitudes toward psychology as a determinant of experimental results. Canadian Journal of Behavioural Science, 3, 268–275. 10.1037/h0082268
8.
AdairJ. G., LindsayR., & CarlopioJ. (1983). Social artifact research and ethical regulations: Their impact on the teaching of experimental methods. Teaching of Psychology, 10, 159–162. 10.1207/s15328023top1003_10
9.
AdairJ. G., & SpinnerB. (1981). Subjects’ access to cognitive processes: Demand characteristics and verbal report. Journal for the Theory of Social Behaviour, 11, 31–52. 10.1111/j.1468-5914.1981.tb00021.x
10.
AllenC. T. (2004). A theory based approach for improving demand artifact assessment in advertising experiments. Journal of Advertising, 33, 63–73. 10.1080/00913367.2004.10639157
11.
ArmitageC. J. (2007). Effects of an implementation intention-based intervention on fruit consumption. Psychology & Health, 22, 917–928. 10.1080/14768320601070662
12.
ArmitageC. J. (2009). Effectiveness of experimenter-provided and self-generated implementation intentions to reduce alcohol consumption in a sample of the general population: A randomized exploratory trial. Health Psychology, 28, 545–553. 10.1037/a0015984
13.
ArmitageC. J., & ArdenM. A. (2008). How useful are the stages of change for targeting interventions? Randomized test of a brief intervention to reduce smoking. Health Psychology, 27, 789–798. 10.1037/0278-6133.27.6.789
14.
AschS. E. (1955). Opinions and social pressure. Science, 193, 31–55.
15.
BakanD. (1954). A reconsideration of the problem of introspection. Psychological Bulletin, 51, 105–118. 10.1037/h0054157
16.
BakanP. (1956). The collection and use of retrospective data. The Journal of Psychology, 41, 369–378. 10.1080/00223980.1956.9713010
17.
BaumardN., & SperberD. (2010). Weird people, yes, but also weird experiments. Behavioral and Brain Sciences, 33, 84–85. 10.1017/S0140525X10000038
18.
BemD. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407–425. 10.1037/a0021524
19.
BerkowitzL. (1968). Impulse, aggression, and the gun. Psychology Today, 2, 19–22.
20.
BerkowitzL. (1971). The “weapons effect,” demand characteristics, and the myth of the compliant subject. Journal of Personality and Social Psychology, 20, 332–338. 10.1037/h0031804
21.
BerkowitzL., & DonnersteinE. (1982). External validity is more than skin deep: Some answers to criticisms of laboratory experiments. American Psychologist, 37, 245–257. 10.1037/0003-066X.37.3.245
22.
BerkowitzL., & LePageA. (1967). Weapons as aggression eliciting stimuli. Journal of Personality and Social Psychology, 7, 202–207. 10.1037/h0025008
23.
BerkowitzL., & TroccoliB. T. (1986). An examination of the assumptions in the demand characteristics thesis: With special reference to the Velten Mood Induction Procedure. Motivation and Emotion, 10, 337–349. 10.1007/BF00992108
24.
BerndsenM., McGartyC., van der PligtJ., & SpearsR. (2001). Meaning-seeking in the illusory correlation paradigm: The active role of participants in the categorization process. British Journal of Social Psychology, 40, 209–233. 10.1348/014466601164821
25.
BishopB. J. (2007). Methodology, values and quantitative world-views in qualitative research in community psychology. Australian Community Psychologist, 19, 9–15.
26.
BlessH., & SchwarzN. (2010). Mental construal and the emergence of assimilation and contrast effects: The inclusion/exclusion model. In ZannaM. P. (), Advances in Experimental Social Psychology (Vol. 42, pp. 319–373). San Diego, CA: Academic Press. 10.1016/S0065-2601(10)42006–7
27.
BlessH., StrackF., & SchwarzN. (1993). The informative functions of research procedures: Bias and the logic of conversation. European Journal of Social Psychology, 23, 149–165. 10.1002/ejsp.2420230204
28.
BoltonT. L. (1894). Rhythm. The American Journal of Psychology, 6, 145–238. 10.2307/1410948
29.
BootW. R., & SimonsD. J. (2012). Advances in video game methods and reporting practices (but still room for improvement): A commentary on Strobach, Frensch, and Schubert (2012). Acta Psychologica, 141, 276–277. 10.1016/j.actpsy.2012.06.011
30.
BoringE. G. (1950). A history of experimental psychology. New York, NY: Appleton-Century-Crofts.
31.
BouchardT. J. (1976). Unobtrusive measures: An inventory of uses. Sociological Methods & Research, 4, 267–300. 10.1177/004912417600400301
32.
BoumansM. (2011). The two model problem in rational decision making. Rationality and Society, 23, 371–400. 10.1177/1043463111414123
33.
BrandtM. J., IjzermanH., DijksterhuisA., FarachF. J., GellerJ., Giner-SorollaR., … van ‘t VeerA. (2014). The replication recipe: What makes for a convincing replication?Journal of Experimental Social Psychology, 50, 217–224. 10.1016/j.jesp.2013.10.005
34.
BrennerE. (2000). Mood induction in children: Methodological implications and clinical implications. Review of General Psychology, 4, 264–283. 10.1037/1089–2680.4.3.264
35.
BrownS. D., FurrowD., HillD. F., GableJ. C., PorterL. P., & JacobsW. J. (2014). A duty to describe: Better the devil you know than the devil you don't. Perspectives on Psychological Science, 9, 626–640. 10.1177/1745691614551749
36.
BuschA. M., KanterJ. W., SedivyS. K., & LeonardJ. L. (2007). A follow-up analogue study on the effectiveness of the cognitive rationale. Cognitive Therapy and Research, 31, 805–815. 10.1007/s10608-007-9121-6
37.
BushmanB. J. (2013). The weapons effect. Journal of the American Medical Association Pediatrics, 167, 1094–1095.
38.
CampbellD. T., & FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. 10.1037/h0046016
39.
CarlopioJ., AdairJ. G., LindsayR. C. L., & SpinnerB. (1983). Avoiding artefact in the search for bias: The importance of assessing subjects’ perceptions of the experiment. Journal of Personality and Social Psychology, 44, 693–701. 10.1037/0022–3514.44.4.693
40.
CarlsonM., Marcus-NewhallA., & MillerN. (1990). Effects of situational aggression cues: A quantitative review. Journal of Personality and Social Psychology, 58, 622–633. 10.1037/0022-3514.58.4.622
41.
CesarioJ. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9, 40–48. 10.1177/1745691613513470
42.
ChapmanJ., & ArmitageC. J. (2010). Evidence that boosters augment the long-term impact of implementation intentions on fruit and vegetable intake. Psychology & Health, 25, 365–381. 10.1080/08870440802642148
43.
ChapmanJ., ArmitageC. J., & NormanP. (2009). Comparing implementation intention interventions in relation to young adults’ intake of fruit and vegetables. Psychology & Health, 24, 317–332. 10.1080/08870440701864538
44.
CochraneR., & DuffyJ. (1974). Psychology and the scientific method. Bulletin of the British Psychological Society, 27, 117–121.
45.
CookT. D., BeanJ. R., CalderB. J., FreyR., KrovetzM. L., & ReismanS. R. (1970). Demand characteristics and three conceptions of the frequently deceived subject. Journal of Personality and Social Psychology, 14, 185–194. 10.1037/h0028849
46.
CookT. D., & CampbellD. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally.
47.
CrabbP. B. (2006). The Bolton legacy. In HantulaD. A. (), Advances in social and organizational psychology: A tribute to Ralph Rosnow (pp. 3–24). Mahwah, NJ: Erlbaum.
48.
CreswellJ. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Thousand Oaks, CA: Sage.
49.
CriswellJ. H. (1958). The psychologist as perceiver. In TagiuriR. & PetrulloL. (), Person perception and interpersonal behavior (pp. 95–109). Stanford, CA: Stanford University Press.
50.
CushmanP. (2013). Because the rock will not read the article: A discussion of Jeremy D. Safran's critique of Irwin Z. Hoffman's “Doublethinking our way to scientific legitimacy.”Psychoanalytic Dialogues, 23, 211–224. 10.1080/10481885.2013.772478
51.
da CostaL., & RemediosR. (2014). Different methods, different results: Examining the implications of methodological divergence and implicit processes for achievement goal research. Journal of Mixed Methods Research, 8, 162–179. 10.1177/1558689813495977
52.
DanekA. H., FrapsT., von MüllerA., GrotheB., & ÖllingerM. (2014). It's a kind of magic-what self-reports can reveal about the phenomenology of insight problem solving. Frontiers in Psychology, 5, 1408. 10.3389/fpsyg.2014.01408
53.
DanzigerK. (1983). Origins and basic principles of Wundt's Volkerpsychologie. British Journal of Social Psychology, 22, 303–313. 10.1111/j.2044-8309.1983.tb00597.x
54.
DennisB. (2013). “Validity crisis” in qualitative research: Still? Movement toward a unified approach. In DennisB., CarspeckenL., & CarspeckenP. F. (), Qualitative research: A reader in philosophy, core concepts, and practice (pp. 3–37). New York, NY: Peter Lang.
55.
DurginF. H., BairdJ. A., GreenburgM., RussellR., ShaughnessyK., & WaymouthS. (2009). Who is being deceived? The experimental demands of wearing a backpack. Psychonomic Bulletin & Review, 16, 964–969. 10.3758/PBR.16.5.964
56.
DurginF. H., KleinB., SpiegelA., StrawserC. J., & WilliamsM. (2012). The social psychology of perception experiments: Hills, backpacks, glucose, and the problem of generalizability. Journal of Experimental Psychology: Human Perception and Performance, 38, 1582–1595. 10.1037/a0027805
57.
EllsworthP. C. (2011). The rise and fall of the high-impact experiment. In GonzalesM. H., TavrisC., & AronsonJ. (), The scientist and the humanist: A festschrift in honor of Elliot Aronson (pp. 79–106). New York, NY: Psychology Press.
58.
ElmsA. C. (1975). The crisis of confidence in social psychology. American Psychologist, 30, 967–976. 10.1037/0003-066X.30.10.967
59.
EngelhardtC. R., & BartholowB. D. (2013). Effects of situational cues on aggressive behavior. Social and Personality Psychology Compass, 7, 762–774. 10.1111/spc3.12067
60.
EricssonK. A., & SimonH. A. (1980). Verbal reports as data. Psychological Review, 87, 215–251. 10.1037/0033-295X.87.3.215
61.
EvansK., & BraseG. L. (2007). Assessing sex differences and similarities in mate preferences: Above and beyond demand characteristics. Journal of Social and Personal Relationships, 24, 781–791. 10.1177/0265407507081471
62.
FarberI. E. (1963). The things people say to themselves. American Psychologist, 18, 185–197. 10.1037/h0044502
FettersM. D., CurryL. A., & CreswellJ. W. (2013). Achieving integration in mixed methods designs-principles and practices. Health Services Research, 48, 2134–2156. 10.1111/1475-6773.12117
65.
FineG. A., & ElsbachK. D. (2000). Ethnography and experiment in social psychology theory building: Tactics for integrating qualitative field data with quantitative lab data. Journal of Experimental Social Psychology, 36, 51–76. 10.1006/jesp.1999.1394
66.
FriedmanN. (1967). The social nature of psychological research: The psychological experiment as a social interaction. New York, NY: Basic Books.
67.
GackenbachJ., RosieM., BownJ., & SampleT. (2011). Dream incorporation of video-game play as a function of interactivity and fidelity. Dreaming, 21, 32–50. 10.1037/a0022868
68.
GergenK. J. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26, 309–320. 10.1037/h0034436
69.
GergenM. M. (1988). Toward a feminist metatheory and methodology in the social sciences. In GergenM. (), Feminist thought and the structure of knowledge (pp. 87–104). New York, NY: New York University Press.
70.
GillihanS. J., KesslerJ., & FarahM. J. (2007). Memories affect mood: Evidence from covert experimental assignment to positive, neutral, and negative memory recall. Acta Psychologica, 125, 144–154. 10.1016/j.actpsy.2006.07.009
71.
Giner-SorollaR. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571. 10.1177/1745691612457576
72.
GoughB., & MadillA. (2012). Subjectivity in psychological science: From problem to prospect. Psychological Methods, 17, 374–384. 10.1037/a0029313
73.
GreenbergJ., & FolgerR. (1988). Controversial issues in social research methods. New York, NY: Springer-Verlag. 10.1007/978-1-4612-3756-3
74.
GreeneJ. C., CaracelliV. J., & GrahamW. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–274. 10.3102/01623737011003255
75.
GreenwoodJ. D. (1982). On the relation between laboratory experiments and social behaviour: Causal explanations and generalizations. Journal for the Theory of Social Behaviour, 12, 225–250. 10.1111/j.1468-5914.1982.tb00449.x
76.
GriceP. (1975). Logic and conversation. In ColeP. & MorganJ. (), Syntax and semantics: Volume 3. Speech acts (pp. 41–58). New York, NY: Academic Press.
77.
HainingP. (1988). The scarecrow: Fact and fable. London, U.K: Robert Hale.
78.
HansonW. E., CreswellJ. W., Plano ClarkV. L., PetskaK. S., & CreswellJ. D. (2005). Mixed methods research designs in counseling psychology. Journal of Counseling Psychology, 52, 224–235. 10.1037/0022–0167.52.2.224
79.
HarreR. (1993). Social being (2nd ed.). Cambridge, MA: Blackwell Publishers.
80.
HarreR., & SecordP. F. (1972). The explanation of social behaviour. Oxford, England: Blackwell.
81.
HasanY., BègueL., & BushmanB. J. (2013). Violent video games stress people out and make them more aggressive. Aggressive Behavior, 39, 64–70. 10.1002/ab.21454
82.
HeisenbergW. (1927). Uber den anschaulichen inhalt der quantentheoretischen kinematik und mechanic [The actual content of quantum kinematics and mechanics]. Zeitscrift fur Physik, 43, 172–198. 10.1007/BF01397280
83.
HenrichJ., HeineS. J., & NorenzayanA. (2010). The weirdest people in the world?Behavioral and Brain Sciences, 33, 61–83. 10.1017/S0140525X0999152X
84.
HenwoodK. (2004). Reinventing validity: Reflections on principles and practices from beyond the quality-quantity divide. In ToddZ., NerlichB., McKeownS., & ClarkeD. D. (), Mixing methods in psychology: The integration of qualitative and quantitative methods in theory and practice (pp. 37–58). Hove, United Kingdom: Psychology Press.
85.
HiltonD. J. (1990). Conversational processes and causal attributions. Psychological Bulletin, 107, 65–81. 10.1037/0033–2909.107.1.65
86.
HiltonD. J. (1995). The social context of reasoning: Conversational inference and rational judgment. Psychological Bulletin, 118, 248–271. 10.1037/0033-2909.118.2.248
87.
InskoC. A., & OakesW. F. (1966). Awareness and the “conditioning” of attitudes. Journal of Personality and Social Psychology, 4, 487–496. 10.1037/h0023874
88.
JacksonC. W.Jr., & PollardJ. C. (1962). Sensory deprivation and suggestion: A theoretical approach. Behavioral Science, 7, 332–342. 10.1002/bs.3830070305
89.
JohanssonP., HallL., SikströmS., TärningB., & LindA. (2006). How something can be said about telling more than we can know: On choice blindness and introspection. Consciousness and Cognition, 15, 673–692. 10.1016/j.concog.2006.09.004
90.
JohnsonR. B., OnwuegbuzieA. J., & TurnerL. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112–133. 10.1177/1558689806298224
JostJ. T., & KruglanskiA. W. (2002). The strangement of social constructionism and experimental social psychology: History of the rift and prospects for reconciliation. Personality and Social Psychology Review, 6, 168–187. 10.1207/S15327957PSPR0603_1
93.
KahnemanD., & TverskyA. (1982). On the study of statistical intuitions. Cognition, 11, 123–141. 10.1016/0010-0277(82)90022-1
94.
KazdinA. E. (2009). Foreward. In Artifacts in behavioral research: Robert Rosenthal and Ralph L. Rosnow's classic books (pp. v–vii). New York, NY: Oxford University Press.
95.
KelmanH. C. (1967). Human use of human subjects: The problem of deception in social psychological experiments. Psychological Bulletin, 67, 1–11. 10.1037/h0024072
96.
KenealyP. M. (1986). The Velten Mood Induction Procedure: A methodological review. Motivation and Emotion, 10, 315–335. 10.1007/BF00992107
97.
KhemlaniS. S., & Johnson-LairdP. N. (2012). Hidden conflicts: Explanations make inconsistencies harder to detect. Acta Psychologica, 139, 486–491. 10.1016/j.actpsy.2012.01.010
98.
KihlstromJ. F. (2014). Hypnosis and cognition. Psychology of Consciousness, 1, 139–152. 10.1037/cns0000014
99.
KihlstromJ. F., & FrankelF. H. (2000). In memoriam: Martin T. Orne, 1927–2000. International Journal of Clinical and Experimental Hypnosis, 48, 355–360. 10.1080/00207140008410365
100.
KimmelA. J. (2001). Ethical trends in marketing and psychological research. Ethics & Behavior, 11, 131–149. 10.1207/S15327019EB1102_2
101.
KleinO., DoyenS., LeysC., Magalhães de Saldanha da GamaP. A., MillerS., QuestienneL., & CleeremansA. (2012). Low hopes, high expectations: Expectancy effects and the replicability of behavioral experiments. Perspectives on Psychological Science, 7, 572–584. 10.1177/1745691612463704
102.
KrosnickJ. A., LiF., & LehmanD. R. (1990). Conversational conventions, order of information acquisition, and the effects of base rates and individuating information on social judgments. Journal of Personality and Social Psychology, 59, 1140–1152. 10.1037/0022-3514.59.6.1140
103.
KruglanskiA. W. (1975). The human subject in the psychological experiment: Fact and artifact. In BerkowitzL. (), Advances in experimental social psychology (Vol. 8, pp. 101–147). New York, NY: Academic Press.
104.
KruglanskiA. W., & EilamZ. (1974). Overcoming the ‘subject artifacts’: A critique. British Journal of Social and Clinical Psychology, 13, 9–16. 10.1111/j.2044–8260.1974.tb00870.x
105.
LanaR. E. (1969). Pretest sensitization. In RosenthalR. & RosnowR. L. (), Artifact in behavioral research (pp. 119–141). New York, NY: Academic Press.
106.
LaneyC., KaasaS. O., MorrisE. K., BerkowitzS. R., BernsteinD. M., & LoftusE. F. (2008). The Red Herring technique: A methodological response to the problem of demand characteristics. Psychological Research, 72, 362–375. 10.1007/s00426-007-0122-6
LinsenS., LeyssenM. H. R., SammartinoJ., & PalmerS. E. (2011). Aesthetic preferences in the size of images of real-world objects. Perception, 40, 291–298.
109.
LloydD. M., LewisE., PayneJ., & WilsonL. (2012). A qualitative analysis of sensory phenomena induced by perceptual deprivation. Phenomenology and the Cognitive Sciences, 11, 95–112. 10.1007/s11097-011-9233-z
110.
LyonsJ. (1964). On the psychology of the psychological experiment. In SheererC. (), Cognition: Theory, research, promise (pp. 89–109). New York, NY: Harper & Row.
111.
MacheryE. (2010). Explaining why experimental behavior varies across cultures: A missing step in ‘The weirdest people in the world?’Behavioral and Brain Sciences, 33, 101–102. 10.1017/S0140525X10000178
112.
MacNeillV., FoleyM., QuirkA., & McCambridgeJ. (2016). Shedding light on research participation effects in behaviour change trials: A qualitative study examining research participant experiences. BMC Public Health, 16, 91. 10.1186/s12889-016-2741-6
113.
MahoneyM. J. (1976). Scientist as subject: The psychological imperative. Cambridge, MA: Ballinger.
114.
MallardD., & BryantR. A. (2006). Hypnotic conflict: A brief report. International Journal of Clinical and Experimental Hypnosis, 54, 292–302. 10.1080/00207140600689496
115.
MaslingJ. (1966). Role-related behavior of the subject and psychologist and its effects upon psychological data. Nebraska Symposium on Motivation, 14, 67–103.
116.
MaxwellJ. A. (2016). Expanding the history and range of mixed methods research. Journal of Mixed Methods Research, 10, 12–27. 10.1177/1558689815571132
117.
McCambridgeJ. (2015). From question-behaviour effects in trials to the social psychology of research participation. Psychology & Health, 30, 72–84. 10.1080/08870446.2014.953527
118.
McCambridgeJ., de BruinM., & WittonJ. (2012). The effects of demand characteristics on research participant behaviours in non-laboratory settings: A systematic review. PLoS ONE, 7, e39116. 10.1371/journal.pone.0039116
119.
McCambridgeJ., KypriK., & ElbourneD. (2014). Research participation effects: A skeleton in the methodological cupboard. Journal of Clinical Epidemiology, 67, 845–849. 10.1016/j.jclinepi.2014.03.002
120.
McFallR. (1971). Demand characteristics, experimental reactivity, and ecological validity. Paper prepared for the symposium “The role of expectancy and demand characteristics in behavior modification.” American Psychological Association Convention, Washington, DC.
121.
McGinleyH., & LaytonB. (1973). Conditioning of evaluative meaning without contingency or demand awareness. Journal of General Psychology, 89, 81–90. 10.1080/00221309.1973.9710820
122.
McGuireW. J. (1969). Suspiciousness of experimenter's intent. In RosenthalR. & RosnowR. L. (), Artifact in behavioral research (pp. 13–57). New York, NY: Academic Press.
123.
MeehlP. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834. 10.1037/0022-006X.46.4.806
124.
MilgramS. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378. 10.1037/h0040525
125.
MilgramS. (1972). Interpreting obedience: Error and evidence. A reply to Orne and Holland. In MillerA. G. (), The social psychology of psychological research (pp. 138–154). New York, NY: Free Press.
126.
MorawskiJ. (2015). Epistemological dizziness in the psychology laboratory: Lively subjects, anxious experimenters, and experimental relations, 1950–1970. Isis, 106, 567–597. 10.1086/683411
127.
NewellB. R., & ShanksD. R. (2014). Unconscious influences on decision making: A critical review. Behavioral and Brain Sciences, 37, 1–19. 10.1017/S0140525X12003214
128.
NicholsA. L., & EdlundJ. E. (2015). Practicing what we preach (and sometimes study): Methodological issues in experimental laboratory research. Review of General Psychology, 19, 191–202. 10.1037/gpr0000027
129.
NisbettR. E. (1978). A guide for reviewers: Editorial hardball in the ‘70s. American Psychologist, 33, 519–520. 10.1037/0003-066X.33.5.519
130.
NisbettR. E., & WilsonT. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259. 10.1037/0033-295X.84.3.231
131.
NoreenS., & MacleodM. D. (2013). It's all in the detail: Intentional forgetting of autobiographical memories using the autobiographical think/no-think task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 375–393. 10.1037/a0028888
132.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. 10.1126/science.aac4716
133.
OrneM. T. (1957). The nature of hypnosis, artifact and essence: An experimental study. Dissertation Abstracts International: B. The Sciences and Engineering, 67, 1207.
134.
OrneM. T. (1959a). The nature of hypnosis: Artifact and essence. The Journal of Abnormal and Social Psychology, 58, 277–299. 10.1037/h0046128
135.
OrneM. T. (1959b). The demand characteristics of an experimental design and their implications. Paper read at the symposium, The Problem of Experimenter Bias, American Psychological Association, Cincinnati, OH, September, 1959.
136.
OrneM. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776–783. 10.1037/h0043424
137.
OrneM. T. (1969). Demand characteristics and the concept of quasi-controls. In RosenthalR. & RosnowR. L. (), Artifacts in social research (pp. 143–179). New York, NY: Academic Press.
138.
OrneM. T. (1970). Hypnosis, motivation, and the ecological validity of the psychological experiment. In ArnoldW. J. & PageM. M. (), Nebraska Symposium on Motivation (Vol. 18, pp. 187–265). Lincoln, NE: University of Nebraska Press.
139.
OrneM. T. (1973). Communication by the total experimental situation: Why it is important, how it is evaluated, and its significance for the ecological validity of findings. In PlinerP., KramesL., & AllowayT. (), Communication and affect (pp. 157–191). New York, NY: Academic Press. 10.1016/B978-0-12-558250-6.50014-6
OrneM. T. (1979b). The use and misuse of hypnosis in court. International Journal of Clinical and Experimental Hypnosis, 27, 311–341. 10.1080/00207147908407571
142.
OrneM. T. (1981). The significance of unwitting cues for experimental outcomes: Toward a pragmatic approach. Annals of the New York Academy of Sciences, 364, 152–159. 10.1111/j.1749-6632.1981.tb34469.x
143.
OrneM. T., & HollandC. C. (1968). Some conditions of obedience and disobedience to authority. On the ecological validity of laboratory deceptions. International Journal of Psychiatry, 6, 282–293.
144.
OrneM. T., & ScheibeK. E. (1964). The contribution of nondeprivation factors in the production of sensory deprivation effects: The psychology of the “panic button.”The Journal of Abnormal and Social Psychology, 68, 3–12. 10.1037/h0048803
145.
PageM. M. (1969). Social psychology of a classical conditioning of attitudes experiment. Journal of Personality and Social Psychology, 11, 177–186. 10.1037/h0027025
146.
PageM. M. (1971). Postexperimental assessment of awareness in attitude conditioning. Educational and Psychological Measurement, 31, 891–906. 10.1177/001316447103100411
147.
PageM. M. (1973). On detecting demand awareness by postexperimental questionnaire. The Journal of Social Psychology, 91, 305–323. 10.1080/00224545.1973.9923053
148.
PageM. M. (1974). Demand characteristics and the classical conditioning of attitudes experiment. Journal of Personality and Social Psychology, 30, 468–476. 10.1037/h0037036
149.
PageM. M. (1981). Demand compliance in laboratory experiments. Impression management theory and social psychological research. In TedeschiJ. (), Impression management theory and social psychological research (pp. 57–82). New York, NY: Academic Press. 10.1016/B978-0-12-685180-9.50009-9
150.
PageM. M., & LumiaA. R. (1968). Cooperation with demand characteristics and the bimodal distribution of verbal conditioning data. Psychonomic Science, 12, 243–244. 10.3758/BF03331291
151.
PageM. M., & ScheidtR. J. (1971). The elusive weapons effect: Demand awareness, evaluation apprehension, and slightly sophisticated subjects. Journal of Personality and Social Psychology, 20, 304–318. 10.1037/h0031806
152.
ParkerI. (1989). The crisis in modern social psychology, and how to end it. London, United Kingdom: Routledge.
153.
PashlerH., & WagenmakersE.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?Perspectives on Psychological Science, 7, 528–530. 10.1177/1745691612465253
154.
PierceA. H. (1908). The subconscious again. The Journal of Philosophy, Psychology and Scientific Methods, 5, 264–271. 10.2307/2011773
155.
Plano ClarkV. L., SchumacherK., WestC., EdringtonJ., DunnL. B., HarzstarkA., … MiaskowskiC. (2013). Practices for embedding an interpretive qualitative approach within a randomized clinical trial. Journal of Mixed Methods Research, 7, 219–242. 10.1177/1558689812474372
156.
PostmanL., & JarrettR. F. (1952). An experimental analysis of ‘learning without awareness’. The American Journal of Psychology, 65, 244–255. 10.2307/1418358
157.
ProffittD. R. (2008). Affordances matter in geographical slant perception. Psychonomic Bulletin and Review, 16, 970–972. 10.3758/PBR.16.5.970
158.
RieckenH. W. (1962). A program for research on experiments in social psychology. In WashburneN. W. (), Decisions, values and groups (Vol. 2, pp. 25–41). New York, NY: Pergamon Press.
159.
RigerS. (1992). Epistemological debates, feminist voices: Science, social values, and the study of women. American Psychologist, 47, 730–740. 10.1037/0003-066X.47.6.730
160.
RosenbergM. J. (1969). The conditions and consequences of evaluation apprehension. In RosenthalR. & RosnowR. L. (), Artifact in behavioral research (pp. 280–350). New York, NY: Academic Press.
161.
RosenthalR. (1964). Experimenter effects in behavioral research. New York, NY: Appleton-Century-Croft.
162.
RosenthalR., & RubinD. B. (1978). Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 1, 377–386. 10.1017/S0140525X00075506
163.
RosenzweigS. (1933). The experimental situation as a psychological problem. Psychological Review, 40, 337–354. 10.1037/h0074916
164.
RosnowR. L., & AikenL. S. (1973). Mediation of artifacts in behavioral research. Journal of Experimental Social Psychology, 9, 181–201. 10.1016/0022-1031(73)90009-7
165.
RosnowR. L., & DavisD. J. (1977). Demand characteristics and the psychological experiment. Et Cetera: A Review of General Semantics, 34, 301–313.
166.
RosnowR. L., & Rosenthal (1997). People studying people: Artifacts and ethics in behavioral research. New York, NY: Freeman and Company.
167.
RubinM., & BadeaC. (2007). Why do people perceive ingroup homogeneity on ingroup traits and outgroup homogeneity on outgroup traits?Personality and Social Psychology Bulletin, 33, 31–42. 10.1177/0146167206293190
168.
SchwarzN. (1996). Cognition and communication: Judgment biases, research methods, and the logic of conversation. Mahwah, NJ: Erlbaum.
169.
SchwarzN. (2010). Measurement as cooperative communication: What research participants learn from questionnaires. In WalfordG., TuckerE., & ViswanathanM. (), The SAGE handbook of measurement (pp. 43–60). Los Angeles, CA: Sage. 10.4135/9781446268230.n4
170.
SchwarzN., & StrackF. (2014). Does merely going through the same moves make for a ‘direct’ replication? Concepts, contexts, and operationalizations. Social Psychology, 45, 305–306.
171.
SchwarzN., StrackF., HiltonD. J., & NadererG. (1991). Base rates, representativeness, and the logic of conversation: The contextual relevance of ‘irrelevant’ information. Social Cognition, 9, 67–84. 10.1521/soco.1991.9.1.67
172.
SharpeD., & FayeC. (2009). A second look at debriefing practices: Madness in our method?Ethics & Behavior, 19, 432–447. 10.1080/10508420903035455
173.
ShimpT. A., HyattE. M., & SnyderD. J. (1991). A critical appraisal of demand artifacts in consumer research. Journal of Consumer Research, 18, 273–283. 10.1086/209259
174.
ShulmanA. D., & SilvermanI. (1972). Profile of social psychology: A preliminary application of “reference analysis.”Journal of the History of the Behavioral Sciences, 8, 232–236. 10.1002/1520-6696(197204)8:2<232::AID-JHBS2300080210>3.0.CO;2-D
175.
ShwederR. A. (2010). Donald Campbell's doubt: Cultural difference or failure of communication?Behavioral and Brain Sciences, 33, 109–110. 10.1017/S0140525X10000245
176.
SigallH., AronsonE., & Van HooseT. (1970). The cooperative subject: Myth or reality?Journal of Experimental Social Psychology, 6, 1–10. 10.1016/0022–1031(70)90072–7
177.
SilvermanI. (1977). The human subject in the psychological experiment. New York, NY: Pergamon Press.
178.
SilvermanI., & ShulmanA. D. (1970). A conceptual model of artifact in artifact change studies. Sociometry, 33, 97–107. 10.2307/2786274
179.
SmithD. M., SchwarzN., RobertsT. R., & UbelP. A. (2006). Why are you calling me? How study introductions change response patterns. Quality of Life Research, 15, 621–630. 10.1007/s11136-005–4529-5
180.
SmithE. R., & MillerF. D. (1978). Limits on perception of cognitive processes: A reply to Nisbett and Wilson. Psychological Review, 85, 355–362. 10.1037/0033-295X.85.4.355
181.
SmithN. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25, 970–975. 10.1037/h0029774
182.
StaatsC. K., & StaatsA. W. (1957). Meaning established by classical conditioning. Journal of Experimental Psychology, 54, 74–80. 10.1037/h0047716
183.
StarS. L., & GersonE. M. (1987). The management and dynamics of anomalies in scientific work. The Sociological Quarterly, 28, 147–169. 10.1111/j.1533-8525.1987.tb00288.x
184.
StrackF., MartinL. L., & SchwarzN. (1988). Priming and communication: Social determinants of information use in judgments of life satisfaction. European Journal of Social Psychology, 18, 429–442. 10.1002/ejsp.2420180505
185.
StrohmetzD. B. (2006). Rebuilding the ship at sea: Coping with artifacts in behavioral research. In HantulaD. (), Advances in social and organizational psychology: A tribute to Ralph Rosnow (pp. 93–112). Mahwah, NJ: Erlbaum.
186.
StrohmetzD. B. (2008). Research artifacts and the social psychology of psychological experiments. Social and Personality Psychology Compass, 2, 861–877. 10.1111/j.1751-9004.2007.00072.x
187.
StrubeM. J. (1989). Assessing subjects’ construal of the laboratory situation. In SchneidermanN., WeissS. M., & KaufmannP. G. (), Handbook of research methods in cardiovascular behavioral medicine (pp. 527–542). New York, NY: Plenum Press. 10.1007/978-1-4899-0906-0_33
188.
SulsJ. M., & RosnowR. L. (1988). Concerns about artifact in psychological experiments. In MorawskiJ. G. (), The rise of experimentation in American Psychology (pp. 163–187). New Haven, CT: Yale University Press.
189.
SweldensS., CorneilleO., & YzerbytV. (2014). The role of awareness in attitude formation through evaluative conditioning. Personality and Social Psychology Review, 18, 187–209. 10.1177/1088868314527832
190.
TaftR. T. (1957). Experimental set as a variable. American Psychologist, 12, 339–340. 10.1037/h0038978
191.
TavrisC. A. (2014). Teaching contentious classics: What should we do when the classics had flaws that would not permit their replication today, but yielded findings that still tell a good story?Observer, 27, 8.
192.
ThorndikeE., & RockR. T. (1934). Learning without awareness of what is being learned or intent to learn it. Journal of Experimental Psychology, 17, 1–19. 10.1037/h0073815
193.
TurnerC. W., LaytonJ. F., & SimonsL. S. (1975). Naturalistic studies of aggressive behavior: Aggressive stimuli, victim visibility, and horn honking. Journal of Personality and Social Psychology, 31, 1098–1107. 10.1037/h0076960
194.
TverskyA., & KahnemanD. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458. 10.1126/science.7455683
195.
TysonN. G. (2016). What science is and how and why it works. Skeptical Inquirer, 40, 38–39.
196.
Van BavelJ. J., Mende-SiedleckiP., BradyW. J., & ReineroD. A. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences of the United States of America, 113, 6454–6459. 10.1073/pnas.1521897113
197.
VeltenE.Jr. (1968). A laboratory task for induction of mood states. Behaviour Research and Therapy, 6, 473–482. 10.1016/0005-7967(68)90028-4
198.
WeberS. J., & CookT. D. (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics and valid inference. Psychological Bulletin, 77, 273–295. 10.1037/h0032351
WertzF. J. (2014). Qualitative inquiry in the history of psychology. Qualitative Psychology, 1, 4–16. 10.1037/qup0000007
201.
WhiteR. W. (1937). Prediction of hypnotic suggestibility from a knowledge of subjects’ attitudes. The Journal of Psychology, 3, 265–277. 10.1080/00223980.1937.9917497
202.
WhittemoreR., ChaseS. K., & MandleC. L. (2001). Validity in qualitative research. Qualitative Health Research, 11, 522–537. 10.1177/104973201129119299
203.
WillisG. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage. 10.4135/9781412983655
204.
WilsonT. D. (2005). The message is the method: Celebrating and exporting the experimental approach. Psychological Inquiry, 16, 185–193. 10.1207/s15327965pli1604_09
205.
WintersA. (2010). Martin Orne and the psychology of the psychological experiment. Paper presented at the Max Planck-University of Chicago workshop Human Science, Human Subjects, April 2nd, 2010.
ZannaM. P., KieslerC. A., & PilkonisP. A. (1970). Positive and negative attitudinal affect established by classical conditioning. Journal of Personality and Social Psychology, 14, 321–328. 10.1037/h0028991
