A challenge of studying information-seeking behavior in open web systems is the unpredictability of those systems. One solution to counteract this issue is employing a simulation to ensure experimental control. However, concerns arise over the realism of such an environment. This paper assesses the realism of a behavioral simulation used to study the evaluation behavior of 175 students from fourth grade through graduate school. We assess realism through the examination of targeted participant feedback about what would have made the simulated environment and tasks more realistic to these participants. Based on this feedback, we reflect on decisions made in designing the simulation and offer recommendations for future studies interested in incorporating behavioral simulation in their research design. We find that a thoughtfully designed simulation can elicit naturalistic behavior when the controlled environment is designed to be realistic in meaningful ways. Because the simulation does not have to perfectly match reality to elicit these behaviors, designing a simulation that is real enough is an effective method to study information-seeking behavior.
AbelsonRP (1968) Simulation of social behavior. In: LindzeyGAronsonE (eds) The Handbook of Social Psychology, 2nd edn.Reading, MA: Addison-Wesley Pub. Co, pp.274–356.
2.
AronsonECarlsmithJM (1968) Experimentation in social psychology. In: LindzeyGAronsonE (eds) The Handbook of Social Psychology, 2nd edn.Reading, MA: Addison-Wesley Pub. Co, pp.1–79.
3.
BelkinNJ (1990) The cognitive viewpoint in information science. Journal of Information Science16(1): 11–15.
4.
BilalDGwizdkaJ (2016) Children’s eye-fixations on google search results. Proceedings of the Association for Information Science and Technology53: 1–6.
5.
BilalDGwizdkaJ (2018) Children’s query types and reformulations in Google search. Information Processing & Management54(6): 1022–1041.
6.
BlomgrenLValloHByströmK (2004) Evaluation of an information system in an information seeking process. In: HeeryRLyonL(eds) Research and Advanced Technology for Digital Libraries. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, pp.57–68.
7.
BorlundP (2000) Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation56(1): 71–90.
8.
BorlundP (2003) The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research-an International Electronic Journal8(3). Available at: http://informationr.net/ir/8-3/paper152.html (accessed 1 January 2021).
9.
BorlundPIngwersenP (1997) The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation53(3): 225–250.
10.
BrannonBBuhlerAGCataldoTT, et al. (In Press) Genre containers: Building a theoretical framework for studying formats in information behavior. Journal of the Association for Information Science and Technology.
11.
BreakwellGMFife SchawCHammondS (2000) Research Methods in Psychology, 2nd edn.London; Thousand Oaks, CA: SAGE Publications.
12.
BuhlerAGFanielIMBrannonB, et al. (2019) Container collapse and the information remix: Students’ evaluations of scientific research recast in scholarly vs. popular sources. In: Proceedings of the association of college and research libraries, Cleveland, OH, 2019, pp. 654–667. Available at: https://ufdc.ufl.edu/IR00010779/00001 (accessed 3 June 2020).
13.
CataldoTTBuhlerAGFanielIM, et al. (2020) Mixed methods data collection using simulated Google results: reflections on the methods of a point-of-selection behaviour study. Information Research: An International Electronic Journal25(4). DOI: 10.47989/irpaper881.
14.
CyrCBrannonBConnawayLS (2021) “I still go ask someone I enjoy talking to.” The use of digital and human sources by educational stage and context. Library & Information Science Research43(1): 101070.
15.
DifonzoNHantulaDABordiaP (1998) Microworlds for experimental research: Having your (control and collection) cake, and realism too. Behavior Research Methods Instruments & Computers30(2): 278–286.
16.
EricssonKASimonHA (1993) Protocol Analysis : Verbal Reports as Data. Cambridge, MA: MIT Press.
17.
GistMEHopperHDanielsD (1998) Behavioral simulation: Application and potential in management research. Organizational Research Methods1(3): 251–295.
18.
GossenTHöbelJNürnbergerA (2014) A comparative study about children’s and adults’ perception of targeted web search engines. In: Proceedings of the SIGCHI conference on human factors in computing systems, New York, NY, 26 April, pp.1821–1824. New York: Association for Computing Machinery.
19.
GravetterFJForzanoLAB (2006) Research Methods for the Behavioral Sciences, 2nd edn.Belmont, CA: Thomson/Wadsworth.
20.
HaasAUnkelJ (2017) Ranking versus reputation: Perception and effects of search result credibility. Behaviour & Information Technology36(12): 1285–1298.
21.
HalttunenKSormunenE (2000) Learning information retrieval through an educational game. Is gaming sufficient for learning?Education for Information18(4): 289–311.
22.
JonesRA (1985) Research Methods in the Social and Behavioral Sciences. Sunderland, MA: Sinauer Associates.
23.
KellyD (2009) Methods for evaluating interactive information retrieval systems with Users. now. Available at: https://ieeexplore.ieee.org/document/8187092 (accessed 13 December 2019).
24.
KellyDSugimotoCR (2013) A systematic review of interactive information retrieval evaluation studies, 1967-2006. Journal of the American Society for Information Science and Technology64(4): 745–770.
25.
LandriscinaF (2013) An introduction to simulation for learning. In: LandriscinaF (ed.) Simulation and Learning: A Model-Centered Approach. New York, NY: Springer, pp.1–12.
26.
LiuJShahC (2019) Interactive IR user study design, evaluation, and reporting. Synthesis Lectures on Information Concepts Retrieval and Services11(2): i–75.
27.
NewellTS (2010) Learning in simulations: Examining the effectiveness of information literacy instruction using Middle school students’ portfolio products. Evidence Based Library and Information Practice5(3): 20–38.
28.
NovinAMeyersE (2017) Making sense of conflicting science information: Exploring bias in the search engine result page. In: Proceedings of the 2017 conference on conference human information interaction and retrieval, New York, NY, 7 March, pp.175–184. New York: Association for Computing Machinery.
PerrusoC (2016) Undergraduates’ use of Google vs. Library resources: A four-year cohort study. College & Research Libraries77(5): 614–630.
31.
RenaudGAzzopardiL (2012) SCAMP: a tool for conducting interactive information retrieval experiments. In: Proceedings of the 4th information interaction in context symposium, New York, NY, 21 August, pp.286–289. New York: Association for Computing Machinery.
32.
SommerRSommerBB (2002) A Practical Guide to Behavioral Research : Tools and Techniques, 5th edn.New York, NY: Oxford University Press.
33.
TomsEGFreundLLiC (2004) WiIRE: The web interactive information retrieval experimentation system prototype. Information Processing & Management40(4): 655–675.
34.
UllahAAmeenK (2018) Account of methodologies and methods applied in LIS research: A systematic review. Library & Information Science Research40: 53–60.
35.
UnderwoodJKimmelSForestD, et al. (2015) Culturally relevant booktalking: Using a mixed reality simulation with preservice school librarians. School Libraries Worldwide; Edmonton21(1): 91–107.
36.
van SomerenMWBarnardYFSandbergJAC (1994) The Think Aloud Method: A Practical Approach to Modelling Cognitive Processes. London: London Academic Press.
37.
Vidal-GomelCFauquet-AlekhineP (2016) Reflections and theoretical contributions regarding trainers’ practice and simulation. In: Fauquet-AlekhinePPehuetN (eds) Simulation Training: Fundamentals and Applications Improving Professional Practice Through Simulation Training. Cham: Springer International Publishing, pp.1–30.
38.
WalshA (2009) Information literacy assessment: Where do we start?Journal of Librarianship and Information Science41(1): 19–28.
39.
WeiXZhangYGwizdkaJ (2014) YASFIIRE: Yet another system for IIR evaluation. In: Proceedings of the 5th information interaction in context symposium, New York, NY, 26 August, pp.316–319. New York: Association for Computing Machinery.
40.
WildemuthBMMarchioniniGFuX, et al. (2019) The usefulness of multimedia surrogates for making relevance judgments about digital video objects. Information Processing & Management56(6): UNSP 102091.
