Assessing Realism in Simulated Work Tasks
Authors: 
Anita Crescenzi, Lan LiAuthors Info & Claims
Anita Crescenzi, Lan LiAuthors Info & ClaimsPages 266 - 271
Abstract
In this paper, we describe our use of realism check questions in an exit questionnaire as a method for assessing the realism of scenarios and simulated tasks. We used realism check questionnaire items found in the decision-making literature to assess the realism of the scenario and tasks (i.e., simulated work task scenarios) used as part of a broader study of decision-making. The realism check results indicated that participants could imagine themselves in the scenario (i.e., making recommendations for a friend) and could imagine making recommendations like the ones in the study. The findings suggest that realism check questions are a relatively low effort method to assess the realism of simulated work task scenarios used in an interactive information retrieval study. Future research is needed to understand the relationship between realism and other important factors in IIR studies (e.g., topic interest or prior knowledge, search behaviors).
References
[1]
Roy F. Baumeister and Kathleen D. Vohs. 2007. Ecological validity. Vol. 1. SAGE Publications, Inc., 276–276. https://doi.org/10.4135/9781412956253.n167
[2]
Roy F. Baumeister and Kathleen D. Vohs. 2007. Experimental realism. Vol. 1. SAGE Publications, Inc., 329–330. https://doi.org/10.4135/9781412956253.n202
[3]
Roy F. Baumeister and Kathleen D. Vohs. 2007. Mundane realism. Vol. 1. SAGE Publications, Inc., 599–600. https://doi.org/10.4135/9781412956253.n357
[4]
Pia Borlund. 2003. The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research 8, 3 (2003), 1–34. http://informationr.net/ir/8-3/paper152.html
[5]
Pia Borlund. 2016. A study of the use of simulated work task situations in interactive information retrieval evaluations. Journal of Documentation 72, 3 (2016), 394–413. https://doi.org/10.1108/jd-06-2015-0068
[6]
Robert Capra, Jaime Arguello, and Yinglong Zhang. 2017. The effects of search task determinability on search behavior. In Advances in Information Retrieval. ECIR 2017., Joemon M Jose, Claudia Hauff, I Sengor Altingovde, Dawei Song, Dyaa Albakour, Stuart Watt, and John Tait (Eds.). Lecture Notes in Computer Science, Vol. 10193. Springer International Publishing, 108–121. https://doi.org/10.1007/978-3-319-56608-5_9
[7]
Anita Crescenzi. 2019. Adaptation in information search and decision-making under time pressure. Ph.D. Dissertation. https://doi.org/10.17615/YT6K-AC37
[8]
Anita Crescenzi, Rob Capra, Bogeum Choi, and Yuan Li. 2021. Adaptation in information search and decision-making under time constraints. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval(CHIIR ’21). Association for Computing Machinery, New York, NY, USA, 95–105. https://doi.org/10.1145/3406522.3446030
[9]
Pratibha A. Dabholkar. 1994. Incorporating choice into an attitudinal framework: Analyzing models of mental comparison processes. Journal of Consumer Research 21, 1 (1994), 100. https://doi.org/10.1086/209385
[10]
Diane Kelly, Jaime Arguello, Ashlee Edwards, and Wan-Ching Wu. 2015. Development and evaluation of search tasks for IIR experiments using a cognitive complexity framework. In Proceedings of the 2015 International Conference on Theory of Information Retrieval(ICTIR ’15). ACM Press, New York, 101–110. https://doi.org/10.1145/2808194.2809465
[11]
Diane Kelly and Cassidy R. Sugimoto. 2013. A systematic review of interactive information retrieval evaluation studies, 1967-2006. Journal of the American Society for Information Science and Technology 64, 4 (2013), 745–770. https://doi.org/10.1002/asi.22799
[12]
William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company, New York. 623 pages.
[13]
Elaine G. Toms, Robert Villa, and Lori McCay-Peet. 2013. How is a search system used in work task completion?Journal of Information Science 39, 1 (2013), 15–25. https://doi.org/10.1177/0165551512469929
[14]
Barbara Wildemuth and Luanne Freund. 2009. Search tasks and their role in studies of search behaviors. In Proceedings of the Third Workshop on Human-Computer Interaction and Information Retrieval (HCIR ’09)(Washington, DC).
[15]
Barbara M. Wildemuth, Luanne Freund, and Elaine G. Toms. [n.d.]. Repository of Assigned Search Tasks (RepAST). https://ils.unc.edu/searchtasks/
[16]
Timothy D. Wilson, Elliot Aronson, and Kevin Carlsmith. 2010. The art of laboratory experimentation.John Wiley & Sons, Inc., 51–81. https://doi.org/10.1002/9780470561119.socpsy001002
Index Terms
- Assessing Realism in Simulated Work Tasks
Index terms have been assigned to the content through auto-classification.
Recommendations
Interactive retrieval using simulated versus real work task situations: differences in sub-facets of tasks and interaction performance
ASIST '13: Proceedings of the 76th ASIS&T Annual Meeting: Beyond the Cloud: Rethinking Information BoundariesTask design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and a simulated work task situation design are used most widely, especially the ...
WalkingVibe: Reducing Virtual Reality Sickness and Improving Realism while Walking in VR using Unobtrusive Head-mounted Vibrotactile Feedback
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsVirtual Reality (VR) sickness is common with symptoms such as headaches, nausea, and disorientation, and is a major barrier to using VR. We propose WalkingVibe, which applies unobtrusive vibrotactile feedback for VR walking experiences, and also reduces ...
Captivating the Senses: Crafting a Multisensory Virtual Experience for Enhanced Realism
UbiComp/ISWC '23 Adjunct: Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable ComputingThis paper explores the challenges and opportunities in achieving a high level of realism and presence in Virtual Reality (VR) experiences. Realism and presence, which refer to the level of authenticity of the virtual environment and the user’s ...
Comments
Information & Contributors
Information
Published In
March 2022
399 pages
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 March 2022
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Conference
CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval
March 14 - 18, 2022
Regensburg, Germany
Acceptance Rates
Overall Acceptance Rate 55 of 163 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 172Total Downloads
- Downloads (Last 12 months)51
- Downloads (Last 6 weeks)3
Reflects downloads up to 23 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format