Better Understanding Procedural Search Tasks: Perceptions, Behaviors, and Challenges
Article No.: 65, Pages 1 - 32
Abstract
People often search for information to acquire procedural knowledge–“how to” knowledge about step-by-step procedures, methods, algorithms, techniques, heuristics, and skills. A procedural search task might involve implementing a solution to a problem, evaluating different approaches to a problem, and brainstorming on the types of problems that can be solved with a specific resource. We report on a study (N=36) that aimed to better understand how people search for procedural knowledge. Much research has investigated how search task characteristics impact people’s perceptions and behaviors. Along these lines, we manipulated procedural search tasks along two orthogonal dimensions: product and goal. The product dimension relates to the main outcome of the task and the goal dimension relates to task’s success criteria. We manipulated tasks across three product categories and two goal categories. The study investigated four research questions. First, we examined the effects of the product and goal on participants’ (RQ1) pre-task perceptions, (RQ2) post-task perceptions, and (RQ3) search behaviors. Second, regardless of the task product and goal, by analyzing participants’ think-aloud comments and screen activities we closely examined how people search for procedural knowledge. Specifically, we report on (RQ4) important relevance criteria, types of information sought, and challenges.
References
[1]
Eyob N. Alemu and Jianbin Huang. 2020. HealthAid: Extracting domain targeted high precision procedural knowledge from on-line communities. Information Processing & Management 57, 6 (2020). DOI:
[2]
Lorin W. Anderson, David R. Krathwohl, Peter W. Airasian, Kathleen A. Cruikshank, Richard E. Mayer, Paul R. Pintrich, James Raths, and Merlin C. Wittrock. 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, Complete Edition.
[3]
Peter Bailey and Li Jiang. 2012. User task understanding: A web search engine perspective. (2012). https://www.microsoft.com/en-us/research/publication/user-task-understanding-a-web-search-engine-perspective/Presentation delivered at the NII Shonan: Whole-Session Evaluation of Interactive Information Retrieval Systems workshop. 8-11 October 2012, Shonan, Japan.
[4]
Panos Balatsoukas and Ian Ruthven. 2012. An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine. Journal of the American Society for Information Science and Technology 63, 9 (2012), 1728–1746.
[5]
Carol L. Barry. 1994. User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science 45, 3 (1994), 149–159.
[6]
Katriina Byström and Kalervo Järvelin. 1995. Task complexity affects information seeking and use. Information Processing and Management 31, 2 (1995), 191–213.
[7]
Donald J. Campbell. 1988. Task complexity: A review and analysis. The Academy of Management Review 13, 1 (1988), 40–52.
[8]
Robert Capra and Jaime Arguello. 2023. How does AI chat change search behaviors? arXiv (2023). https://arxiv.org/abs/2307.03826
[9]
Robert Capra, Jaime Arguello, Anita Crescenzi, and Emily Vardell. 2015. Differences in the use of search assistance for tasks of varying complexity. Association for Computing Machinery, New York, NY, USA, 23–32.
[10]
M. Ariel Cascio, Eunlye Lee, Nicole Vaudrin, and Darcy A. Freedman. 2019. A team-based approach to open coding: Considerations for creating intercoder consensus. Field Methods 31, 2 (2019), 116–130.
[11]
Bogeum Choi and Jaime Arguello. 2020. A qualitative analysis of the effects of task complexity on the functional role of information. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. 328–332.
[12]
Bogeum Choi, Jaime Arguello, and Robert Capra. 2023. Understanding procedural search tasks “in the wild”. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (CHIIR’23). Association for Computing Machinery, New York, NY, USA, 24–33. DOI:
[13]
Bogeum Choi, Sarah Casteel, Robert Capra, and Jaime Arguello. 2022. Procedural knowledge search by intelligence analysts(CHIIR ’22). Association for Computing Machinery, New York, NY, USA, 169–179.
[14]
Bogeum Choi, Austin Ward, Yuan Li, Jaime Arguello, and Robert Capra. 2019. The effects of task complexity on the use of different types of information in a search assistance tool. ACM Trans. Inf. Syst. 38, 1, Article 9 (Dec2019), 28 pages. DOI:
[15]
Cuong Xuan Chu, Niket Tandon, and Gerhard Weikum. 2017. Distilling task knowledge from how-to communities. In Proceedings of the 26th International Conference on World Wide Web (WWW ’17). International World Wide Web Conferences Steering Committee, 805–814. DOI:
[16]
Michael J. Cole, Jacek Gwizdka, Chang Liu, Ralf Bierig, Nicholas J. Belkin, and Xiangmin Zhang. 2011. Task and user effects on reading patterns in information search. Interacting with Computers 23, 4 (052011), 346–362. DOI:
[17]
Abe Crystal and Jane Greenberg. 2006. Relevance criteria identified by health information users during Web searches. Journal of the American Society for Information Science and Technology 57, 10 (2006), 1368–1382.
[18]
Carsten Eickhoff, Jaime Teevan, Ryen White, and Susan Dumais. 2014. Lessons from the Journey: A Query Log Analysis of Within-Session Learning. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (WSDM ’14). ACM, New York, NY, USA, 223–232. DOI:
[19]
Elsa Eiriksdottir and Richard Catrambone. 2011. Procedural instructions, principles, and examples: How to structure instructions for procedural tasks to enhance performance, learning, and transfer. Human Factors 53, 6 (2011), 749–770.
[20]
Satu Elo and Helvi Kyngäs. 2008. The qualitative content analysis process. Journal of Advanced Nursing 62, 1 (2008), 107–115.
[21]
Bernhard Ertl. 2009. Conceptual and procedural knowledge construction in computer supported collaborative learning. In Proceedings of the 9th International Conference on Computer Supported Collaborative Learning (CSCL’09). International Society of the Learning Sciences, 137–141.
[22]
Luanne Freund, Elaine G. Toms, and Julie Waterhouse. 2005. Modeling the information behaviour of software engineers using a work-task framework. Proceedings of the American Society for Information Science and Technology (2005).
[23]
Alexander Frummet, David Elsweiler, and Bernd Ludwig. 2022. “What can I cook with these ingredients?” — understanding cooking-related information needs in conversational search. ACM Transactions of Information Systems 40, 4, Article 81 (2022).
[24]
M. P. Georgeff and A. L. Lansky. 1986. Procedural knowledge. Proc. IEEE 74, 10 (1986), 1383–1398. DOI:
[25]
Sara M. González-Betancor, Alicia Bolívar-Cruz, and Domingo Verano-Tacoronte. 2019. Self-assessment accuracy in higher education: The influence of gender and performance of university students. Active Learning in Higher Education 20, 2 (July2019), 101–114. Publisher: SAGE Publications.
[26]
Diane Kelly, Jaime Arguello, Ashlee Edwards, and Wan-ching Wu. 2015. Development and evaluation of search tasks for IIR experiments using a cognitive complexity framework. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval (ICTIR ’15). Association for Computing Machinery, New York, NY, USA, 101–110.
[27]
Helvi Kyngäs. 2020. Inductive content analysis. The Application of Content Analysis in Nursing Science Research (2020), 13–21.
[28]
Jo-Anne LeFevre and Peter Dixon. 1986. Do written instructions need examples? Cognition and Instruction 3, 1 (1986), 1–30.
[29]
Yuelin Li and Nicholas J. Belkin. 2008. A faceted approach to conceptualizing tasks in information seeking. Information Processing & Management 44, 6 (2008), 1822–1837.
[30]
Jingjing Liu, Michael J. Cole, Chang Liu, Ralf Bierig, Jacek Gwizdka, Nicholas J. Belkin, Jun Zhang, and Xiangmin Zhang. 2010. Search behaviors in different task types. In Proceedings of the 10th Annual Joint Conference on Digital Libraries (JCDL ’10). Association for Computing Machinery, New York, NY, USA, 69–78.
[31]
Jingjing Liu, Chang Liu, and Nicholas Belkin. 2013. Examining the effects of task topic familiarity on searchers’ behaviors in different task types. In Proceedings of the 76th ASIS&T Annual Meeting: Beyond the Cloud: Rethinking Information Boundaries (ASIST ’13). American Society for Information Science, USA.
[32]
Irene C. Michas and Dianne C. Berry. 2000. Learning a procedural task: Effectiveness of multimedia presentations. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 14, 6 (2000), 555–575.
[33]
Dena Mujtaba and Nihar Mahapatra. 2019. Recent trends in natural language understanding for procedural knowledge. In 2019 International Conference on Computational Science and Computational Intelligence (CSCI). 420–424. DOI:
[34]
Georg Pardi, Yvonne Kammerer, and Peter Gerjets. 2019. Search and justification behavior during multimedia web search for procedural knowledge. In Companion Publication of the 10th ACM Conference on Web Science (WebSci ’19). ACM, New York, NY, USA, 17–20.
[35]
Hogun Park and Hamid Reza Motahari Nezhad. 2018. Learning procedures from text: Codifying how-to procedures in deep neural networks. In Companion Proceedings of The Web Conference 2018 (WWW ’18). International World Wide Web Conferences Steering Committee, 351–358. DOI:
[36]
Suppanut Pothirattanachaikul, Takehiro Yamamoto, Sumio Fujita, Akira Tajima, and Katsumi Tanaka. 2017. Mining alternative actions from community Q&A corpus for task-oriented web search. In Proceedings of the International Conference on Web Intelligence (WI ’17). ACM, New York, NY, USA, 607–614. DOI:
[37]
Victoria Reyes, Elizabeth Bogumil, and Levin Elias Welch. 2021. The living codebook: Documenting the process of qualitative data analysis. Sociological Methods & Research (2021), 0049124120986185.
[38]
Reijo Savolainen and Jarkko Kari. 2006. User-defined relevance criteria in web searching. Journal of Documentation (2006).
[39]
Michael Schneider and Elsbeth Stern. 2010. The developmental relations between conceptual and procedural knowledge: A multimethod approach. Developmental Psychology 46, 1 (2010), 178.
[40]
Pol Schumacher, Mirjam Minor, Kirstin Walter, and Ralph Bergmann. 2012. Extraction of procedural knowledge from the web: A comparison of two workflow extraction approaches. In Proceedings of the 21st International Conference on World Wide Web (WWW ’12 Companion). ACM, New York, NY, USA, 739–747. DOI:
[41]
Eliot R. Smith. 1994. Procedural knowledge and processing strategies in social cognition. Handbook of Social Cognition 2 (1994), 99–152.
[42]
Arthur Taylor. 2012. User relevance criteria choices and the information search process. Information Processing & Management 48, 1 (2012), 136–153.
[43]
Timon ten Berge and Rene van Hezewijk. 1999. Procedural and declarative knowledge: An evolutionary perspective. Theory & Psychology 9, 5 (1999), 605–624. DOI:
[44]
Timon Ten Berge and René Van Hezewijk. 1999. Procedural and declarative knowledge: An evolutionary perspective. Theory & Psychology 9, 5 (1999), 605–624.
[45]
Kelsey Urgo and Jaime Arguello. 2022. Understanding the “pathway” towards a searcher’s learning objective. ACM Transactions of Information Systems 40, 4 (2022).
[46]
Kelsey Urgo, Jaime Arguello, and Robert Capra. 2020. The effects of learning objectives on searchers’ perceptions and behaviors(ICTIR ’20). ACM, New York, NY, USA, 77–84. DOI:
[47]
Michael Völske, Pavel Braslavski, Matthias Hagen, Galina Lezina, and Benno Stein. 2015. What users ask a search engine: Analyzing one billion Russian question queries. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management (CIKM ’15). ACM, New York, NY, USA, 1571–1580. DOI:
[48]
Ingmar Weber, Antti Ukkonen, and Aris Gionis. 2012. Answers, not links: Extracting tips from yahoo! answers to address how-to web queries. In Proceedings of the 5th ACM International Conference on Web Search and Data Mining (WSDM ’12). ACM, New York, NY, USA, 613–622. DOI:
[49]
Barbara Wildemuth, Elaine G. Toms, and Luanne Freund. 2014. Untangling search task complexity and difficulty in the context of interactive information retrieval studies. Journal of Documentation 70, 6 (Oct.2014), 1118–1140.
[50]
Monte B. Wynder and Peter F. Luckett. 1999. The effects of understanding rules and a worked example on the acquisition of procedural knowledge and task performance. Accounting & Finance 39, 2 (1999), 177–203.
[51]
Yunjie Xu and Zhiwei Chen. 2006. Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science and Technology 57, 7 (2006), 961–973.
[52]
Zi Yang and Eric Nyberg. 2015. Leveraging procedural knowledge for task-oriented search(SIGIR ’15). ACM, New York, NY, USA, 513–522. DOI:
[53]
Ziqi Zhang, Philip Webster, Victoria Uren, Andrea Varga, and Fabio Ciravegna. 2012. Automatically extracting procedural knowledge from instructional texts using natural language processing. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC ’12). European Language Resources Association (ELRA), Istanbul, Turkey, 520–527.
Index Terms
- Better Understanding Procedural Search Tasks: Perceptions, Behaviors, and Challenges
Recommendations
Understanding Procedural Search Tasks “in the Wild”
CHIIR '23: Proceedings of the 2023 Conference on Human Information Interaction and RetrievalPeople often search online for procedural (i.e., “how-to”) knowledge. A procedural search task might involve a do-it-yourself project, cooking a dish, fixing a problem, or learning a new skill. Prior research has studied procedural search tasks from ...
Attitudes and Perceptions of Older Chinese People in Hong Kong Towards Silver Gaming
Human Aspects of IT for the Aged Population. Applications in Health, Assistance, and EntertainmentAbstractThis preliminary study aimed to explore the perceptions and attitudes of older Chinese people (over 60 years of age) in Hong Kong towards gaming by adopting a qualitative research method and to investigate the factors that motivate them to ...
What do exploratory searchers look at in a faceted search interface?
JCDL '09: Proceedings of the 9th ACM/IEEE-CS joint conference on Digital librariesThis study examined how searchers interacted with a web-based, faceted library catalog when conducting exploratory searches. It applied eye tracking, stimulated recall interviews, and direct observation to investigate important aspects of gaze behavior ...
Comments
Information & Contributors
Information
Published In
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 December 2023
Online AM: 23 October 2023
Accepted: 05 October 2023
Revised: 23 September 2023
Received: 27 April 2023
Published in TOIS Volume 42, Issue 3
Check for updates
Author Tags
Author Tags
Qualifiers
- Research-article
Funding Sources
- U.S. Department of Defense (DoD)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 282Total Downloads
- Downloads (Last 12 months)282
- Downloads (Last 6 weeks)17
Reflects downloads up to 24 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in