skip to main content
research-article

My synthetic wingman must understand me: modelling intent for future manned–unmanned teaming

Published: 29 November 2023 Publication History

Abstract

With advances in artificial intelligence, machine learning, and cognitive modelling, unmanned aircraft are expected to act as human-like wingmen in the near future. For fluent and effective manned–unmanned teaming, synthetic wingmen must be able to account for and adapt to their partners’ intent with little or no communication. To enable such abilities, it becomes crucial to identify the requirements that makes intent explainable to synthetic wingmen, necessitating approaches to describe and analyse intent from a human-centric perspective. To address this issue, this paper reports on findings from using Work Domain Analysis to design and analyse models of situated intent with six levels of cognitive control (frames, effects, values, generic, implementations, and physical). Through a literature review and seven subject matter expert interviews, a synthesized model was designed to represent fighter pilots’ intent in a manned–unmanned teaming scenario. Using the synthesized model as the context, a transfer of control and a link loss situation were further described and analysed. Experiences show that Work Domain Analysis can provide a practical and applicable means to model situated intent, particularly since designed models can be re-utilised to model intent in similar situations. Furthermore, the model analyses show the importance of accounting for fighter pilots’ adopted frames since small variations of the framing of the situations can propagate throughout the model resulting in conflicting or inconsistent intent. The paper concludes that synthetic wingmen must be able to reason about all six levels of cognitive control, requiring a more holistic approach to make intent explainable.

References

[1]
Abubshait A, Perez-Osorio J, De Tommaso D, Wykowska A (2021) Collaboratively framed interactions increase the adoption of intentional stance towards robots. In: 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN). IEEE, pp 886–891.
[2]
Adams JA, Cooper JL, Goodrich MA, Humphrey C, Quigley M, Buss BG, Morse BS (2007) Camera-equipped mini UAVs for wilderness search support: task analysis and lessons from field trials. BYUHCMI TECHNICAL REPORT 2007-1
[3]
Albrecht SV and Stone P Autonomous agents modelling other agents: a comprehensive survey and open problems Artif Intell 2018 258 66-95
[4]
Amelink MHJ, Mulder M, van Paassen MM (2008) Designing for human-automation interaction: abstraction-sophistication analysis for UAV control. In: Proceedings of the International MultiConference of Engineers and Computer Scientiests 2008 (IMECS 2008), 1:1–6
[5]
Anderson E, Fannin T, Nelson B (2018) Levels of aviation autonomy. In: 2018 IEEE/AIAA 37th Digital Avionics Systems Conference (DASC). IEEE, 1–8.
[6]
Argyris C and Schön DA Theory in practice: increasing professional effectiveness 1974 San Fransisco Jossey-Bass
[7]
Back Y, Zak Y, Parmet Y, and Oron-Gilad T Combining cognitive work analysis and empirical evaluations to understand map use by operators of small carry-on unmanned aerial systems Appl Ergon 2021 90 103218
[8]
Baltrusch SJ, Krause F, de Vries AW, van Dijk W, and de Looze MP What about the human in human robot collaboration? Ergonomics 2022 65 5 719-740
[9]
Bauer A, Wollherr D, and Buss M Human–robot collaboration: a survey Int J Humanoid Rob 2008 5 1 47-66
[10]
Benishek LE and Lazzara EH Teams in a new era: some considerations and implications Front Psychol 2019 10 MAY 1-15
[11]
Bishop P, Hines A, and Collins T The current state of scenario development: an overview of techniques Foresight 2007 9 1 5-25
[12]
Börjeson L, Höjer M, Dreborg K-H, Ekvall T, and Finnveden G Scenario types and techniques: towards a user’s guide Futures 2006 38 7 723-739
[13]
Bratman ME Intention, plans, and practical reason 1987 Cambridge Harvard University Press
[14]
Braun V and Clarke V Using thematic analysis in psychology Qual Res Psychol 2006 3 2 77-101
[15]
Burns CM, Bisantz AM, and Roth EM Lessons from a comparison of work domain models: representational choices and their implications Hum Factors 2004 46 4 711-727
[16]
Burns CM, Bryant DJ, Chalmers BA (2001) Scenario mapping with work domain analysis. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting 45 (4): 424–428.
[17]
Castro LNG, Pritchett AR (2005) Work domain analysis for improvement of uninhabited aerial vehicle (UAV) operations. In: 2005 IEEE Design Symposium, Systems and Information Engineering, IEEE, 65–74.
[18]
Chen JYC, Lakhmani SG, Stowers K, Selkowitz AR, Wright JL, and Barnes M Situation awareness-based agent transparency and human-autonomy teaming effectiveness Theor Issues Ergon Sci 2018 19 3 259-282
[19]
Chen JYC, Procci K, Boyce M, Wright J, Garcia A, Barnes MJ (2014) Situation awareness–based agent transparency. situation awareness-based agent transparency (ARL-TR-6905). https://apps.dtic.mil/sti/pdfs/AD1143367.pdf
[20]
Clark HH and Brennan SE Grounding in communication Perspectives on socially shared cognition 1991 Washington American Psychological Association 127-149
[21]
Cohen PR and Levesque HJ Intention is choice with commitment Artif Intell 1990 42 2–3 213-261
[22]
Cooke NJ, Gorman JC, Myers CW, and Duran JL Interactive team cognition Cogn Sci 2013 37 2 255-285
[23]
de Visser EJ, Peeters MMM, Jung MF, Kohn S, Shaw TH, Pak R, and Neerincx MA Towards a theory of longitudinal trust calibration in human-robot teams Int J Soc Robot 2020 12 2 459-478
[24]
Demir M, McNeese NJ, and Cooke NJ Team situation awareness within the context of human-autonomy teaming Cogn Syst Res 2017 46 December 3-12
[25]
Dennett DC The intentional stance 1987 Cambridge MIT Press
[26]
Department of Defence (2014) Unmanned systems integration roadmap FY2013–2038. https://www.hsdl.org/?view&did=747559
[27]
Department of Defence (2018) Unmanned systems integrated roadmap 2017–2042. https://www.hsdl.org/?view&did=826737
[28]
Elliott G, Crawford J, Watson M, Sanderson P (2000) Knowledge elicitation techniques for modelling intentional systems with cognitive work analysis. Society
[29]
Endsley MR (2015) Autonomous horizons: system autonomy in the air force—a path to the future. Report No. AF/ST TR 15-01
[30]
Freedman RG, Zilberstein S (2019) A unifying perspective of plan, activity, and intent recognition. In: Proceedings of the AAAI Workshops: Plan, Activity, Intent Recognition, 1–8
[31]
Frith CD and Frith U The neural basis of mentalizing Neuron 2006 50 4 531-534
[32]
Fuchs C, Borst C, de Croon GCHE, van Paassen MMR, and Mulder M An ecological approach to the supervisory control of UAV swarms Int J Micro Air Veh 2014 6 4 211-229
[33]
Geddes ND, Lizza CS (2001) Practical applications of a real time, dynamic model of intentions. In: AAAI Fall Symposium
[34]
Geddes ND (1994) A model for intent interpretation for multiple agents with conflicts. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, 3:2080–2085. IEEE.
[35]
Hajdukiewicz JR, Burns CM, Vicente KJ, Eggleston RG (1999) Work domain analysis for intentional systems. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting 43 (3): 333–337.
[36]
Han TA Intention recognition, commitment and their roles in the evolution of cooperation 9 studies in applied philosophy, epistemology and rational ethics 2013 Berlin, Heidelberg Springer, Berlin Heidelberg
[37]
Heinze C (2004) Modelling intention recognition for intelligent agent systems. Doctoral thesis. University of Melbourne, Australia
[38]
Hiatt LM, Narber C, Bekele E, Khemlani SS, and Gregory Trafton J Human modeling for human-robot collaboration Int J Robot Res 2017 36 5–7 580-596
[39]
Hobbs A and Lyall B Human factors guidelines for unmanned aircraft systems Ergon Des 2016 24 3 23-28
[40]
Hoc J-M From human–machine interaction to human–machine cooperation Ergonomics 2000 43 7 833-843
[41]
Hoffman RR, Shadbolt NR, Mike Burton A, and Klein G Eliciting knowledge from experts: a methodological analysis Organ Behav Hum Decis Process 1995 62 2 129-158
[42]
Holder E, Huang L, Chiou E, Jeon M, and Lyons JB Designing for bi-directional transparency in human-AI-robot-teaming Proc Hum Factors Ergon Soc Annu Meet 2021 65 1 57-61
[43]
Howard N and Cambria E Intention awareness: improving upon situation awareness in human-centric environments HCIS 2013 3 1 9
[44]
Jain S and Argall B Probabilistic human intent recognition for shared autonomy in assistive robotics ACM Trans Hum-Robot Interact 2020 9 1 1-23
[45]
Jenkins DP Using cognitive work analysis to describe the role of UAVs in military operations Theor Issues Ergon Sci 2012 13 3 335-357
[46]
Jordan J The future of unmanned combat aerial vehicles: an analysis using the three horizons framework Futures 2021 134 December 102848
[47]
Klein G Levis AH and Levis IS A script for the commander’s intent Science of command and control: part III: coping with change 1994 Fairfax AFCEA International Press 75-85
[48]
Klein G Sources of power: how people make decisions 1999 Cambridge The MIT Press
[49]
Klein G, Snowden D, and Pin CL Mosier KL and Fischer UM Anticipatory thinking Informed by knowledge: expert performance in complex situations 2010 Psychology Press 235-245
[50]
Klein G, Phillips JK, Rall EL, Peluso DA (2007) A data–frame theory of sensemaking. In: Hoffman RR (ed) Expertise out of Context: Proceedings of the Sixth International Conference on Naturalistic Decision Making. Lawrence Erlbaum Associates, New York, pp 113–155
[51]
Leveson NG Intent specifications: an approach to building human-centered specifications IEEE Trans Software Eng 2000 26 1 15-35
[52]
Lui F and Watson M Mapping cognitive work analysis (CWA) to an intelligent agents software architecture: command agents Proc Defence Hum Factors Spec Interest Group (DHFSIG) 2002 2002 1-5
[53]
Lundberg J and Johansson BJE Systemic resilience model Reliab Eng Syst Saf 2015 141 22-32
[54]
Lundberg J and Johansson BJE Resilience is not a silver bullet—harnessing resilience as core values and resource contexts in a double adaptive process Reliab Eng Syst Saf 2019 188 110-117
[55]
Lundberg J and Johansson BJE A framework for describing interaction between human operators and autonomous, automated, and manual control systems Cogn Technol Work 2021 23 3 381-401
[56]
Lyons JB, Sycara K, Lewis M, and Capiola A Human–autonomy teaming: definitions, debates, and directions Front Psychol 2021 12 1-15
[57]
Lyons JB, Havig PR (2014) Transparency in a human−machine context: approaches for fostering shared awareness/intent. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8525 LNCS, pp 181–190.
[58]
Lyons JB (2013) Being transparent about transparency: a model for human-robot interaction. In: Trust and Autonomous Systems: Papers from the AAAI Spring Symposium, SS-13-07:48–53. Technical Report SS-13-07
[59]
Malle BF and Knobe J The folk concept of intentionality J Exp Soc Psychol 1997 33 2 101-121
[60]
Marathe AR, Schaefer KE, Evans AW, Metcalfe JS (2018) Bidirectional communication for effective human–agent teaming. In: Chen JYC, Fragomeni G (eds) Virtual, augmented and mixed reality: interaction, navigation, visualization, embodiment, and simulation. VAMR 2018. Lecture Notes in Computer Science, 10909. Springer, Cham, 338–350.
[61]
Meyer C, Schulte A (2020) Operator controlled, reactive UAV behaviors in manned–unmanned teaming scenarios with selective datalink availability. In: 2020 International Conference on Unmanned Aircraft Systems, ICUAS 2020, 1673–1679.
[62]
Militello LG and Hutton RJB Applied cognitive task analysis (ACTA): a practitioner’s toolkit for understanding cognitive task demands Ergonomics 1998 41 11 1618-1641
[63]
Minsky M (1975) Minsky’s frame system theory. In: Nash-Webber BL, Schank R (eds) TINLAP ‘75: Proceedings of the 1975 Workshop on theoretical issues in natural language processing, 104–116.
[64]
Miranda L and Garibary OO Approaching (super)human intent recognition in stag hunt with the naïve utility calculus generative model Comput Math Organ Theory 2022
[65]
Naikar N Work domain analysis: concept, guidelines, and cases work domain analysis 2013 CRC Press
[66]
Neale M, Schultz M (2007) Current and future unmanned aircraft system control & communications datalinks. In: AIAA Infotech@Aerospace 2007 Conference and Exhibit, 3:2755–2764. American Institute of Aeronautics and Astronautics, Reston.
[67]
Norling E (2008) What should the agent know? The challenge of capturing human knowledge. In: Padgham L and Parkes D (eds) AAMAS ‘08: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland, 1225–1228
[68]
Norling EJ (2012) Modelling human behaviour with BDI agents. Doctoral thesis, University of Melbourne, Australia
[69]
Nylin M, Westberg JJ, and Lundberg J Reduced autonomy workspace (RAW)—an interaction design approach for human−automation cooperation Cogn Technol Work 2022 24 2 261-273
[70]
O’Hare D, Wiggins M, Williams A, and Wong W Cognitive task analyses for decision centred design and training Ergonomics 1998 41 11 1698-1718
[71]
O’Neill T, McNeese N, Barron A, and Schelble B Human–autonomy teaming: a review and analysis of the empirical literature Hum Factors 2020
[72]
Ohlander U, Alfredson J, Riveiro M, and Falkman G Fighter pilots’ teamwork: a descriptive study Ergonomics 2019 62 7 880-890
[73]
Okcu H Operational requirements of unmanned aircraft systems data link and communication systems J Adv Comput Netw 2016 4 1 28-32
[74]
Perez-Osorio J and Wykowska A Adopting the intentional stance toward natural and artificial agents Philos Psychol 2020 33 3 369-395
[75]
Pigeau R and McCann C English A Establishing common intent: the key to co-ordinated military action The operational art: Canadian perspectives: leadership and command 2006 Kingston Canadian Defence Academy Press
[76]
Premack D and Woodruff G Does the chimpanzee have a theory of mind? Behav Brain Sci 1978 4 515-526
[77]
Rasmussen J (1998) Ecological interface design for complex systems: an example: SEAD—UAV systems. Wright-Patterson AFB OH 45433–7022. Dayton, OH
[78]
Reily B, Gao P, Han F, Wang H, and Zhang H Real-time recognition of team behaviors by multisensory graph-embedded robot learning Int J Robot Res 2022 41 8 798-811
[79]
Rix J (2022) From tools to teammates: conceptualizing humans’ perception of machines as teammates with a systematic literature review. In: Proceedings of the 55th Hawaii International Conference on System Sciences, 398–407.
[80]
Roth EM, Sushereba C, Militello LG, Diiulio J, and Ernst K Function allocation considerations in the era of human autonomy teaming J Cogn Eng Decis Mak 2019 13 4 199-220
[81]
Sadraey MH (2018) Manned−unmanned aircraft teaming. In: 2018 IEEE Aerospace Conference, 2018-March:1–12. IEEE.
[82]
Salas E, Sims DE, and Shawn Burke C Is there a “big five” in teamwork? Small Group Res 2005 36 5 555-599
[83]
Salas E, Cooke NJ, and Rosen MA On teams, teamwork, and team performance: discoveries and developments Hum Factors 2008 50 3 540-547
[84]
Scassellati B Theory of mind for a humanoid robot Auton Robot 2002 12 1 13-24
[85]
Schaefer KE, Straub ER, Chen JYC, Putney J, and Evans AW Communicating intent to develop shared situation awareness and engender trust in human-agent teams Cogn Syst Res 2017 46 26-39
[86]
Schelble BG, Flathmann C, McNeese NJ, Freeman G, and Mallick R ‘Let’s think together! Assessing shared mental models, performance, and trust in human−agent teams Proc ACM Hum Comput Interact 2022 6 3 1-29
[87]
Schelble BG, Flathmann C, McNeese N (2020) Towards meaningfully integrating human-autonomy teaming in applied settings. In: Proceedings of the 8th International Conference on Human−Agent Interaction. ACM, New York, 149–156.
[88]
Schneider MF, Miller ME, Ford TC, Peterson G, and Jacques D Intent integration for human−agent teaming Syst Eng 2022
[89]
Schneider MF, Miller ME (2018) Operationalized intent for communication in human−agent teams. In: 2018 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA). IEEE, 117–123.
[90]
Schulte A Cognitive automation for tactical mission management: concept and prototype evaluation in flight simulator trials Cogn Technol Work 2002 4 3 146-159
[91]
Snyder H Literature review as a research methodology: an overview and guidelines J Bus Res 2019 104 July 333-339
[92]
Stansbury RS, Vyas MA, and Wilson TA A survey of UAS technologies for command, control, and communication (C3) J Intell Rob Syst 2009 54 1–3 61-78
[93]
St-Maurice JD and Burns CM Using cognitive work analysis to compare complex system domains Theor Issues Ergon Sci 2018 19 5 553-577
[94]
Suck S, Fortmann F (2016) Aircraft pilot intention recognition for advanced cockpit assistance systems. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9744:231–240.
[95]
Sukthankar G, Goldman RP, Geib C, Pynadath DV, and Bui HH Plan, activity, and intent recognition: theory and practice 2014 Elsevier
[96]
Tabrez A, Luebbers MB, and Hayes B A survey of mental modeling techniques in human–robot teaming Curr Robot Rep 2020 1 4 259-267
[97]
Theissing N, Schulte A (2013) Intent-based UAV mission management using an adaptive mixed-initiative operator assistant system. In: AIAA Infotech@Aerospace (I@A) Conference, 1–17.
[98]
Vanderhaegen F Heuristic-based method for conflict discovery of shared control between humans and autonomous systems—a driving automation case study Robot Autonom Systms 2021 146 December 103867
[99]
Van-Horenbeke FA and Peer A Activity, plan, and goal recognition: a review Front Robot AI 2021 8 1-18
[100]
Warren A and Hillas A Friend or frenemy? The role of trust in human−machine teaming and lethal autonomous weapons systems Small Wars Insurgencies 2020 31 4 822-850
[101]
Whittemore R and Knafl K The integrative review: updated methodology J Adv Nurs 2005 52 5 546-553
[102]
Wynne KT and Lyons JB An integrative model of autonomous agent teammate-likeness Theor Issues Ergon Sci 2018 19 3 353-374

Cited By

View all
  • (2024)Plan and Goal Recognition System for Adaptive Pilot Assistance in Tactical Helicopter OperationsEngineering Psychology and Cognitive Ergonomics10.1007/978-3-031-60728-8_16(201-213)Online publication date: 29-Jun-2024

Index Terms

  1. My synthetic wingman must understand me: modelling intent for future manned–unmanned teaming
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Cognition, Technology and Work
    Cognition, Technology and Work  Volume 26, Issue 1
    Feb 2024
    180 pages

    Publisher

    Springer-Verlag

    Berlin, Heidelberg

    Publication History

    Published: 29 November 2023
    Accepted: 06 October 2023
    Received: 15 July 2022

    Author Tags

    1. Manned–unmanned teaming
    2. Work domain analysis
    3. Intent
    4. Model

    Qualifiers

    • Research-article

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)0
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 24 Oct 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Plan and Goal Recognition System for Adaptive Pilot Assistance in Tactical Helicopter OperationsEngineering Psychology and Cognitive Ergonomics10.1007/978-3-031-60728-8_16(201-213)Online publication date: 29-Jun-2024

    View Options

    View options

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media