skip to main content
research-article

Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual Reality

Published: 13 September 2023 Publication History

Abstract

Biocybernetic loops encompass users' state detection and system adaptation based on physiological signals. Current adaptive systems limit the adaptation to task features such as task difficulty or multitasking demands. However, virtual reality allows the manipulation of task-irrelevant elements in the environment. We present a physiologically adaptive system that adjusts the virtual environment based on physiological arousal, i.e., electrodermal activity. We conducted a user study with our adaptive system in social virtual reality to verify improved performance. Here, participants completed an n-back task, and we adapted the visual complexity of the environment by changing the number of non-player characters. Our results show that an adaptive virtual reality can control users' comfort, performance, and workload by adapting the visual complexity based on physiological arousal. Thus, our physiologically adaptive system improves task performance and perceived workload. Finally, we embed our findings in physiological computing and discuss applications in various scenarios.

Supplementary Material

7Z File (v7mhci196aux.7z)
1. AdaptiveVR-EDA.zip contains the VR adaptive system and the VR experiment setup 2. MHCI_Step_01_Read_Questionnaires.html contains the script for analyzing the questionnaires 3. MHCI_Step_02_Read_Data-Per-Block.html contains the script for analyzing the behavioral and physiological data 4. MHCI_Step_03_Plots.html contains the code for replicate the figures in the paper. 5. MHCI_Step_04_stats_R.html contains the code for replicating statistical results.

References

[1]
Daniel Afergan, Evan M. Peck, Erin T. Solovey, Andrew Jenkins, Samuel W. Hincks, Eli T. Brown, Remco Chang, and Robert J.K. Jacob. 2014. Dynamic Difficulty Using Brain Metrics of Workload. In Proceedings of the SIGCHI Conf. on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). ACM, NY, NY, USA, 3797--3806. https://doi.org/10.1145/2556288.2557230
[2]
Richard J Allen, Alan D Baddeley, and Graham J Hitch. 2017. Executive and perceptual distraction in visual working memory. Journal of Experimental Psychology: Human Perception and Performance 43, 9 (2017), 1677. https://doi.org/10.1037/xhp0000413
[3]
George A Alvarez and Patrick Cavanagh. 2004. The capacity of visual short-term memory is set both by visual information load and by number of objects. Psychological science 15, 2 (2004), 106--111. https://doi.org/10.1111/j.0963--7214.2004.01502006.x
[4]
Lena M Andreessen, Peter Gerjets, Detmar Meurers, and Thorsten O Zander. 2021. Toward neuroadaptive support technologies for improving digital reading: a passive BCI-based assessment of mental workload imposed by text difficulty and presentation speed during reading. User Modeling and User-Adapted Interaction 31 (2021), 75--104. https://doi.org/10.1007/s11257-020-09273--5
[5]
Justin Asbee, Kimberly Kelly, Timothy McMahan, and Thomas D Parsons. 2023. Machine learning classification analysis for an adaptive virtual reality Stroop task. Virtual Reality (2023), 1--17. https://doi.org/10.1007/s10055-022-00744--1
[6]
Ebrahim Babaei, Benjamin Tag, Tilman Dingler, and Eduardo Velloso. 2021. A Critique of Electrodermal Activity Practices at CHI. In Proceedings of the 2021 CHI Conf. on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). ACM, NY, NY, USA, Article 177, 14 pages. https://doi.org/10.1145/3411764.3445370
[7]
Mathias Benedek and Christian Kaernbach. 2010. A continuous measure of phasic electrodermal activity. Journal of Neuroscience Methods 190, 1 (2010), 80--91. https://doi.org/10.1016/j.jneumeth.2010.04.028
[8]
Mathias Benedek and Christian Kaernbach. 2010. Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 47, 4 (2010), 647--658. https://doi.org/10.1111/j.1469--8986.2009.00972.x
[9]
Yulong Bian, Chenglei Yang, Chao Zhou, Juan Liu,Wei Gai, Xiangxu Meng, Feng Tian, and Chia Shen. 2018. Exploring the Weak Association between Flow Experience and Performance in Virtual Environments. In Proceedings of the 2018 CHI Conf. on Human Factors in Computing Systems. ACM, NY, NY, USA, 1--12. https://doi.org/10.1145/3173574.3173975
[10]
Felix Born and Maic Masuch. 2017. Increasing presence in a mixed reality application by integrating a real time tracked full body representation. In Int. Conf. on Advances in Computer Entertainment. Springer, 46--60. https://doi.org/10.1007/978--3--319--76270--8_4
[11]
Wolfram Boucsein and RichardWBacks. 2000. Engineering psychophysiology as a discipline: Historical and theoretical aspects. Engineering psychophysiology: Issues and applications (2000), 3--30.
[12]
Wolfram Boucsein, Ruediger Baltissen, and Manfred Euler. 1984. Dependence of skin conductance reactions and skin resistance reactions upon previous level. Psychophysiology 21, 2 (1984), 212--218. https://doi.org/10.1111/j.1469--8986.1984.tb00207.x
[13]
Wolfram Boucsein, Andrea Haarmann, and Florian Schaefer. 2007. Combining skin conductance and heart rate variability for adaptive automation during simulated IFR flight. In Int. Conf. on Engineering Psychology and Cognitive Ergonomics. Springer, 639--647. https://doi.org/10.1007/978--3--540--73331--7_70
[14]
Peter Carruthers. 2013. Evolution of working memory. Proceedings of the National Academy of Sciences 110, supplement_2 (2013), 10371--10378. https://doi.org/10.1073/pnas.1301195110
[15]
Alan Chalmers and Andrej Ferko. 2008. Levels of realism: From virtual reality to real virtuality. In Proceedings of the 24th Spring Conf. on Computer Graphics. 19--25. https://doi.org/10.1145/1921264.1921272
[16]
Weixuan Chen, Natasha Jaques, Sara Taylor, Akane Sano, Szymon Fedor, and Rosalind W Picard. 2015. Wavelet-based motion artifact removal for electrodermal activity. In 2015 37th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, NY, NY, USA, 6223--6226. https://doi.org/10.1109/EMBC.2015.7319814
[17]
Francesco Chiossi, Thomas Kosch, Luca Menghini, Steeven Villa, and Sven Mayer. 2023. SensCon: Embedding Physiological Sensing into Virtual Reality Controllers. Proc. ACM Hum.-Comput. Interact. 7, MHCI, Article 223, 32 pages. https://doi.org/10.1145/3604270
[18]
Francesco Chiossi and Sven Mayer. 2023. How Can Mixed Reality Benefit From Physiologically-Adaptive Systems? Challenges and Opportunities for Human Factors Applications. Proceedings ofWorkshop on the Future of Computational Approaches for Understanding and Adapting User Interfaces (2023). https://doi.org/10.48550/arXiv.2303.17978
[19]
Francesco Chiossi, Changkun Ou, and Sven Mayer. 2023. Exploring Physiological Correlates of Visual Complexity Adaptation: Insights from EDA, ECG, and EEG Data for Adaptation Evaluation in VR Adaptive Systems. In ACM Conf. on Human-Computer Interaction (CHI '23 Extended Abstracts). ACM, Hamburg, Germany. https://doi.org/10.1145/3544549.3585624
[20]
Francesco Chiossi, Yagiz Turgut, Robin Welsch, and Sven Mayer. 2023. Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual Reality. https://doi.org/10.17605/OSF.IO/AXVFY
[21]
Francesco Chiossi, Robin Welsch, Steeven Villa, Lewis Chuang, and Sven Mayer. 2022. Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience. Big Data and Cognitive Computing 6, 2 (2022), 55. https://doi.org/10.3390/bdcc6020055
[22]
Francesco Chiossi, Johannes Zagermann, Jakob Karolus, Nils Rodrigues, Priscilla Balestrucci, Daniel Weiskopf, Benedikt Ehinger, Tiare Feuchtner, Harald Reiterer, Lewis L. Chuang, Marc Ernst, Andreas Bulling, Sven Mayer, and Albrecht Schmidt. 2022. Adapting visualizations and interfaces to the user. it - Information Technology (2022). https://doi.org/10.1515/itit-2022-0035
[23]
Chris Christou, Kyriakos Herakleous, Aimilia Tzanavari, and Charalambos Poullis. 2015. Psychophysiological responses to virtual crowds: implications for wearable computing. In 2015 Int. Conf. on Affective Computing and Intelligent Interaction (ACII). IEEE, NY, NY, USA, 35--41. https://doi.org/10.1109/ACII.2015.7344548
[24]
Taylor W Cleworth, Brian C Horslen, and Mark G Carpenter. 2012. Influence of real and virtual heights on standing balance. Gait & posture 36, 2 (2012), 172--176. https://doi.org/10.1016/j.gaitpost.2012.02.010
[25]
Giorgia Cona, Francesco Chiossi, Silvia Di Tomasso, Giovanni Pellegrino, Francesco Piccione, Patrizia Bisiacchi, and Giorgio Arcara. 2020. Theta and alpha oscillations as signatures of internal and external attention to delayed intentions: A magnetoencephalography (MEG) study. NeuroImage 205 (2020), 116295. https://doi.org/10.1016/j.neuroimage.2019.116295
[26]
William Jay Conover. 1999. Practical nonparametric statistics. Vol. 350. John Wiley & Sons. https://doi.org/10.2307/2981807
[27]
Mihaly Csikszentmihalyi, Sami Abuhamdeh, and Jeanne Nakamura. 2005. Flow. Handbook of competence and motivation (2005), 598--608.
[28]
Ralph B d'Agostino. 1971. An omnibus test of normality for moderate and large size samples. Biometrika 58, 2 (1971), 341--348. https://doi.org/10.2307/2334522
[29]
Jan W de Fockert, Geraint Rees, Christopher D Frith, and Nilli Lavie. 2001. The role of working memory in visual selective attention. Science 291, 5509 (2001), 1803--1806. https://doi.org/10.1126/science.1056496
[30]
Janet A Desor. 1972. Toward a psychological theory of crowding. Journal of personality and social psychology 21, 1 (1972), 79. https://doi.org/10.1037/h0032112
[31]
Tilman Deuschel. 2018. On the Influence of Human Factors in Adaptive User Interface Design. In Adjunct Publication of the 26th Conf. on User Modeling, Adaptation and Personalization (Singapore, Singapore) (UMAP '18). ACM, NY, NY, USA, 187--190. https://doi.org/10.1145/3213586.3213587
[32]
Arindam Dey, Alex Chatburn, and Mark Billinghurst. 2019. Exploration of an EEG-Based Cognitively Adaptive Training System in Virtual Reality. In 2019 IEEE Conf. on Virtual Reality and 3D User Interfaces. IEEE, NY, NY, USA, 220--226. https://doi.org/10.1109/VR.2019.8797840
[33]
John Duncan and Glyn W Humphreys. 1989. Visual search and stimulus similarity. Psychological review 96, 3 (1989), 433. https://doi.org/10.1037/0033--295X.96.3.433
[34]
Mai ElKomy, Yomna Abdelrahman, Markus Funk, Tilman Dingler, Albrecht Schmidt, and Slim Abdennadher. 2017. ABBAS: an adaptive bio-sensors based assistive system. In Proceedings of the 2017 CHI Conf. extended abstracts on human factors in computing systems. 2543--2550. https://doi.org/10.1145/3027063.3053179
[35]
Hing Yee Eng, Diyu Chen, and Yuhong Jiang. 2005. Visual working memory for simple and complex visual stimuli. Psychonomic bulletin & review 12, 6 (2005), 1127--1133. https://doi.org/10.3758/BF03206454
[36]
Kate C Ewing, Stephen H Fairclough, and Kiel Gilleade. 2016. Evaluation of an adaptive game that uses EEG measures validated during the design process as inputs to a biocybernetic loop. Frontiers in human neuroscience 10 (2016), 223. https://doi.org/10.3389/fnhum.2016.00223
[37]
Stephen H Fairclough. 2009. Fundamentals of physiological computing. Interacting with computers 21, 1--2 (2009), 133--145. https://doi.org/10.1016/j.intcom.2008.10.011
[38]
Stephen H Fairclough. 2017. Physiological computing and intelligent adaptation. In Emotions and affect in human factors and human-computer interaction. Elsevier, 539--556. https://doi.org/10.1016/B978-0--12--801851--4.00020--3
[39]
Stephen H Fairclough, Christopher Burns, and Ute Kreplin. 2018. FNIRS activity in the prefrontal cortex and motivational intensity: impact of working memory load, financial reward, and correlation-based signal improvement. Neurophotonics 5, 3 (2018), 035001. https://doi.org/10.1117/1.NPh.5.3.035001
[40]
Stephen H Fairclough and Louise Venables. 2006. Prediction of subjective states from psychophysiology: A multivariate approach. Biological psychology 71, 1 (2006), 100--110. https://doi.org/10.1016/j.biopsycho.2005.03.007
[41]
Ying Xing Feng, Tong Boon Tang, and Eric Tatt Wei Ho. 2021. Phasic Electrodermal Activity Indicates Changes in Workload and Affective States. In 2021 Int. Conf. on Intelligent Cybernetics Technology & Applications (ICICyTA). IEEE, NY, NY, USA, 133--137. https://doi.org/10.1109/ICICyTA53712.2021.9689112
[42]
Daniel Fitousi and Michael J Wenger. 2011. Processing capacity under perceptual and cognitive load: a closer look at load theory. Journal of experimental psychology: human perception and performance 37, 3 (2011), 781. https://doi.org/10.1037/a0020675
[43]
Society for Psychophysiological Research Ad Hoc Committee on Electrodermal Measures, Wolfram Boucsein, Don C Fowles, Sverre Grimnes, Gershon Ben-Shakhar, Walton T Roth, Michael E Dawson, and Diane L Filion. 2012. Publication recommendations for electrodermal measurements. Psychophysiology 49, 8 (2012), 1017--1034. https://doi.org/10.1111/j.1469--8986.2012.01384.x
[44]
Guo Freeman and Divine Maloney. 2021. Body, avatar, and me: The presentation and perception of self in social virtual reality. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1--27. https://doi.org/10.1145/3432938
[45]
Doron Friedman. 2015. Brain-computer interfacing and virtual reality. Handbook of digital games and entertainment technologies (2015), 1--22. https://doi.org/10.1007/978--981--4560--52--8_2--1
[46]
Keisuke Fukuda and Edward K Vogel. 2011. Individual differences in recovery time from attentional capture. Psychological science 22, 3 (2011), 361--368. https://doi.org/10.1177/0956797611398493
[47]
Alireza Mazloumi Gavgani, Keith V Nesbitt, Karen L Blackmore, and Eugene Nalivaiko. 2017. Profiling subjective Symptoms and autonomic changes associated with cybersickness. Autonomic Neuroscience 203 (2017), 41--50. https://doi.org/10.1016/j.autneu.2016.12.004
[48]
Guilherme Gonçalves, Hugo Coelho, Pedro Monteiro, Miguel Melo, and Maximino Bessa. 2022. Systematic Review of Comparative Studies of the Impact of Realism in Immersive Virtual Experiences. ACM Comput. Surv. (apr 2022). https://doi.org/10.1145/3533377
[49]
Klaus Gramann, Ryan McKendrick, Carryl Baldwin, Raphaëlle N Roy, Camille Jeunet, Ranjana K Mehta, and Giovanni Vecchiato. 2021. Grand field challenges for cognitive neuroergonomics in the coming decade., 643969 pages. https://doi.org/10.3389/fnrgo.2021.643969
[50]
Alberto Greco, Gaetano Valenza, and Enzo Pasquale Scilingo. 2016. Modeling for the Analysis of the EDA. Springer Int. Publishing, Cham, 19--33. https://doi.org/10.1007/978--3--319--46705--4_2
[51]
Joe Guna, Gregor Ger?ak, Iztok Humar, Maja Krebl, Marko Orel, Huimin Lu, and Matev? Pogacnik. 2020. Virtual reality sickness and challenges behind different technology and content settings. Mobile Networks and Applications 25, 4 (2020), 1436--1445. https://doi.org/10.1007/s11036-019-01373-w
[52]
Hugo Hammond, Graham Thomas, and Iain D Gilchrist. 2022. Dynamically changing attention in complex visual stimuli. Journal of Vision 22, 14 (2022), 3602--3602. https://doi.org/10.1167/jov.22.14.3602
[53]
PA Hancock and JL Szalma. 2003. The future of neuroergonomics. Theoretical Issues in Ergonomics Science 4, 1--2 (2003), 238--249. https://doi.org/10.1080/1463922021000020927
[54]
Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9 (2006), 904--908. https://doi.org/10.1177/154193120605000909
[55]
Ann Huang, Pascal Knierim, Francesco Chiossi, Lewis L Chuang, and RobinWelsch. 2022. Proxemics for Human-Agent Interaction in Augmented Reality. In Proceedings of the 2022 CHI Conf. on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). ACM, NY, NY, USA, Article 421, 13 pages. https://doi.org/10.1145/3491102.3517593
[56]
Yizhen Huang, Eric Richter, Thilo Kleickmann, Axel Wiepke, and Dirk Richter. 2021. Classroom complexity affects student teachers' behavior in a VR classroom. Computers & Education 163 (2021), 104100. https://doi.org/10.1016/j.compedu.2020.104100
[57]
Wijnand A IJsselsteijn, Yvonne AWde Kort, and Karolien Poels. 2013. The game experience questionnaire. Eindhoven: Technische Universiteit Eindhoven 46, 1 (2013).
[58]
Susanne M Jaeggi, Martin Buschkuehl, Walter J Perrig, and Beat Meier. 2010. The concurrent validity of the N-back task as a working memory measure. Memory 18, 4 (2010), 394--412. https://doi.org/10.1080/09658211003702171
[59]
Manolya Kavakli, Iwan Kartiko, John Porte, and Nolwenn Bigoin. 2008. Effects of digital content on motion sickness in immersive virtual environments. In 3rd Int. Conf. on Computer Science & Information Systems, July. 23--24.
[60]
Jackob N Keynan, Avihay Cohen, Gilan Jackont, Nili Green, Noam Goldway, Alexander Davidov, Yehudit Meir-Hasson, Gal Raz, Nathan Intrator, Eyal Fruchter, et al. 2019. Electrical fingerprint of the amygdala guides neurofeedback training for stress resilience. Nature human behaviour 3, 1 (2019), 63--73. https://doi.org/10.1038/s41562-018-0484--3
[61]
Madison Klarkowski, Daniel Johnson, Peta Wyeth, Cody Phillips, and Simon Smith. 2018. Don't Sweat the Small Stuff: The Effect of Challenge-Skill Manipulation on Electrodermal Activity. In Proceedings of the 2018 Annual Symp. on Computer-Human Interaction in Play (Melbourne, VIC, Australia) (CHI PLAY '18). ACM, NY, NY, USA, 231--242. https://doi.org/10.1145/3242671.3242714
[62]
Ian Kleckner, Jolie B Wormwood, Rebecca M Jones, Erika Siegel, Eva Culakova, James Heathers, Lisa Feldman Barrett, Catherine Lord, Karen Quigley, and Matthew Goodwin. 2021. Adaptive thresholding increases ability to detect changes in rate of skin conductance responses to psychologically arousing stimuli. (2021). https://doi.org/10.31234/osf.io/b4agz
[63]
Thomas Kosch, Jakob Karolus, Havy Ha, and Albrecht Schmidt. 2019. Your skin resists: exploring electrodermal activity as workload indicator during manual assembly. In Proceedings of the ACM SIGCHI Symp. on Engineering Interactive Computing Systems. 1--5. https://doi.org/10.1145/3319499.3328230
[64]
Thomas Kosch, Jakob Karolus, Johannes Zagermann, Harald Reiterer, Albrecht Schmidt, and Pawel W. Wozniak. 2023. A Survey on Measuring Cognitive Workload in Human-Computer Interaction. ACM Comput. Surv. (jan 2023). https://doi.org/10.1145/3582272
[65]
Thomas Kosch, Robin Welsch, Lewis Chuang, and Albrecht Schmidt. 2022. The Placebo Effect of Artificial Intelligence in Human-Computer Interaction. ACM Trans. Comput.-Hum. Interact. (mar 2022). https://doi.org/10.1145/3529225
[66]
Arthur Kramer and John Spinks. 1991. Capacity views of human information processing. (1991).
[67]
Daniel Kramer. 2007. Predictions of performance by EEG and skin conductance. Indiana undergraduate journal of cognitive science 2 (2007), 3--13.
[68]
Nilli Lavie. 1995. Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human perception and performance 21, 3 (1995), 451. https://doi.org/10.1037/0096--1523.21.3.451
[69]
Nilli Lavie, Tony Ro, and Charlotte Russell. 2003. The role of perceptual load in processing distractor faces. Psychological science 14, 5 (2003), 510--515. https://doi.org/10.1111/1467--9280.03453
[70]
Jaewook Lee, Fanjie Jin, Younsoo Kim, and David Lindlbauer. 2022. User Preference for Navigation Instructions in Mixed Reality. In 2022 IEEE Conf. on Virtual Reality and 3D User Interfaces (VR). IEEE, NY, NY, USA, 802--811. https://doi.org/10.1109/VR51125.2022.00102
[71]
Jingyi Li, Ceenu George, Andrea Ngao, Kai Holländer, Stefan Mayer, and Andreas Butz. 2021. Rear-seat productivity in virtual reality: Investigating vr interaction in the confined space of a car. Multimodal Technologies and Interaction 5, 4 (2021), 15. https://doi.org/10.3390/mti5040015
[72]
Jingyi Li, Luca Woik, and Andreas Butz. 2022. Designing Mobile MR Workspaces: Effects of Reality Degree and Spatial Configuration During Passenger Productivity in HMDs. Proc. ACM Hum.-Comput. Interact. 6, MHCI, Article 181 (sep 2022), 17 pages. https://doi.org/10.1145/3546716
[73]
CL Lim, RJ Barry, E Gordon, A Sawant, C Rennie, and C Yiannikas. 1996. The relationship between quantified EEG and skin conductance level. Int. Journal of Psychophysiology 21, 2--3 (1996), 151--162. https://doi.org/10.1016/0167--8760(95)00049--6
[74]
JJ-W Lin, Henry Been-Lirn Duh, Donald E Parker, Habib Abi-Rached, and Thomas A Furness. 2002. Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. In Proceedings ieee virtual reality 2002. IEEE, NY, NY, USA, 164--171. https://doi.org/10.1109/VR.2002.996519
[75]
Joan Llobera, Bernhard Spanlang, Giulio Ruffini, and Mel Slater. 2010. Proxemics with multiple dynamic characters in an immersive virtual environment. ACM Transactions on Applied Perception (TAP) 8, 1 (2010), 1--12. https://doi.org/10.1145/1857893.1857896
[76]
Elizabeth S Lorenc, Remington Mallett, and Jarrod A Lewis-Peacock. 2021. Distraction in visual working memory: Resistance is not futile. Trends in cognitive sciences 25, 3 (2021), 228--239. https://doi.org/10.1016/j.tics.2020.12.004
[77]
Svein Magnussen, Mark W Greenlee, Rolf Asplund, and Stein Dyrnes. 1991. Stimulus-specific mechanisms of visual short-term memory. Vision research 31, 7--8 (1991), 1213--1219. https://doi.org/10.1016/0042--6989(91)90046--8
[78]
Elisa Magosso, Francesca De Crescenzio, Giulia Ricci, Sergio Piastra, and Mauro Ursino. 2019. EEG alpha power is modulated by attentional changes during cognitive tasks and virtual reality immersion. Computational intelligence and neuroscience 2019 (2019). https://doi.org/10.1155/2019/7051079
[79]
Dominique Makowski, Tam Pham, Zen J Lau, Jan C Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and SH Chen. 2021. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior research methods 53, 4 (2021), 1689--1696. https://doi.org/10.3758/s13428-020-01516-y
[80]
Guido Makransky, Thomas S Terkildsen, and Richard E Mayer. 2019. Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction 60 (2019), 225--236. https://doi.org/10.1016/j.learninstruc.2017.12.007
[81]
Hadas Marciano and Yaffa Yeshurun. 2017. Large inter-individual and intra-individual variability in the effect of perceptual load. PLoS One 12, 4 (2017), e0175060. https://doi.org/10.1371/journal.pone.0175060
[82]
Nicolas Martin, Nicolas Mathieu, Nico Pallamin, Martin Ragot, and Jean-Marc Diverrez. 2020. Virtual reality sickness detection: An approach based on physiological signals and machine learning. In 2020 IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR). IEEE, NY, NY, USA, 387--399. https://doi.org/10.1109/ISMAR50242.2020.00065
[83]
Matteo Marucci, Gianluca Di Flumeri, Gianluca Borghini, Nicolina Sciaraffa, Michele Scandola, Enea Francesco Pavone, Fabio Babiloni, Viviana Betti, and Pietro Aricò. 2021. The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence. Scientific Reports 11, 1 (2021), 1--15. https://doi.org/10.1038/s41598-021--84196--8
[84]
Alireza Mazloumi Gavgani, Deborah M Hodgson, and Eugene Nalivaiko. 2017. Effects of visual flow direction on signs and Symp.toms of cybersickness. PloS one 12, 8 (2017), e0182790. https://doi.org/10.1371/journal.pone.0182790
[85]
Michael E McCauley and Thomas J Sharkey. 1992. Cybersickness: Perception of self-motion in virtual environments. Presence: Teleoperators & Virtual Environments 1, 3 (1992), 311--318. https://doi.org/10.1162/pres.1992.1.3.311
[86]
Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays. In Proceedings of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). ACM, NY, NY, USA, 2143--2152. https://doi.org/10.1145/2702123.2702382
[87]
DJ McKeefry, MP Burton, and C Vakrou. 2007. Speed selectivity in visual short term memory for motion. Vision research 47, 18 (2007), 2418--2425. https://doi.org/10.1016/j.visres.2007.05.011
[88]
Kathryn M McMillan, Angela R Laird, Suzanne T Witt, and M Elizabeth Meyerand. 2007. Self-paced working memory: Validation of verbal variations of the n-back paradigm. Brain research 1139 (2007), 133--142. https://doi.org/10.1016/j.brainres.2006.12.058
[89]
Fiona McNab and Raymond J Dolan. 2014. Dissociating distractor-filtering at encoding and during maintenance. Journal of Experimental Psychology: Human Perception and Performance 40, 3 (2014), 960. https://doi.org/10.1037/a0036013
[90]
Bruce Mehler, Bryan Reimer, Joseph F Coughlin, and Jeffery A Dusek. 2009. Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transportation research record 2138, 1 (2009), 6--12. https://doi.org/10.3141/2138-02
[91]
Abdul Momin and Sudip Sanyal. 2019. Analysis of Electrodermal Activity Signal Collected During Visual Attention Oriented Tasks. IEEE Access 7 (2019), 88186--88195. https://doi.org/10.1109/ACCESS.2019.2925933
[92]
Andreas Mühlberger, Martin J Herrmann, Georg Wiedemann, Heiner Ellgring, and Paul Pauli. 2001. Repeated exposure of flight phobics to flights in virtual reality. Behaviour research and therapy 39, 9 (2001), 1033--1050. https://doi.org/10.1016/S0005--7967(00)00076-0
[93]
Michael Nelson and Christos Mousas. 2020. A Virtual Reality Framework for Human-Virtual Crowd Interaction Studies. In 2020 IEEE Int. Conf. on Artificial Intelligence and Virtual Reality (AIVR). IEEE, NY, NY, USA, 209--213. https://doi.org/10.1109/AIVR50618.2020.00043
[94]
Mansoor Niaz and Robert H Logie. 1993. Working memory, mental capacity and science education: towards an understanding of the "working memory overload hypothesis'. Oxford Review of Education 19, 4 (1993), 511--525. https://doi.org/10.1080/0305498930190407
[95]
Reiner Nikula. 1991. Psychological correlates of nonspecific skin conductance responses. Psychophysiology 28, 1 (1991), 86--90. https://doi.org/10.1111/j.1469--8986.1991.tb03392.x
[96]
Domen Novak, Matja? Mihelj, and Marko Munih. 2012. A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing. Interacting with computers 24, 3 (2012), 154--172. https://doi.org/10.1016/j.intcom.2012.04.003
[97]
Klaus Oberauer, Stephan Lewandowsky, Edward Awh, Gordon DA Brown, Andrew Conway, Nelson Cowan, Christopher Donkin, Simon Farrell, Graham J Hitch, Mark J Hurlstone, et al. 2018. Benchmarks for models of short-term and working memory. Psychological bulletin 144, 9 (2018), 885. https://doi.org/10.1037/bul0000153
[98]
Aude Olivia, Michael L Mack, Mochan Shrestha, and Angela Peeper. 2004. Identifying the perceptual dimensions of visual complexity of scenes. In Proceedings of the annual meeting of the cognitive science society, Vol. 26.
[99]
Bettina Olk, Alina Dinu, David J Zielinski, and Regis Kopper. 2018. Measuring visual search and distraction in immersive virtual reality. Royal Society open science 5, 5 (2018), 172331. https://doi.org/10.1098/rsos.172331
[100]
Avinash Parnandi and Ricardo Gutierrez-Osuna. 2015. A comparative study of game mechanics and control laws for an adaptive physiological game. Journal on Multimodal User Interfaces 9, 1 (2015), 31--42. https://doi.org/10.1007/s12193-014-0159-y
[101]
Thomas D Parsons and James L Reinebold. 2012. Adaptive virtual environments for neuropsychological assessment in serious games. IEEE Transactions on Consumer Electronics 58, 2 (2012), 197--204. https://doi.org/10.1109/TCE.2012.6227413
[102]
Andrea C Pierno, Andrea Caria, Scott Glover, and Umberto Castiello. 2005. Effects of increasing visual load on aurally and visually guided target acquisition in a virtual environment. Applied ergonomics 36, 3 (2005), 335--343. https://doi.org/10.1016/j.apergo.2004.11.002
[103]
Alan T Pope, Edward H Bogart, and Debbie S Bartolome. 1995. Biocybernetic system evaluates indices of operator engagement in automated task. Biological psychology 40, 1--2 (1995), 187--195. https://doi.org/10.1016/0301-0511(95)05116--3
[104]
Eric D Ragan, Doug A Bowman, Regis Kopper, Cheryl Stinson, Siroberto Scerbo, and Ryan P McMahan. 2015. Effects of field of view and visual complexity on virtual reality training effectiveness for a visual scanning task. IEEE transactions on visualization and computer graphics 21, 7 (2015), 794--807. https://doi.org/10.1109/VR.2002.996519
[105]
Eric Redlinger, Bernhard Glas, and Yang Rong. 2021. Enhanced Cognitive Training using Virtual Reality: Examining a Memory Task Modified for Use in Virtual Environments. In 2021 5th Int. Conf. on Artificial Intelligence and Virtual Reality (AIVR). 1--8. https://doi.org/10.1145/3480433.3480435
[106]
M. Richter, G.H.E. Gendolla, and R.A. Wright. 2016. Three Decades of Research on Motivational Intensity Theory: WhatWe Have Learned About Effort and WhatWe Still Don't Know. Advances in Motivation Science, Vol. 3. Elsevier, 149--186. https://doi.org/10.1016/bs.adms.2016.02.001
[107]
Luana L Righi, Gilberto F Xavier, Marcus VC Baldo, and Hamilton Haddad. 2019. Responses to auditory distractors during a class depend on participant's enjoyment: An eye-tracking and skin conductance study. Psychology & Neuroscience 12, 4 (2019), 473. https://doi.org/10.1037/pne0000178
[108]
Radiah Rivu, Yumeng Zhou, Robin Welsch, Ville Mäkelä, and Florian Alt. 2021. When friends become strangers: Understanding the influence of avatar gender on interpersonal distance in virtual reality. In IFIP Conf. on Human-Computer Interaction. Springer, 234--250. https://doi.org/10.1007/978--3-030--85607--6_16
[109]
Ruth Rosenholtz, Yuanzhen Li, and Lisa Nakano. 2007. Measuring visual clutter. Journal of vision 7, 2 (2007), 17--17. https://doi.org/10.1167/7.2.17
[110]
Jacqueline A Rushby, Robert J Barry, Adam R Clarke, and Mohammad VaezMousavi. 2007. Arousal and activation effects on physiological and behavioral responding during a continuous performance task. (2007).
[111]
Tsugunosuke Sakai, Harunya Tamaki, Yosuke Ota, Ryohei Egusa, Shigenori Imagaki, Fusako Kusunoki, Masanori Sugimoto, and Hiroshi Mizoguchi. 2017. Eda-based estimation of visual attention by observation of eye blink frequency. Int. Journal on Smart Sensing and Intelligent Systems 10, 2 (2017), 1--12. https://doi.org/10.21307/ijssis-2017--212
[112]
Rajwant Sandhu and Benjamin James Dyson. 2016. Cross-modal perceptual load: the impact of modality and individual differences. Experimental Brain Research 234 (2016), 1279--1291. https://doi.org/10.1007/s00221-015--4517-0
[113]
RL Schneider. 1987. A mathematical-model of human-skin conductance. In Psychophysiology, Vol. 24. SOC PSYCHOPHYSIOL RES 1010 VERMONT AVE NW SUITE 1100, WASHINGTON, DC 20005, 610--610.
[114]
Alexander Skulmowski, Steve Nebel, Martin Remmele, and Günter Daniel Rey. 2021. Is a preference for realism really naive after all? A cognitive model of learning with realistic visualizations. Educational Psychology Review (2021), 1--27. https://doi.org/10.1007/s10648-021-09638--1
[115]
Evgeny N Sokolov, John A Spinks, Risto Näätänen, and Heikki Lyytinen. 2002. The orienting response in information processing. Lawrence Erlbaum Associates Publishers. https://doi.org/10.4324/9781410601490
[116]
Jan-Philipp Stauffert, Florian Niebling, and Marc Erich Latoschik. 2020. Latency and cybersickness: impact, causes, and measures. A review. Frontiers in Virtual Reality 1 (2020), 582204. https://doi.org/10.3389/frvir.2020.582204
[117]
Roy Stripling, Joseph T. Coyne, Anna Cole, Daniel Afergan, Raymond L. Barnes, Kelly A. Rossi, Leah M. Reeves, and Dylan D. Schmorrow. 2007. Automated SAF Adaptation Tool (ASAT). In Foundations of Augmented Cognition, Dylan D. Schmorrow and Leah M. Reeves (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 346--353. https://doi.org/10.1007/978--3--540--73216--7_39
[118]
Philipp Sykownik and Maic Masuch. 2020. The experience of social touch in multi-user virtual reality. In 26th ACM Symp. on virtual reality software and technology. 1--11. https://doi.org/10.1145/3385956.3418944
[119]
Margaret Toms, Neil Morris, and Peter Foley. 1994. Characteristics of visual interference with visuospatial working memory. British Journal of Psychology 85, 1 (1994), 131--144. https://doi.org/10.1111/j.2044--8295.1994.tb02513.x
[120]
Samaikya Valluripally, Vaibhav Akashe, Michael Fisher, David Falana, Khaza Anuarul Hoque, and Prasad Calyam. 2021. Rule-based Adaptations to Control Cybersickness in Social Virtual Reality Learning Environments. In 2021 8th Int. Conf. on Future Internet of Things and Cloud (FiCloud). IEEE, NY, NY, USA, 350--358. https://doi.org/10.1109/FiCloud49777.2021.00057
[121]
Peter H Venables and Margaret J Christie. 1980. Electrodermal activity. Techniques in psychophysiology 54, 3 (1980).
[122]
Matias Volante, Sabarish V Babu, Himanshu Chaturvedi, Nathan Newsome, Elham Ebrahimi, Tania Roy, Shaundra B Daily, and Tracy Fasolino. 2016. Effects of virtual human appearance fidelity on emotion contagion in affective inter-personal simulations. IEEE transactions on visualization and computer graphics 22, 4 (2016), 1326--1335. https://doi.org/10.1109/TVCG.2016.2518158
[123]
Jialin Wang, Hai-Ning Liang, Diego Monteiro, Wenge Xu, and Jimin Xiao. 2022. Real-time Prediction of Simulator Sickness in Virtual Reality Games. IEEE Transactions on Games (2022). https://doi.org/10.1109/TG.2022.3178539
[124]
Frank H Wilhelm, Monique C Pfaltz, James J Gross, Iris B Mauss, Sun I Kim, and Brenda K Wiederhold. 2005. Mechanisms of virtual reality exposure therapy: The role of the behavioral activation and behavioral inhibition systems. Applied psychophysiology and biofeedback 30, 3 (2005), 271--284. https://doi.org/10.1007/s10484-005--6383--1
[125]
Rex A Wright. 2008. Refining the prediction of effort: Brehm's distinction between potential motivation and motivation intensity. Social and Personality Psychology Compass 2, 2 (2008), 682--701. https://doi.org/10.1111/j.1751--9004.2008.00093.x
[126]
Dongrui Wu, Christopher G Courtney, Brent J Lance, Shrikanth S Narayanan, Michael E Dawson, Kelvin S Oie, and Thomas D Parsons. 2010. Optimal arousal identification and classification for affective computing using physiological signals: Virtual reality stroop task. IEEE Transactions on Affective Computing 1, 2 (2010), 109--118. https://doi.org/10.1109/T-AFFC.2010.12
[127]
Jong H Yoon, Clayton E Curtis, and Mark D'Esposito. 2006. Differential effects of distraction during working memory on delay-period activity in the prefrontal cortex and the visual association cortex. Neuroimage 29, 4 (2006), 1117--1126. https://doi.org/10.1016/j.neuroimage.2005.08.024
[128]
Xuguang Zhang, Xiuxin Yang, Weiguang Zhang, Gongfa Li, and Hui Yu. 2021. Crowd emotion evaluation based on fuzzy inference of arousal and valence. Neurocomputing 445 (2021), 194--205. https://doi.org/10.1016/j.neucom.2021.02.047
[129]
Magdalena Zukowska, Pawel Bun, Filip Górski, and Beata Starzy'ska. 2019. Cyber sickness in industrial virtual reality training. In Advances in Manufacturing II: Volume 1-Solutions for Industry 4.0. Springer, 137--149. https://doi.org/10.1007/978--3-030--18715--6_12

Cited By

View all
  • (2024)Broadening the mind: how emerging neurotechnology is reshaping HCI and interactive system designi-com10.1515/icom-2024-000723:2(165-177)Online publication date: 23-May-2024
  • (2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
  • (2024)Your Eyes on Speed: Using Pupil Dilation to Adaptively Select Speed-Reading Parameters in Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/36765318:MHCI(1-17)Online publication date: 24-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction
Proceedings of the ACM on Human-Computer Interaction  Volume 7, Issue MHCI
MHCI
September 2023
1017 pages
EISSN:2573-0142
DOI:10.1145/3624512
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2023
Published in PACMHCI Volume 7, Issue MHCI

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adaptive systems
  2. electrodermal activity
  3. physiological computing
  4. virtual reality
  5. visual complexity
  6. working memory

Qualifiers

  • Research-article

Funding Sources

  • Deutsche Forschungsgemeinschaft (DFG)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)238
  • Downloads (Last 6 weeks)29
Reflects downloads up to 22 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Broadening the mind: how emerging neurotechnology is reshaping HCI and interactive system designi-com10.1515/icom-2024-000723:2(165-177)Online publication date: 23-May-2024
  • (2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
  • (2024)Your Eyes on Speed: Using Pupil Dilation to Adaptively Select Speed-Reading Parameters in Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/36765318:MHCI(1-17)Online publication date: 24-Sep-2024
  • (2024)Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36765288:MHCI(1-33)Online publication date: 24-Sep-2024
  • (2024)PhysioCHI: Towards Best Practices for Integrating Physiological Signals in HCIExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636286(1-7)Online publication date: 11-May-2024
  • (2024)The Differential Effects of Multisensory Attentional Cues on Task Performance in VR Depending on the Level of Cognitive Load and Cognitive CapacityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337212630:5(2703-2712)Online publication date: May-2024
  • (2024)Contribution & Power of Virtual Reality in Corporate World: A Conceptual Review2024 Third International Conference on Distributed Computing and High Performance Computing (DCHPC)10.1109/DCHPC60845.2024.10454076(1-15)Online publication date: 14-May-2024
  • (2024)Dynamic virtual reality horror sports enhanced by artificial intelligence and player modelingMultimedia Tools and Applications10.1007/s11042-024-18414-683:32(77415-77432)Online publication date: 24-Feb-2024
  • (2023)Cognitive Load Measurement with Physiological Sensors in Virtual Reality during Physical ActivityProceedings of the 29th ACM Symposium on Virtual Reality Software and Technology10.1145/3611659.3615704(1-11)Online publication date: 9-Oct-2023
  • (2023)My Heart Will Go On: Implicitly Increasing Social Connectedness by Visualizing Asynchronous Players’ Heartbeats in VR GamesProceedings of the ACM on Human-Computer Interaction10.1145/36110577:CHI PLAY(976-1001)Online publication date: 4-Oct-2023
  • Show More Cited By

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media