skip to main content
research-article

VibEmoji: Exploring User-authoring Multi-modal Emoticons in Social Communication

Published: 29 April 2022 Publication History

Abstract

Emoticons are indispensable in online communications. With users’ growing needs for more customized and expressive emoticons, recent messaging applications begin to support (limited) multi-modal emoticons:, enhancing emoticons with animations or vibrotactile feedback. However, little empirical knowledge has been accumulated concerning how people create, share and experience multi-modal emoticons in everyday communication, and how to better support them through design. To tackle this, we developed VibEmoji, a user-authoring multi-modal emoticon interface for mobile messaging. Extending existing designs, VibEmoji grants users greater flexibility to combine various emoticons, vibrations, and animations on-the-fly, and offers non-aggressive recommendations based on these components’ emotional relevance. Using VibEmoji as a probe, we conducted a four-week field study with 20 participants, to gain new understandings from in-the-wild usage and experience, and extract implications for design. We thereby contribute to both a novel system and various insights for supporting users’ creation and communication of multi-modal emoticons.

Supplementary Material

Supplemental Materials (3491102.3501940-supplemental-materials.zip)
MP4 File (3491102.3501940-video-figure.mp4)
Video Figure
MP4 File (3491102.3501940-video-preview.mp4)
Video Preview
MP4 File (3491102.3501940-talk-video.mp4)
Talk Video

References

[1]
Jessalyn Alvina, Chengcheng Qu, Joanna McGrenere, and Wendy E. Mackay. 2019. MojiBoard: Generating Parametric Emojis with Gesture Keyboards. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312771
[2]
Apple. 2021. Use Memoji on your iPhone or iPad Pro. https://support.apple.com/en-ca/HT208986
[3]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa arXiv:https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa
[4]
Scott Brave and Andrew Dahley. 1997. InTouch: A Medium for Haptic Interpersonal Communication. In CHI ’97 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia) (CHI EA ’97). Association for Computing Machinery, New York, NY, USA, 363–364. https://doi.org/10.1145/1120212.1120435
[5]
Brian. 2012. PLATO Emoticons, revisited. http://www.platohistory.org/blog/2012/09/plato-emoticons-revisited.html
[6]
Keith Broni. 2021. Emoji Use At All-Time High. https://blog.emojipedia.org/emoji-use-at-all-time-high/
[7]
Jeremy Burge. 2017. 5 Billion Emojis Sent Daily on Messenger. https://blog.emojipedia.org/5-billion-emojis-sent-daily-on-messenger/
[8]
Jessica R. Cauchard, Janette L. Cheng, Thomas Pietrzak, and James A. Landay. 2016. ActiVibe: Design and Evaluation of Vibrations for Progress Monitoring. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 3261–3271. https://doi.org/10.1145/2858036.2858046
[9]
Zekun Chang, Tung D. Ta, Koya Narumi, Heeju Kim, Fuminori Okuya, Dongchi Li, Kunihiro Kato, Jie Qi, Yoshinobu Miyamoto, Kazuya Saito, and Yoshihiro Kawahara. 2020. Kirigami Haptic Swatches: Design Methods for Cut-and-Fold Haptic Feedback Mechanisms. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376655
[10]
Zhenpeng Chen, Yanbin Cao, Huihan Yao, Xuan Lu, Xin Peng, Hong Mei, and Xuanzhe Liu. 2021. Emoji-Powered Sentiment and Emotion Detection from Software Developers’ Communication Data. ACM Trans. Softw. Eng. Methodol. 30, 2, Article 18 (Jan. 2021), 48 pages. https://doi.org/10.1145/3424308
[11]
Marcel Danesi. 2016. The semiotics of emoji: The rise of visual language in the age of the internet. Bloomsbury Publishing, London, UK.
[12]
Paul Ekman. 1992. Are there basic emotions?Psychological Review 3, 99 (1992), 550–553.
[13]
Abdallah El Ali, Torben Wallbaum, Merlin Wasmann, Wilko Heuten, and Susanne CJ Boll. 2017. Face2Emoji: Using Facial Emotional Expressions to Filter Emojis. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems(Denver, Colorado, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 1577–1584. https://doi.org/10.1145/3027063.3053086
[14]
Mario J Enriquez and Karon E MacLean. 2003. The hapticon editor: a tool in support of haptic communication research. In 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings. IEEE, IEEE, 356–362.
[15]
Facebook. 2021. Messenger. https://www.messenger.com/
[16]
Scott Fahlman. Retrieved 2021. Smiley Lore :-). https://www.cs.cmu.edu/~sef/sefSmiley.htm
[17]
Yunhe Feng, Zheng Lu, Wenjun Zhou, Zhibo Wang, and Qing Cao. 2020. New Emoji Requests from Twitter Users: When, Where, Why, and What We Can Do About Them. Trans. Soc. Comput. 3, 2, Article 8 (apr 2020), 25 pages. https://doi.org/10.1145/3370750
[18]
Google. 2021. Gboard. https://blog.google/products/android/emoji-kitchen-new-mashups-mixing-experience/
[19]
Carla F Griggio, Joanna McGrenere, and Wendy E Mackay. 2019. Customizations and Expression Breakdowns in Ecosystems of Communication Apps. Proc. ACM Hum.-Comput. Interact. 3 (11 2019), 26 pages. Issue CSCW. https://doi.org/10.1145/3359128
[20]
Carla F. Griggio, Arissa J. Sato, Wendy E. Mackay, and Koji Yatani. 2021. Mediating Intimacy with DearBoard: A Co-Customizable Keyboard for Everyday Messaging. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 342, 16 pages. https://doi.org/10.1145/3411764.3445757
[21]
Loni Hagen, Mary Falling, Oleksandr Lisnichenko, AbdelRahim A. Elmadany, Pankti Mehta, Muhammad Abdul-Mageed, Justin Costakis, and Thomas E. Keller. 2019. Emoji Use in Twitter White Nationalism Communication. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing(Austin, TX, USA) (CSCW ’19). Association for Computing Machinery, New York, NY, USA, 201–205. https://doi.org/10.1145/3311957.3359495
[22]
Chris Harrison, Gary Hsieh, Karl D.D. Willis, Jodi Forlizzi, and Scott E. Hudson. 2011. Kineticons: Using Iconographic Motion in Graphical User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 1999–2008. https://doi.org/10.1145/1978942.1979232
[23]
Jiaxiong Hu, Qianyao Xu, Limin Paul Fu, and Yingqing Xu. 2019. Emojilization: An Automated Method For Speech to Emoji-Labeled Text. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3313071
[24]
Huawei. 2021. MeeTime. https://consumer.huawei.com/cn/mobileservices/meetime/
[25]
Ali Israr, Siyan Zhao, and Oliver Schneider. 2015. Exploring Embedded Haptics for Social Networking and Interactions. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI EA ’15). Association for Computing Machinery, New York, NY, USA, 1899–1904. https://doi.org/10.1145/2702613.2732814
[26]
Jialun ”Aaron” Jiang, Jed R. Brubaker, and Casey Fiesler. 2017. Understanding Diverse Interpretations of Animated GIFs. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 1726–1732. https://doi.org/10.1145/3027063.3053139
[27]
Ryan Kelly and Leon Watts. 2015. Characterising the inventive appropriation of emoji as relationally meaningful in mediated close personal relationships. Experiences of technology appropriation: Unanticipated users, usage, circumstances, and design 2(2015), 7 pages.
[28]
Sujay Khandekar, Joseph Higg, Yuanzhe Bian, Chae Won Ryu, Jerry O. Talton Iii, and Ranjitha Kumar. 2019. Opico: A Study of Emoji-First Communication in a Mobile Social App. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 450–458. https://doi.org/10.1145/3308560.3316547
[29]
Erin Kim and Oliver Schneider. 2020. Defining Haptic Experience: Foundations for Understanding, Communicating, and Evaluating HX. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376280
[30]
Jihun Kim, Amitash ojha, Yongsik Jin, and Minho Lee. 2015. Pictogram Generator from Korean Sentences Using Emoticon and Saliency Map. In Proceedings of the 3rd International Conference on Human-Agent Interaction (Daegu, Kyungpook, Republic of Korea) (HAI ’15). Association for Computing Machinery, New York, NY, USA, 259–262. https://doi.org/10.1145/2814940.2814991
[31]
Philippe Kimura-Thollander and Neha Kumar. 2019. Examining the ”Global” Language of Emojis: Designing for Cultural Representation. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300725
[32]
Yehuda Koren and Robert Bell. 2015. Advances in collaborative filtering. Springer, New York, NY, USA, 77–118.
[33]
Miki Liu, Austin Wong, Ruhi Pudipeddi, Betty Hou, David Wang, and Gary Hsieh. 2018. ReactionBot: Exploring the Effects of Expression-Triggered Emoji in Text Messages. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 110 (Nov. 2018), 16 pages. https://doi.org/10.1145/3274379
[34]
Albert Mehrabian. 1980. Basic dimensions for a general psychological theory implications for personality, social, environmental, and developmental studies. Cambridge: Oelgeschlager, Gunn & Hain, Cambridge, UK.
[35]
Paritosh Mittal, Kunal Aggarwal, Pragya Paramita Sahu, Vishal Vatsalya, Soumyajit Mitra, Vikrant Singh, Viswanath Veera, and Shankar M. Venkatesan. 2020. Photo-Realistic Emoticon Generation Using Multi-Modal Input. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 254–258. https://doi.org/10.1145/3377325.3377533
[36]
Joe Mullenbach, Craig Shultz, J. Edward Colgate, and Anne Marie Piper. 2014. Exploring Affective Communication through Variable-Friction Surface Haptics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 3963–3972. https://doi.org/10.1145/2556288.2557343
[37]
Hideyuki Nakanishi, Kazuaki Tanaka, and Yuya Wada. 2014. Remote Handshaking: Touch Enhances Video-Mediated Social Telepresence. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2143–2152. https://doi.org/10.1145/2556288.2557169
[38]
Young-Woo Park and Tek-Jin Nam. 2013. POKE: A New Way of Sharing Emotional Touches during Phone Conversations. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 2859–2860. https://doi.org/10.1145/2468356.2479548
[39]
Anand Rajaraman and Jeffrey David Ullman. 2011. Data Mining. Cambridge University Press, Cambridge, UK, 1–17. https://doi.org/10.1017/CBO9781139058452.002
[40]
David Rodrigues, Marília Prada, Rui Gaspar, Margarida V Garrido, and Diniz Lopes. 2018. Lisbon Emoji and Emoticon Database (LEED): Norms for emoji and emoticons in seven evaluative dimensions. Behavior research methods 50, 1 (2018), 392–405.
[41]
James A Russell and Lisa Feldman Barrett. 1999. Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant.Journal of personality and social psychology 76, 5(1999), 805.
[42]
Oliver S. Schneider, Hasti Seifi, Salma Kashani, Matthew Chun, and Karon E. MacLean. 2016. HapTurk: Crowdsourcing Affective Ratings of Vibrotactile Icons. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 3248–3260. https://doi.org/10.1145/2858036.2858279
[43]
Hasti Seifi, Mattew Chun, and Karon E. Maclean. 2018. Toward Affective Handles for Tuning Vibrations. ACM Trans. Appl. Percept. 15, 3, Article 22 (jul 2018), 23 pages. https://doi.org/10.1145/3230645
[44]
Hasti Seifi, Kailun Zhang, and Karon E. MacLean. 2015. VibViz: Organizing, visualizing and navigating vibration libraries. In 2015 IEEE World Haptics Conference (WHC). IEEE, New York, NY, USA, 254–259. https://doi.org/10.1109/WHC.2015.7177722
[45]
Andreas Sonderegger, Klaus Heyden, Alain Chavaillaz, and Juergen Sauer. 2016. AniSAM & AniAvatar: Animated Visualizations of Affective States. Association for Computing Machinery, New York, NY, USA, 4828–4837. https://doi.org/10.1145/2858036.2858365
[46]
Telegram. 2021. Telegram. https://telegram.org/
[47]
Tencent. 2021. WeChat. https://www.wechat.com/
[48]
Garreth W. Tigwell, Benjamin M. Gorman, and Rachel Menzies. 2020. Emoji Accessibility for Visually Impaired People. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376267
[49]
Maggie Tillman. 2021. What are Animoji? How to create and use Apple’s animated emoji. https://www.pocket-lint.com/apps/news/apple/142230-what-are-animoji-how-to-use-apple-s-animated-emoji
[50]
Yuki Urabe, Rafal Rzepka, and Kenji Araki. 2013. Emoticon Recommendation System for Effective Communication. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining(Niagara, Ontario, Canada) (ASONAM ’13). Association for Computing Machinery, New York, NY, USA, 1460–1461. https://doi.org/10.1145/2492517.2492594
[51]
Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image and vision computing 27, 12 (2009), 1743–1759.
[52]
Rongrong Wang, Francis Quek, Deborah Tatar, Keng Soon Teh, and Adrian Cheok. 2012. Keep in Touch: Channel, Expectation and Experience. Association for Computing Machinery, New York, NY, USA, 139–148. https://doi.org/10.1145/2207676.2207697
[53]
Sarah Wiseman and Sandy J. J. Gould. 2018. Repurposing Emoji for Personalised Communication: Why images/emojipizza Means “I Love You”. Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3173574.3173726
[54]
Koji Yatani and Khai Nhut Truong. 2009. SemFeel: A User Interface with Semantic Tactile Feedback for Mobile Touch-Screen Devices. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST ’09). Association for Computing Machinery, New York, NY, USA, 111–120. https://doi.org/10.1145/1622176.1622198
[55]
Amy X. Zhang, Michele Igo, Marc Facciotti, and David Karger. 2017. Using Student Annotated Hashtags and Emojis to Collect Nuanced Affective States. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (Cambridge, Massachusetts, USA) (L@S ’17). Association for Computing Machinery, New York, NY, USA, 319–322. https://doi.org/10.1145/3051457.3054014
[56]
Mingrui Ray Zhang, Ruolin Wang, Xuhai Xu, Qisheng Li, Ather Sharif, and Jacob O. Wobbrock. 2021. Voicemoji: Emoji Entry Using Voice for Visually Impaired People. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 37, 18 pages. https://doi.org/10.1145/3411764.3445338
[57]
Zhuoming Zhang, Jessalyn Alvina, Robin Héron, Stéphane Safin, Françoise Détienne, and Eric Lecolinet. 2021. Touch without Touching: Overcoming Social Distancing in Semi-Intimate Relationships with SansTouch. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 651, 13 pages. https://doi.org/10.1145/3411764.3445612
[58]
Zhuoming Zhang, Robin Héron, Eric Lecolinet, Françoise Detienne, and Stéphane Safin. 2019. VisualTouch: Enhancing Affective Touch Communication with Multi-Modality Stimulation. In 2019 International Conference on Multimodal Interaction (Suzhou, China) (ICMI ’19). Association for Computing Machinery, New York, NY, USA, 114–123. https://doi.org/10.1145/3340555.3353733
[59]
Rui Zhou, Jasmine Hentschel, and Neha Kumar. 2017. Goodbye Text, Hello Emoji: Mobile Communication on WeChat in China. Association for Computing Machinery, New York, NY, USA, 748–759. https://doi.org/10.1145/3025453.3025800

Cited By

View all
  • (2024)EmoWear: Exploring Emotional Teasers for Voice Message Interaction on SmartwatchesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642101(1-16)Online publication date: 11-May-2024
  • (2024)RecHap: An Interactive Recommender System for Navigating a Large Number of Mid-Air Haptic DesignsIEEE Transactions on Haptics10.1109/TOH.2023.327681217:2(165-176)Online publication date: Apr-2024
  • (2024)EmojiChat: Toward Designing Emoji-Driven Social Interaction in VR MuseumsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2387902(1-17)Online publication date: 16-Aug-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
April 2022
10459 pages
ISBN:9781450391573
DOI:10.1145/3491102
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Social communication
  2. animation
  3. emoticons
  4. emotional expression
  5. haptic feedback
  6. mobile interfaces.
  7. multi-modal interaction

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

CHI '22
Sponsor:
CHI '22: CHI Conference on Human Factors in Computing Systems
April 29 - May 5, 2022
LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)320
  • Downloads (Last 6 weeks)28
Reflects downloads up to 22 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)EmoWear: Exploring Emotional Teasers for Voice Message Interaction on SmartwatchesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642101(1-16)Online publication date: 11-May-2024
  • (2024)RecHap: An Interactive Recommender System for Navigating a Large Number of Mid-Air Haptic DesignsIEEE Transactions on Haptics10.1109/TOH.2023.327681217:2(165-176)Online publication date: Apr-2024
  • (2024)EmojiChat: Toward Designing Emoji-Driven Social Interaction in VR MuseumsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2387902(1-17)Online publication date: 16-Aug-2024
  • (2024)I'm not upset–I get itInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103218185:COnline publication date: 25-Jun-2024
  • (2023)Affective Affordance of Message Balloon Animations: An Early Exploration of AniBalloonsCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3607017(138-143)Online publication date: 14-Oct-2023
  • (2023)“Hey, can we talk?”: Exploring How Revealing Implicit Emotional Responses Tangibly Could Foster Empathy During Mobile TextingProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3573124(1-7)Online publication date: 26-Feb-2023
  • (2023)Mediated Social Touch With Mobile Devices: A Review of Designs and EvaluationsIEEE Transactions on Haptics10.1109/TOH.2023.332750616:4(785-804)Online publication date: 25-Oct-2023
  • (2022)Enhancing Social Messaging with Mediated Social TouchInternational Journal of Human–Computer Interaction10.1080/10447318.2022.214888340:7(1669-1688)Online publication date: 7-Dec-2022

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media