Human-Robot Interaction Conversational User Enjoyment Scale (HRI CUES)

A popular definition of enjoyment is being in the state of flow [17]. Flow is defined as the optimal experience, which provides a deep sense of enjoyment. It happens when an individual is fully engaged in a task that provides an optimal amount of challenge and engagement. Flow is characterized by a set of factors, such as a fading sense of ‘self’, a sense that duration is altered, and deep and effortless involvement in the task. The theory states that enjoyment is not obtained in a relaxed state, that it is necessary to be challenged, and links repeated experiences of flow to mastery of a skill.

Another theory derives from the flow theory to define true fun as the experience that occurs when a person is experiencing flow, playfulness, and connection all at the same time [18]. If one or two of the three components that constitute true fun are present, the experience will make a person feel joy or satisfaction, but not true fun. Similarly to what characterizes being in a state of flow, experiencing true fun is characterized by losing track of time, letting go, and being completely present in the moment, with the addition of laughter, feeling free, a sense of child-like excitement, and joy.

While both flow and true fun emphasize high engagement through optimal challenge, other theories recognize enjoyment in less intense states. For instance, flow-like states can be differentiated from the overall positive valence of an experience [19]. This relates to the circumplex model of emotion [20], which features arousal (low to high) on one axis and pleasure or valence (negative to positive) on the other axis [21]. Certain theories of enjoyment focus on high valence values that can contain lower-engagement positive emotions (e.g., content or calm), while others prioritize high-arousal states (e.g., excitement) [19]. The ‘happy’ emotion in the circumplex model reflects a balance of high arousal and positive valence. Our work aligns with the theory of true fun in seeking both higher levels of arousal and valence, while incorporating other elements for classifying lower levels of enjoyment. Casual, open-domain conversations between older adults and robots may involve aspects of both arousal and positive emotions. As such, both factors will be relevant to our holistic model of enjoyment.

Enjoyment is generally evaluated through self-reported questionnaires tailored to the specific application domain. For instance, the Quality of Life Enjoyment and Satisfaction Questionnaire (Q-LES-Q) [22] and Physical Activity Enjoyment Scale (PACES) [23] are used in healthcare applications. In Human-Computer Interaction (HCI) and HRI research, enjoyment is not often the primary focus for evaluating user perceptions, but is typically included as part of a more comprehensive model [24]. Enjoyment frequently appears as a self-reported measure, either as a construct within established models like the Unified Theory of Acceptance and Use of Technology (UTAUT) [25], or as single-item measures in custom questionnaires (e.g.,“Did you feel fun?” [26], “Was playing with the robot enjoyable/not enjoyable?” [27]). Enjoyment was found to be highly correlated with ‘satisfying’, ‘entertaining’, ‘exciting’,‘fun’, and ‘interesting’ in HRI [28]. Technology Acceptance Model (TAM) [29] was also adapted to HRI by incorporating measures of affect and cognition to improve its accuracy in explaining technology adoption, and this adaptation included questions about perceived enjoyment [30].

User enjoyment has been shown to correlate with the intention to use a robot among older adults [1], highlighting its importance for long-term interactions. The Almere model [31] is an extended version of the UTAUT that is widely used in research on robots for older adults [32]. It incorporates enjoyment, social interaction, and social influence as factors mediating the acceptance and intention to use robots.

While user enjoyment is commonly measured through self-reporting in HRI, it has several limitations, such as conforming to perceived norms or researcher expectations or the (in)ability to recall the events and report correctly from memory [33, 14]. In addition, it is often desirable to estimate what a user is feeling by assessing it from an external perspective when self-reporting is not possible or the goal is to automate behavior at the dialogue exchange level during interactions in real-time. External assessments and self-reporting are not mutually exclusive and can instead complement each other. The results of one method can even be used to validate the other. While prior research used smiles and laughter for automatic classification of user enjoyment (within the context of story-telling)  [15], these signals can be contradictory, ambiguous and highly context dependent [34, 35], and enjoyment is often expressed through other multimodal cues.

User satisfaction, which correlates significantly with user enjoyment in conversations [36, 37], is typically evaluated in relation to a task [38, 39]. For instance, the Paradigm for Dialogue System Evaluation (PARADISE) [38] is a framework for evaluating user satisfaction in dialogue systems, based on self-reported satisfaction on a dialogue level and is influenced by other metrics, such as task success in travel booking and accessing emails. Similarly, interaction quality, which evaluates user satisfaction from an external perspective on the exchange level, was analyzed by three annotators in bus schedule inquiries with chatbots over phone calls from a data corpus of 200 dialogues and a lab study with 38 subjects [39]. An autonomous system was developed based on their ratings, which correlated highly with them, but not with users’ self-reported satisfaction scores, which was attributed to the subjectivity of the measure and variability in user perceptions.

User satisfaction has also been measured in the text domain with an annotation protocol similar to our study, based on a dataset of 1000 dialogues between 50 users and a chatbot on attentive listening and conversations about animals [40]. Two annotators were recruited and an annotator alignment session was conducted. The annotators were requested to go through the conversation exchanges once, without going back or looking at the history. The annotators used three metrics of user satisfaction: ‘smoothness of the conversation’, ‘closeness perceived by the user towards the system’, and ‘willingness to continue the conversation’, rated from 1 to 7. However, no agreement was found between the annotators, even after changing the granularity of the scale to two levels, low and high, showing the complexity of evaluating a subjective measure from an external perspective. The study focused on developing an algorithm to measure user satisfaction.

A similar study, also exploring user satisfaction, was conducted to develop a multimodal model based on annotated data [41]. The data corpus consisted of conversations of 60 participants with a virtual agent that was controlled through Wizard of Oz. The wizard annotated the user satisfaction from their perspective after each dialogue exchange, and the user and wizard both provided the satisfaction level of the user at the end of the conversation. In addition, five annotators were recruited to annotate the dialogues from an external perspective. All annotators gave each exchange in the conversation a score in the following metrics: topic continuance (1: ‘strongly change the topic’ to 7: ‘strongly continue the topic’), external sentiment (1: ‘the participants seemed bored with the dialogue’ to 7: ‘participants seemed to enjoy the dialogue’), and self-sentiment (1: ‘want to stop talking/confused about the system utterances’ to 7: ‘enjoy talking/satisfied with the talk’). Notably, the two latter were labeled enjoyment on the upper end of the scale. They received a good level of agreement between the annotators on the exchange level. To annotate the dialogue level, a questionnaire was used with 18 items that were designed to represent three labels: ‘coordinateness’, ‘awkwardness’, and ‘friendliness’. Based on these criteria, the developed multimodal model has outperformed the annotators’ in evaluating user satisfaction at the overall interaction level.

Our user enjoyment scale was developed based on the work of Reimnitz and Rauer [16], which is the only scale that specifically evaluates user enjoyment in conversations from an external perspective. The study assessed the enjoyment in conversations between 64 married couples and compared that to each spouse’s marital happiness. For measuring enjoyment, they developed a scale for observational coding that took into consideration affective signs and the tone of the interaction. The scale ranged from 1 (very low enjoyment) to 7 (very high enjoyment), with 3 as a neutral anchor. Two annotators took into account both affective signs (e.g., mutuality of the interaction, tone of voice, consistent mutual gaze, facial expressions, physical touching, body language) and the tone of the interaction (i.e., neutral, enthusiastic, and delightful) when rating enjoyment. The annotators had good to moderate agreement, with intraclass correlation (ICC) on 20% of the interactions. The study found that couples who displayed high enjoyment in their conversations also reported having a happier marriage. This aligns with prior research in human relationships, which found that mutually enjoyable behavior leads to increased intimacy, trust, security, and satisfaction in long-term relationships [42], signaling that enjoyment could be highly influential in achieving long-term HRIs.

Similar to this scale, our study analyzes dyadic conversations outside of task-oriented settings, but with a focus on evaluating enjoyment in conversations with autonomous robots. Although prior research described in this section evaluated autonomous conversational systems using user satisfaction metrics, which touch upon aspects of enjoyment, these metrics are often more focused on task-specific aspects. In contrast, our study emphasizes understanding enjoyment within daily (open-domain) conversations with robots, based on older adults’ interactions with a companion robot. To the best of our knowledge, no prior study measures enjoyment in open-domain conversations with robots from an external perspective that take into account multimodal aspects of HRI. Our work seeks to bridge this gap by proposing a scale that not only emphasizes the importance and complexity of user enjoyment, but also provides a methodological framework to assess it from an external perspective in other application domains of HRI. The scale aims to provide an additional tool to evaluate enjoyment complementary to self-reported measures from users within conversation exchange and overall interaction levels, with the potential to be used for autonomous systems to adapt the conversations on the fly.

The Furhat robot was employed in the study, featuring a neutral-looking face that underwent user validation before interactions. The robot’s face engine incorporated smiles and eyebrow raises during conversations to enhance naturalness and provide non-verbal feedback to users without context analysis. To further refine the interactions, the robot incorporated subtle behaviors like blinking, eye shifts, and brief gaze aversion while speaking, based on silences in user input.

GPT-3.5 (text-davinci-003, OpenAI) was used for dialogue generation, as it was the most capable LLM at the time (March 2023). Prompting was used to give an empathetic persona to the robot, guiding it to ask open questions, listen actively with follow-up questions, and reflect on situations.

Initially, English was chosen as the communication language for speech recognition (Google Cloud Speech-to-Text), dialogue generation, and synthesis (Amazon Polly) due to the more extensive training data available for LLMs. However, an initial study with Swedish-speaking older adults showed the need for communicating in their native language [12]. Hence, Swedish was used in the follow-up study investigated in this paper. A USB microphone array (Seeed Studio) was used in both studies to obtain clear audio for speech recognition.

While the conversation with the robot was autonomous, a wizard interface was used to start the interaction with the user by entering the participant ID. The initial and final²²2Robot response to start the interaction: “Hello! I am Furhat, the personalized companion robot. What is your name?” Robot response to end the interaction: “I would love to talk more another time, but for the sake of time, I need to say goodbye. Thank you for talking with me. Take care!” robot responses were pre-scripted to ensure that the interaction started and ended the same way for all participants. The rest of the interaction was fully autonomous based on the user’s responses and the responses generated by the LLM. The wizard interface was also used to end the interaction if necessary, i.e., if the participant wants to end the conversation early or an error occurs in the system that requires a restart to continue the conversation where it is left off. A 7-minute timer was set (checked automatically after each user response), after which the robot would say its pre-scripted response to ensure a fair comparison between users.

To understand older adults’ perceived benefits and challenges in interaction with a conversational companion robot, and correspondingly develop a robot that meets their expectations, we conducted preliminary interviews with Swedish-speaking older adults aged 65 and over, in which they talked with a robot autonomously and individually in English. The study had 6 (3 men, 3 women) Swedish-speaking healthy older adults, between 66 to 86 years old ($M=78.3$, $SD=8.3$). However, due to it being the very first evaluation of an LLM on a social robot with older adults, the interactions had a lot of failures that are analyzed thoroughly in [12]. Interaction failures can lead to lower likeability and satisfaction [44] and cause negative tone and emotion in user responses [45]. Thus, to prevent biasing the enjoyment scale solely towards negative experiences, but also have an understanding of user reactions to frequent technical failures in current architectures, we chose the most successful interaction (as defined by the least number of failures and the longest length of interaction) from this study to be included as a basis of alignment for developing the enjoyment scale, as described in Sections IV-B and IV-C. The other criterion was that the interaction contained ‘highs and lows’, that is, the participant reacted positively (e.g., smile, laugh), neutrally, and negatively (e.g., frown, getting impatient) in the video to enable the annotators to understand the spectrum of responses. The corresponding subject (denoted as S0) was an 83-year-old male without prior experience of robots. The participant lived with their partner in their own home. The interaction lasted 13.5 min (53 turns). The video was recorded from a side angle, facing both the participant and the robot. The participant gave informed consent for recording, analysis of data, and anonymized (blurred and without a name) image and video sharing for publications.

Following the preliminary interviews, technical improvements were made for turn-taking, the robot’s persona, and architecture to overcome the interaction failures. Based on the feedback from the interviews, the interaction language was changed to Swedish. Subsequently, four participatory design workshops were conducted with 28 older adults having an autonomous open-domain conversation with the robot individually for approximately 7 minutes. Prior to the robot interactions, the robot’s capabilities were demonstrated through a researcher having a conversation with the robot (2 min), and focus group discussions were made using design scenarios of everyday activities to understand their expectations of companion robots. The researcher(s) were present in the room (to interfere if necessary) during the individual robot interactions. Following the interactions, the participants completed a 68-question Likert scale (1 to 5) questionnaire based on HRI and open-domain dialogue literature. Based on several studies described in Section II, user satisfaction, fun, and interestingness of the conversation were evaluated and categorized under the user enjoyment construct. To account for discomfort in the conversation [46], the strangeness of the conversation was also evaluated, similar to [47]³³3“Did you feel something strange in that dialogue with the robot?” was used in [47]. We adapted it slightly for the Likert scale., by reverse-coding it in analysis:

1. 1.

   I was satisfied with my conversation with the robot.

2. 2.

   It was fun talking to the robot.

3. 3.

   The conversation with the robot was interesting.

4. 4.

   It felt strange talking to the robot.

All interactions with the robot were video-recorded by an external camera facing both the participant and the robot at a side angle, as well as through the robot camera to record the participant’s face. All participants gave informed consent for recording, analysis of data, and anonymized (blurred and without a name) image and video sharing for publications.

Participants were recruited by distributing the invitation at our university’s communication channels, social media, and platforms for gathering senior citizens. In total, 28 (13 men, 15 women) Swedish-speaking healthy older adults between 66 and 86 years old registered as volunteers. We divided this data for the purposes of this study.

Two of the subjects were selected for annotator alignment: a 69-year-old woman (denoted as S1) and a 75-year-old man (denoted as S2). S1 and S2 did not have any prior interaction with a robot. S1 and S2 lived with their partners in their own homes. The selection basis was to find interactions containing a range of ‘highs and lows’, as in the previous study, aiming to complement S0 with interactions with fewer failures. S1’s interaction lasted 7.5 min (27 turns) and S2’s interaction lasted 7.3 min (27 turns).

After the initial alignment of annotators with 3 videos to create HRI CUES (Section IV), the remaining participant interactions (except one due to lack of a side-video) were used for HRI CUES evaluation (Section V). The resulting data consisted of 25 participants’ (12 men, 13 women) interactions, with a mean age of 74.6 ($SD=5.8$). 20 participants had no prior interaction with a robot, and only one had previously talked with a robot. Interaction duration was $M=7.4$ min ($SD=1.5$) with 12 to 29 turns. Each turn lasted 5 to 61 seconds ($M=17.7$, $SD=7.2$). The total duration of the videos was 174 min, corresponding to 590 turns.

We provide the dataset for HRI CUES that includes anonymized transcripts, annotation scores, and self-reported user perceptions online⁴⁴4HRI CUES Dataset: https://paperswithcode.com/dataset/hri-cues-dataset. Videos of the interactions are available upon request, contingent upon a signed agreement to maintain data confidentiality in accordance with General Data Protection Regulation.

Due to the lack of a clear definition of user enjoyment and its subjectivity resulting in high variability in both user perceptions and understanding by a third party, the selection of the right experts as annotators is critical. This is important in general, especially in multidisciplinary fields like HRI, in particular when investigating complex concepts like enjoyment that have relatively different meanings in different academic traditions [49]. The annotators in this study should not only be able to detect and understand multimodal cues exhibited by the users to detect enjoyment, but also align well in their perceptions, such that this measure can be used by other researchers based on their understanding and recommendations. In addition, being well-versed in the literature of the user metric in question (user enjoyment) as well as its difference from similar metrics (e.g., user satisfaction) is necessary to ensure correlations with prior literature, as well as users’ reported perceptions of such metrics.

Familiarity with the target population (participant group) is also important in establishing a better understanding of their needs and reactions. Researchers whose backgrounds focus on HRI with the target population (e.g., older or young adults, children, people with disabilities) can put their interactions in context from social, cognitive, and ethnographic perspectives. Annotators also need to be thoroughly familiar with the socio-cultural background of the participants, as culture affects their perceptions of robots and their interactions [50]. In addition, understanding the nuances and culture-specific idioms (e.g., ‘cold turkey’) and proverbs (e.g., ‘bite the bullet’) in conversations will be easier for an annotator that is a native speaker.

While a combination of all these aspects is difficult to find in a single annotator, a group of annotators would be able to complement each other, such that during alignment and development of the scale, their horizons can be expanded by the perspectives of the others. While typically two annotators establish inter-rater reliability in qualitative analysis, employing three annotators could better suit the complexity of the task, allowing for tie-breaking and alignment across diverse backgrounds [51, 48]. Correspondingly, we selected three annotators ($M_{age}=30$, $SD=2.94$) who are researchers in the mid-late stages of their PhD, with a background in user enjoyment (Annotator 1, denoted as A1), HRI with older adults and cognitive science (A2), and multimodal HRI and cognitive science (A3). The annotators were native Swedish speakers and thoroughly familiar with Swedish culture.

As a starting point to familiarize annotators with the data and develop a user enjoyment scale for conversational HRI, the annotators were given a slightly adapted version of the enjoyment scale by Reimnitz and Rauer⁵⁵51 (very low): no evidence of pleasure. Pair never has fun or enjoys the interaction, although there may be joint interaction. There is no mutual enjoyment of positive affect or negative interaction. 3 (neutral anchor): there is occasional positivity that is not strong or frequently displayed and may be displayed by only one partner towards the other. Pair is doing OK together but without real joy or enthusiasm for their shared interactions. 7 (very high): the pair is very satisfied with the interaction and activity. The couple shows mutual enjoyment in their interaction marked with shared exuberance and/or delight. There is consistent visual regard coupled with affective sharing. [16] on human-human conversations of married couples, in which ‘user’ was used instead of ‘couple’, and references that relate to couples (‘mutual enjoyment’, ‘affective sharing’, and ‘exuberance’) were removed. Instead of a 7-point Likert scale, which may be difficult to align given only the lowest, natural anchor, and highest enjoyment values, a 5-point scale was used:

• •

  1 (very low): no evidence of pleasure. The user never has fun or enjoys the conversation, although there may be joint interaction.

• •

  3 (neutral anchor): there is occasional positivity that is not strong or frequently displayed. The user does not have real joy or enthusiasm for the conversation.

• •

  5 (very high): the user is very satisfied with the conversation. The user shows enjoyment in their conversation marked with enthusiasm and/or delight.

Annotators were encouraged to use the full scale (i.e., not abstaining from giving 1 or 5).

Due to the subjectivity of user enjoyment, it was necessary to establish common grounds on the levels of user enjoyment prior to the annotators analyzing all the robot interactions individually [48]. Thus, three exemplar videos (S0, S1, and S2) as explained in Section III-B were chosen that contain a range of negative and positive responses from the user and a variety of technical failures.

In order to incrementally familiarize the annotators with the modalities that a front view (taken from the robot’s camera) and side view (external camera facing robot and participant) of robot interaction may introduce, the first exemplar video (S0) contained only the side view, the next one (S1) contained only the front view, and finally the third one (S2) contained both views, as shown in Fig. 2 and as used in the final annotations.

Conversational turns (exchanges) were chosen as the basis of annotations, because they were mostly similar in duration for participants, as well as for paving the way for understanding user enjoyment via autonomous systems to adapt and improve the interaction continuously. As such, the segments to be annotated were created automatically based on the turns (Robot-Participant pairs) in manually-corrected (for timing and content) transcripts. All videos started with the robot’s first phrase (greeting of the user). A turn ends (and a new turn starts) when the participant stops speaking, as that holds the potential to be detected by an automatic system for evaluating the turn that could be used to generate a new response.

Annotators were guided to apply the rating scale on a per-turn basis, assessing both the robot’s response and the participant’s subsequent input within each turn. Furthermore, they were tasked with delivering an overarching assessment of enjoyment encompassing the entire interaction, referred to as overall enjoyment. Within this context, annotators were encouraged to provide an in-depth rationale for their ratings, adopting an open-ended approach to offer comprehensive insights. They were asked to elaborate on the aspects and multimodal cues they considered in shaping their evaluations, along with the methodology they employed. Additionally, they were asked to provide whether any challenges or difficulties were encountered while evaluating overall user enjoyment for the interaction and its details. The ratings for each turn were to be recorded within the ELAN file (Fig. 2), while a separate document was designated for annotators to record their overall interaction rating and provide open-ended responses.

Based on their individual annotations of three robot interaction videos (S0-S2), annotators were asked to meet to align themselves to decide more objectively what each level of the scale corresponds to, such that an agreement can be reached for the analysis of the remaining interaction videos. In addition, they were asked to discuss the aspects and multimodal cues used to give the corresponding scores, in a turn-by-turn fashion, as well as the overall user enjoyment.

To facilitate discussions, a list of aspects and multimodal cues from HRI [52, 53, 54, 55], HCI [54], and human-human interaction [56, 57, 58, 59, 16] literature was given to the annotators, which were previously used in affective computing, user engagement, user enjoyment, conversation, and turn-taking analysis, in addition to the principal researcher’s analysis of the challenges of applying LLMs into conversational robots [12]:

• •

  Facial expressions: smile, laughter, frown, rolling eyes, sigh, other expressions (e.g., smirk, squinting eyes, raising eyebrows). Emotion models [60, 20, 61] were described for further context.

• •

  Gaze: Mutual gaze, gaze length, gaze aversion, other gaze targets (e.g., objects, experimenter)

• •

  Body language: Gestures, gesture duration, gesture frequency, gesture intensity, posture, body orientation, head orientation, arm position (e.g., folded/ open), movement, physical contact, pointing, adaptors (e.g., touching hair, bouncing legs), nodding/ head shakes, proximity (distance to the robot)

• •

  Vocal features: Tone, pitch, pace, volume/ loudness, energy

• •

  Dialogue responses: Content, sentiment, length, mirroring, pauses in response, rephrasing/ clarifications, anthropomorphism, disengagement cues (responses that bring the conversation to a halt, e.g., “That is good to know”)

• •

  Conversation: Context, topic, topic initiation, topic closure, topic duration, tone (e.g., neutral, enthusiastic, and delightful), vocal fillers (e.g., “uh”, “erm”), conversation length, repairs (dealing with failures in interaction), referral to previous topics/parts in a conversation, willingness to talk about personal matters, asking questions about the conversation partner (robot), agreement/ disagreement

• •

  Turn-taking: Speaker dominance, willingness to take a turn, interruption, response time, backchannelling

• •

  Interacting with others: Interacting with the experimenter/ third party during the conversation with the robot

Annotators were encouraged to discuss whether they made use of these elements in their analysis, their usefulness and importance in assessing user enjoyment (even if these aspects were not present in the videos), including any other aspects/ cues they have previously used during familiarization.

Annotators were requested to systematically review the three videos, examining each conversation turn individually. They were prompted to assess various factors, including rating, cues, and aspects, while also considering the detectability of user enjoyment in each turn. Annotators were further instructed to identify contrasting and supporting arguments for their ratings, as the reasons behind the divergence between the annotators can be just as, if not more, valuable than the concordance between the annotators [62]. Upon completion of the turn analysis, annotators were guided to discuss their overall user enjoyment rating, identify which aspects, multimodal cues, and conversation turns contributed most significantly to their conclusions, and consider the relative importance (weights) assigned to these modalities in their assessments. Annotators were also encouraged to evaluate whether specific aspects or cues were observable from both the frontal view (the robot’s camera that directly captures participants) and the side view (the external camera that captures both entities) and reflect on how analyzing these different perspectives may have influenced their assessments. Annotators were advised to keep their ratings, both within ELAN and the accompanying document that justifies their scores, readily accessible on their screens as a reference point during the discussions.

Self-reported user perceptions were given to the annotators at this stage, as reported by questionnaire ratings in terms of the level of user satisfaction, fun, interestingness of the conversation, and strangeness of talking to the robot, as described in Section III-B. These metrics correlate highly with each other (Cronbach’s $\alpha=0.84$), with lower $\alpha$ when any of the metrics are excluded. The annotators were instructed to view the ratings after the overall user enjoyment in the interaction had been discussed for the corresponding participant. Based on the user perceptions, how these results correlate with their findings and the reasons behind discrepancies were discussed.

Finally, the annotators were asked to develop a user enjoyment scale for conversational HRI that serves as both a guideline for their remaining annotations and a reference for future research on user enjoyment (Section IV-D).

The discussions between the annotators (A1, A2, and A3) lasted four hours, during which they carefully went through three example videos (S0-S2) that they had previously annotated and discussed the cases one by one. The annotators viewed every turn in the interaction several times and exchanged the reasons behind their rating in terms of multimodal cues, until they all aligned on the rating for each turn. Simultaneously, they created a list of signs of enjoyment and dis-enjoyment that they had used during their enjoyment evaluation. Towards the end of the session, they settled on a 5-item scale based on the initial provided scale, ranking from very low enjoyment to very high enjoyment. The final user enjoyment scale, namely the Human-Robot Interaction Conversational User Enjoyment Scale (HRI CUES), is:

1. 1

   Very low enjoyment — Discomfort and/or frustration

2. 2

   Low enjoyment — Boredom or interaction failure

3. 3

   Neutral enjoyment — Politely keeping up the interaction

4. 4

   High enjoyment — Smooth and effortless interaction

5. 5

   Very high enjoyment — Immersion in the conversation and/or deeper connection with the robot

Two exchanges per rating of the scale from the alignment interactions are provided as illustrative examples in the video provided in the footnote⁶⁶6HRI CUES exemplary exchanges: https://youtu.be/VmKvGM0pyec.

To rate an exchange higher on the user enjoyment scale (4 or 5), the annotators looked for different signs of enjoyment, which, for example, were smirking, movement, flow of conversation (the topic is moving forward), no strain or discomfort, asking questions [to the robot], smooth turn-taking, dynamic tonality (and dynamic phrasing of sentences), being playful, sharing personal experiences [to the robot], sharing an understanding (common ground) [with the robot], and anthropomorphizing [the robot].

To rate an exchange lower on the scale (1 or 2), the annotators looked for signs of dis-enjoyment, which, for example, were low energy, sighing, tiredness, long breaths, restless movements (i.e., adaptors, such as moving in the chair from side to side or changing arm position), flat tonality, silence, awkward and negative facial expressions, flaring nostrils, disengagement cues (e.g., turning away from the robot, or responding in a way that disrupts the conversation flow, such as “That is true”), and topic closure (e.g., “Let’s talk about something else”). In addition, robot behaviors that disrupted the interaction flow, such as repeated questions, were considered to be strong causes of dis-enjoyment.

Neutral enjoyment (3) refers to a lack of these cues, in which conversation content (and context) becomes more relevant, such as having small talk or continuing the conversation without having much interest in the topic.

In cases where the exchange has cues from multiple scale levels, the annotators determined the dominant level in that interaction. This could be done by observing the intensity of the cues, the significance of the cues, or the interaction trajectory. On the other hand, if an annotator observed strong cues from two moderately or highly distinct levels (as opposed to subsequent levels), they would annotate using a level between those. For instance, as evident in this exchange⁷⁷7Exchange with cues from multiple levels: https://youtu.be/2HA-_5B9JHs, when there is discomfort at the beginning (1), but the user continues to politely keep up the interaction (3), the exchange would be annotated as a 2, the mid-point between the levels.

There were also a few cases that were difficult to categorize as enjoyment or dis-enjoyment, and therefore were interpreted as more context-dependent, which, for example, were gaze aversion, attention on the experimenter or camera, topic duration, and initiation. For instance, gaze aversion could be due to thinking, floor management, intimacy regulation (cf., [63]), or as a reaction to something the robot said or did.

As general guidelines for annotating user enjoyment, it became clear that it was important to get acquainted with the participants, where different participants had different sets of signals. While watching the videos, the annotators learned each person’s rhythm and gestures for what was interpreted as a ‘baseline’ behavior from which the person could deviate during the interaction. This means that the same type of gesture (e.g., keeping one’s arms crossed) could be interpreted differently for different participants. Instead, an emphasis was placed on the change in behavior. It was also important to separate content from context, i.e., it is essential to be mindful of what is being said (conversation content, e.g., topic), but the focus should be more on the whole feeling of the exchange.

The interaction failure does not necessarily refer to a robot failure (e.g., incorrect response, speech recognition failure, turn-taking error, disengagement cue), since robot failures can lead to amusement, anthropomorphism, or empathy in the user, thereby increasing user enjoyment. Interaction failure rather refers to the situation when either the user (e.g., interrupting the robot) or the robot made a failure that resulted in the interruption being disrupted, leading to low enjoyment.

When annotating the videos, the annotators assumed that in the future, robots would be able to judge the level of user enjoyment in real-time while having a conversation. Therefore, the videos were annotated segment by segment (turn by turn), with each segment being watched only once, similar to [40].

After establishing HRI CUES, all annotators independently rated the remaining 25 videos described in Section III-B. The same methodology was employed as in Section IV-B, with the only difference being the enjoyment scale, as HRI CUES was used instead of the initial scale. That is, the annotators rated 590 turns (174 min) using ELAN with both side and frontal view of the interaction (Fig. 2), viewing each turn only once, in addition to providing an overall enjoyment score per interaction. They also provided an explanation for their overall ratings and the challenges they faced during the annotation, as in Section IV-B. The annotation was conducted over 8 days. The results are reported in the next section.

Fig. 3 shows how each annotator rated the interaction exchanges (turns), indicating that the interactions mainly were ($45.9$%) regarded as neutral in enjoyment, with rare occurrences of very low ($9.2$%) and very high ($13.9$%) enjoyment, showing a near Gaussian distribution of user enjoyment for each annotator. Fig. 8 (in the Appendix) shows the rating distributions of annotators per participant, which display a similar Gaussian distribution of perceived enjoyment by the annotators, with some participants (e.g., P25, P27) perceived to have a more enjoyable interaction than others (e.g., P3).

To evaluate the reliability of the annotators’ enjoyment ratings, we employed the Intraclass Correlation Coefficient (ICC), similar to [16], which is a statistical measure used to assess the reliability or consistency of ratings provided by multiple raters (or annotators) [51]. ICC values range from 0 to 1, with higher values indicating greater agreement among raters⁸⁸8ICC value less than 0.5 is poor reliability, between 0.5 and 0.75 is moderate, between 0.75 ad 0.9 is good, and above 0.9 is excellent reliability [51].. This study focuses on two specific forms of ICC:

• •

  ICC(2) - Single Random Raters: Designed for situations where each subject is rated by the same raters, and those raters are considered to be randomly selected from a larger population of possible raters.

• •

  ICC(2,k) - Average Random Raters: An extension of ICC(2), applied when the average ratings of k raters are considered, enhancing the reliability of the measurement.

Similarly to how our data was coded, we present rater reliability for each conversation exchange and the overall enjoyment score provided by each annotator for the interactions.

Users’ subjective ratings of enjoyment during the interactions were obtained from the questionnaire in the participatory design workshops after their interaction with the robot, in terms of user satisfaction, fun, interestingness, and strangeness of the conversation, as described in Section III-B, which are presented in Fig. 2 (in the Appendix) along with the annotator overall enjoyment ratings per interaction. The items had high correlation (Cronbach’s alpha = $0.84$), with the removal of each item reducing the correlation in the construct. These self-reported scores and the average of these scores were compared against annotators’ overall enjoyment scores to evaluate how well the annotators could perceive their enjoyment. Spearman correlation was used across four Likert scale items and the average of these scores, with 95% confidence interval (ranging from $0.71$ to $0.92$). The results were as follows:

• •

  Overall vs. User Average: Not statistically significant ($p=0.08$) moderate positive correlation ($r=0.36$),

• •

  Overall vs. User Satisfaction: Not statistically significant ($p=0.06$) moderate positive correlation ($r=0.39$),

• •

  Overall vs. User Fun Talking: Not statistically significant ($p=0.21$) weak positive correlation ($r=0.26$),

• •

  Overall vs. User Conversation Interesting: Not statistically significant ($p=0.68$) very weak positive correlation ($r=0.09$),

• •

  Overall vs. User Felt Strange (Reversed): Statistically significant ($p=0.04$) moderate positive correlation ($r=0.42$).

While having a similar distribution of ratings by all annotators, the reliability analysis revealed a marked distinction between ICC(2), which assesses the reliability of single random raters, with ICC(2,k), which considers the average ratings of multiple annotators. The latter consistently demonstrated higher reliability across both overall enjoyment scores and conversational turn ratings. This finding highlights the substantial benefit of collective annotator wisdom over individual assessments in assessing user enjoyment.

The superior reliability of ICC(2,k) highlights the inherent variability in subjective experiences and perceptions of enjoyment, suggesting that averaging across multiple annotators can effectively mitigate individual biases and variations in judgment, leading to a more reliable representation of true user enjoyment. This finding is critical, as it emphasizes the importance of incorporating multiple perspectives to achieve a more accurate and consistent evaluation of user enjoyment in conversational interactions with robots. Consequently, the distinction between ICC(2) and ICC(2,k) results not only validates the robustness of our user enjoyment scale but also illustrates the methodological importance of employing multiple annotators for capturing the complex and subjective nature of enjoyment in human-robot conversations.

To further validate our user enjoyment scale, we analyzed correlations between the annotators’ overall enjoyment scores and the participants’ subjective enjoyment ratings. Results revealed a statistically significant, moderate positive correlation between overall enjoyment scores and the (reversed) ‘felt strange’ item, indicating that higher enjoyment scores were associated with decreased feelings of awkwardness during the interaction. This appears to greatly align with our annotator’s discussions and the resulting enjoyment scale that classifies the presence of signs of discomfort as the scale’s lowest level.

However, the correlations between annotator scores and other user-reported measures — such as satisfaction, average enjoyment, fun in talking, and interest in conversation — although trending towards moderate positive correlations, did not reach statistical significance. These findings suggest a nuanced relationship between observed enjoyment and user self-reported experiences. While annotators can detect general levels of comfort and ease within interactions, capturing comprehensive internal subjective enjoyment may require additional data, such as initial expectations towards the robot, personality traits, as well as physiological responses, not accessible through direct observation. This discrepancy underscores the complexity of correlating observed behavior with subjective internal states and highlights the challenge of fully capturing user enjoyment in HRI. Despite these considerations, having an additional tool to evaluate user enjoyment in conversations is highly valuable for HRI research to assess and develop robots that are enjoyable to interact with, within all parts of the conversation, especially in daily life. This approach aligns with how humans naturally adapt our conversations in real-time based on the external cues we observe in others. The measure presented in this paper offers a valuable tool for collecting observational data to train autonomous enjoyment detection systems that can be used on robots or other agents.

After analyzing the results, the annotators were asked to have another set of discussions by presenting them with Fig. 9 and 10 in the Appendix (without reliability or correlation scores), which lasted four hours. They were asked to pinpoint instances of rating divergence and concordance within their assessments. Particular emphasis was placed on identifying turns where annotators disagreed or agreed most fervently. Subsequently, the annotators were prompted to watch the corresponding video segments to explore the reasons behind their ratings and the rationale for their agreement or disagreement. The primary objective was to gain insight into potential major concordance (Section VI-C1) and divergence (Section VI-C2) in the way dialogue exchanges were annotated and elucidate the underlying reasons for these variations.

Following the turn-by-turn analysis, annotators were asked to identify the two most significant discrepancies (highest and lowest) between their ratings and self-reported participant perceptions (Section VI-C3). They were asked to engage in discussions exploring potential reasons underlying the disparities and similarities between their perceptions and those of the users from multiple aspects based on their expertise.

A final round of annotator discussions (lasting 3 hours) was conducted to adapt the developed scale to other domains in HRI, extending its application beyond the context of companion robots for older adults to encompass other conversational contexts where analyzing enjoyment is crucial.

The process of using the Human-Robot Interaction Conversational User Enjoyment Scale (HRI CUES) is twofold. First, the scale is the primary supportive instrument for engaging with the dataset and finding an agreement among annotators regarding which level of enjoyment an exchange represents. Secondly, it is important to highlight the cultural and context-dependent changes when assessing enjoyment with the scale, therefore, we recommend that the annotators reach an agreement on which multimodal cues are important in their study. In the previous section, we presented the relevant cues in our study located in a Nordic Western setting, but these could differ in another setting. For example, gestures, such as thumbs up, mean different things in different parts of the world. Our proposed scale can already be directly applied in many settings where users engage with social robots (or potentially other agents) in conversations. However, in many other cases, we encourage further adapting the scale to the particular domain.

To replicate our methodology and adapt HRI CUES to another domain, the following framework should be applied:

1. 1.

   Recruit three annotators with relevant and complementary backgrounds who are familiar with the specific culture and context of the study.

2. 2.

   Establish the intended usage of the annotations such that the annotators can tailor their annotations to fit that use case and align their views on the practical meaning of enjoyment in that context.

3. 3.

   Ask annotators to systematically annotate three example videos (from the dataset) using the HRI CUES. Example videos should exemplify different interaction outcomes from the dataset and be represented from various angles (front, side, both). Encourage the annotators to look for their own cues of enjoyment or dis-enjoyment, and describe the reasons behind their overall enjoyment scores based on those after viewing each interaction, and the corresponding challenges of detecting enjoyment.

4. 4.

   Arrange a discussion between the annotators to identify contrasting and supportive arguments for multimodal cues associated with the scale, aiming to precisely determine the cues for each segment and strive for consensus for the corresponding rating. Give each segment sufficient time for discussions while avoiding getting stuck on small details. When faced with a difficult case, note what is not agreed on and move to the next segment.

5. 5.

   Based on the discussion, construct an annotation schema, which should contain the cues that were agreed on for assessing the enjoyment in relation to the scale, especially emphasizing the cues that were discussed and not immediately agreed on.

6. 6.

   Annotate the remaining dataset turn-by-turn using the HRI CUES and the multimodal cues. This is done by looking at each exchange once and annotating in real-time without going back in the data to not influence the evaluation of the beginning segments of the interaction by already knowing the end segments.

In this paper, we evaluated HRI CUES through the interactions of older adults with a conversational companion robot using Furhat at the university premises. However, this was only one example of how the scale can be used; HRI CUES is generalizable to other contexts in which a user interacts with a social robot. Therefore, the second to fifth steps of the framework above are crucial for the annotators to adapt the communication cues to their context and setting of the study. HRI CUES does not require any adjustment as such, but it requires interpretation with respect to the context of the use case. The interpretation is facilitated by the six-step framework, hence, it is important to find appropriate annotators who are familiar with the particular application area.

In this work, we introduced a novel scale for annotators to evaluate user enjoyment in conversational HRI. However, enjoyment is a subjective measure, and thus, is challenging to evaluate and agree on between annotators and correlate with users perceptions, given the multitude of aspects connected to it. For instance, the context and length of the exchange affected how the enjoyment was perceived. When segments were overly brief, the exchange did not always contain sufficient information for a fair assessment. More frequently, however, excessively long segments were complex to analyze. For example, as described in Section VI-C2, the annotators often interpreted the exchange differently due to long segments that contained several cues belonging to separate levels of the scale. In these cases, a different assessment approach may be necessary as the longer segments introduced an additional factor for the annotators to consider: which part or aspect of the exchange to emphasize in the assessment. Due to the complex nature of enjoyment as a concept, which highlights the necessity of pre-coding discussions among the annotators, aligning on the definition of enjoyment of relevance to the particular study is challenging. One solution is to focus on the change of behavior within the exchange. For instance, if the robot’s response improved the user’s demeanor towards the robot, the exchange should be rated towards the level that contains higher enjoyment cues in the scale, and conversely if it had a negative impact. Other alternatives could be to use an aggregated score, the rating that corresponds to the majority of the segment, the most/least extreme rating, or the rating that corresponds to the first or last part of the segment. The choice of method for these situations should be determined during the annotation alignment process, taking into account the specific use case and domain.

The differences between the annotators’ overall rating on user enjoyment and user perceptions (as discussed in Section VI-C3) might be due to the participant’s expectation of the robot’s social and technical level, while the annotators only look at the interaction itself. In addition, since this is their first interaction with a robot for most of the participants (20 out of 25), the ‘novelty effect’ might have changed their perceptions more positively or negatively, given that the duration (7 minutes) is not long enough to overcome it [65]. However, the variability observed by the annotators in user enjoyment states throughout the interaction (e.g., Fig. 5) shows that conversation context may alter the novelty effect, providing a more complete picture of the enjoyment throughout the interaction than a self-reported score at the end of the interaction. In addition, the users’ responses to the questionnaire may differ from their actual attitudes towards the robot [66]. These support the importance of HRI CUES as an additional tool to evaluate user enjoyment, providing means for real-time estimation of enjoyment in conversational agents.

Our scale is designed to serve as a tool for assessing perceived user enjoyment during interactions with robots, intended primarily for researchers in the field of HRI. However, given that open-domain dialogue may involve sensitive information that individuals may be hesitant to share with unfamiliar parties, it is imperative to obtain explicit consent from participants before they talk with the robot. In our studies, we ensured this consent prior to participants’ interactions with the robot, employing the use of the term ‘sentiment analysis’ in the consent form, and explained to all participants that their interactions would be analyzed in terms of their affective states (‘identification of feelings’) by both researchers and automated systems. While obtaining consent is essential for researchers utilizing our scale in future studies, it is crucial to recognize that this process may influence participants’ behavior and conversation topics, as they may be reluctant to share sensitive memories or be concerned about being judged by others. Consequently, this can result in a disconnection with the robot, posing challenges in achieving high levels of enjoyment during interactions. Nonetheless, this impact is likely to diminish over the course of the interaction or across multiple interactions, particularly in long-term settings. Additionally, researchers must exercise caution to uphold the privacy and confidentiality of participants when sharing data, ensuring that they remain unidentifiable in images and videos (as demonstrated in the examples provided for the scale), and removing any sensitive information. Moreover, researchers should remain mindful of their own biases and subjectivity, which may lead to variations in the interpretation of enjoyment compared to the participants’ experiences. User enjoyment is often context-specific, indicating that users’ behavioral and affective expressions are connected to specific socio-cultural contexts, including values, norms, and expectations of what is considered appropriate in certain situations [67]. These underscore the significance of employing multiple annotators for the scale that are familiar with the socio-cultural background of the target population, and using the scale to complement self-reported user perceptions to have a deeper understanding of the interaction quality. As human-robot interactions continue to evolve, ensuring a deep understanding of user enjoyment not only elevates the quality of these interactions but also paves the way for more empathetic and meaningful connections.