ASDs: Autism Spectrum Disorders; TD: Typically Developing; HF: High-functioning; LF: Low-functioning

In recent years, robots have been used in a variety of rehabilitation contexts. The field of socially assistive robotics uses various robots to assist special populations through social interactions that do not involve physical contact [1]. In the field of pediatric rehabilitation, robots have been used as therapeutic tools in children with Autism Spectrum Disorders [ASDs] to develop socially directed behaviors [2,3]. Robots could be a promising tool to facilitate social and communication skills in children with ASDs because they are simpler and more predictable social entities compared to humans, they can be programmed to provide structured and individualized interventions, and they are a highly motivating context to promote social communication skills in children with ASDs [2-7]. There is some anecdotal evidence for the use of robots to facilitate joint attention, turn taking, vocalization and imitation skills in children with ASDs [5,7-10]. Within triadic contexts involving a robot, an adult and a child, robots act as the focus of shared attention and elicit prosocial behaviors such as joint attention and verbalization between the child and the other individual [10-13]. Nevertheless, a recent systematic review suggested that the current literature on robots as facilitators of social and communication skills in ASDs is limited due to small sample size studies, a need for better experimental designs, and a lack of systematic measurement of treatment effects [5]. Moreover, there is a clear lack of normative data on the use of robots in TD children [5]. Therefore, the overall goal of this study was to address these limitations in the literature by examining the effects of a structured, 8-session protocol of robot-adult-child interactions on the social attention and verbalization patterns of 15 typically developing [TD] children. We further extended this protocol to two children with ASDs. These data have served as a foundation for an ongoing rigorous randomized controlled trial examining the effects of robot-child interactions in children with ASDs.

Social attention emerges during early infancy and allows infants to share their interests with their caregivers [14]. Young infants engage in dyadic interactions with their caregivers for the first six months. Towards the end of the first year, infants transition to triadic interactions between caregivers and objects/events within their environment, also known as joint attention [15-17]. Early on infants may follow the looks and gestures of their caregivers [18,19]; however, later they direct their caregiver’s attention through looking and pointing to objects or events in the environment.In contrast, children with ASDs show poor joint attention skills from early on in life. Moreover, they showed persistent deficits in sharing attention with their caregivers throughout childhood [20-22]. Even high risk infants who did not go onto develop ASDs showed reduced spontaneous social orienting suggesting that social attention may be affected in the broader autism phenotype [23]. Taken together, social attention skills are clearly impaired in children with ASDs and could be facilitated using social-object engagement in triadic contexts. The robotic intervention in our study involved a triadic context inclusive of the child, an adult trainer and the robot. Hence, the first aim of this study was to examine changes in the attention patterns of TD children and children with ASDs towards the robot and the trainer/tester across testing and training sessions. We hypothesized that both TD children and the children with ASDs will demonstrate greater attention to the robot than the trainer across all weeks. However, over the weeks of training, we expected that both groups of children would increase their attention to the trainer and decrease their attention to the robot.

Social interactions between the infant and his/her caregiver provide a scaffold for language development [24]. Shared attention skills such as gaze alternation, non-verbal requests and pointing are related to the language skills of TD children [15]. Toddlers’ joint attention episodes with mothers positively correlated with vocabulary size at 21 months [24]. During joint attention episodes, caregivers point to and label objects providing opportunities for word learning [18,25-28]. These associations between social attention skills and language abilities continue into early childhood [29]. On the contrary, children with autism often demonstrate impairments in verbal communication [30], that clearly affect their long-term outcomes [31,32]. Similar to the findings in TD children, shared attention skills in children with autism during the preschool years are strongly predictive of verbal outcomes in late childhood and adolescence [33]. Moreover, joint attention interventions are known to improve verbal outcomes of preschool children with autism [34]. Given the correlations between social attention and language development, the second aim of our study was to examine changes in verbalization patterns of TD children and children with ASDs across weeks of training. We hypothesized that over weeks of training within the triadic robot-adult-child context, spontaneous verbalization would increase in both, TD children and children with ASDs.

Imitation-based skill learning is critical for social and cognitive development in TD children [35]. Longitudinal associations have been demonstrated between imitation skills and social attention [36,37], language skills [38] and gestural development [22,39]. TD infants learn to imitate their caregiver’s actions on objects during shared attention episodes as early as nine months [40]. Infants typically learn their first words through imitation of gestures and words [41]. Children with ASDs demonstrate impairments in the imitation of orofacial, manual and gross motor actions [42]. Deficits in imitation abilities of children with ASDs are strongly correlated with their deficits in joint attention, play and language abilities [43]. Specifically, imitation of body movements at two years correlated with expressive language skills at three years [44], and object imitation skills correlated with joint attention skills of children with ASDs [40]. Moreover, imitation-based interventions have been used to facilitate language skills in children with autism [41]. Hence, the present study used an imitation game to facilitate interactions between a child, robot and the adult trainer. We hypothesized that an imitation game between the robot and the child would provide opportunities for TD children and children with ASDs to engage in social attention episodes with the adult trainer, as well as spontaneously verbalize to the robot or use the robot as a topic of conversation with the adult trainer. Our data on changes in imitation performance are accepted for publication and indicate improvements in imitation accuracy following training in TD children and children with ASDs.

Participants

Fifteen typically developing children (9 males and 6 females) between 4 and 8 years of age (mean age ± SE: 5.79 ± 0.35 years) participated in this study. In addition, we also extended this protocol to one high-functioning [HF], 8-years-old male child with ASD and one low-functioning [LF], 7-years-old male child with ASD. The diagnosis of ASD was confirmed using clinical evaluations involving the Autism Diagnostic Observation Schedule [ADOS] [45], detailed clinical history, and overall clinical judgment. The HF child with ASD was highly verbal and well-integrated into a regular public school, whereas the LF child with ASD was low-verbal, provided 1 to 2-word responses and attended a special needs classroom in a public school setting. All families were Caucasian American in their origin, except one family was Asian and another was Hispanic in origin. The average socioeconomic status score for all families was 58 ± 1.90, indicating upper-middle to upper class families [46]. Children were recruited through the university listservs and from local daycare centers. Children were admitted in the study following written parental consent. This study was approved by the ethics committee of the Institutional Review Board at the University of Connecticut.

Procedures

The study was conducted over six weeks. The pretest and posttest sessions were conducted during the first and last weeks of the study. The training was provided in the intermediate four weeks. Each child interacted with the robot twice each week for a total of eight sessions. Only three TD children missed one training session due to scheduling issues.

Training protocol: Eight training sessions were offered over four weeks, with two sessions provided each week. Each session lasted for approximately 30 minutes. The training was provided using a commercially available, 7-inch humanoid robot called Isobot [Tomy, Inc] (Figure 1A), controlled by an adult trainer via a laptop system. The robot, the adult trainer and the child were arranged in a triadic spatial arrangement to facilitate engagement of the child with the trainer and the robot (Figure 1B). Training sessions alternated between karate and dance themes. Across training sessions, we progressed from simple, two-step actions involving dual-limb motions to complex fourstep patterns involving multilimb motions for the TD children and the HF child with ASD. However, the LF child had a relatively simple progression from one-step, dual-limb actions to two-step, multilimb actions. Each training session involved three imitation phases. In the baseline condition, the robot would greet the child and perform an introductory karate bow or a hula dance depending on the theme of the session. Following this, the trainer explained the theme for the day (karate or dance), and the actions that the child should copy during the next two phases. In the robot-led condition, each child was asked to copy the actions performed by the robot. During each session, the child copied four actions performed by the robot and each action was copied four times. Each action was associated with a verbal label such as the “backhand” or the “sidechop”. If the child missed out certain steps or parts of actions, the trainer would ask them to carefully attend to specific components of the action by providing verbal cues, for example, “Make sure to look at the robot’s leg”. In the child-led condition, children were asked to recall the actions they had practiced in the robot-led condition and demonstrate these actions to the robot, who would then copy them. Each action was recalled twice. If the child failed to recall the action, the trainer would prompt using the verbal label of the action. If the child still did not recall the action, the trainer would demonstrate a part of the action to the child. Once the child performed the action for the robot, the trainer would trigger the robot’s actions using the control software. Small toys were provided at the end of each session and each child received $50 as participation reimbursement at the end of the study. All the training sessions were videotaped using an oblique view; which included the robot, the trainer, and the child.

Figure 1: (A) The 7-inch humanoid robot Isobot. (B) The experimental setup including the child, the robot, and an adult controlling the robot via a laptop control system. The child is imitating the robot’s action.

Testing protocol: A novel tester administered the pretest and the posttest sessions. The design of the robot imitation test was similar to that of the training sessions and included the baseline, the robot-led and the child-led conditions. The session involved five novel actions that were not practiced during the training sessions. We used the video data from the robot-led and the child-led conditions of the pretest, session 1, session 4, session 8, and the posttest to code for social attention and verbalization patterns. Custom coding software was used to code for social attention and verbalization measures.

Dependent variables for attention and verbalization

Percent duration of attention: Percent duration of attention is the percent of time that the child looked at “the tester/trainer”, “the robot”, and “elsewhere” within a session. In the pretest and the posttest, attention directed to the “novel tester” was coded, whereas during the training sessions, attention directed towards the “familiar trainer” was coded.

Percent duration of verbalization: Percent duration of verbalization to the tester/trainer was coded as spontaneous or responsive within a session. Verbalization initiated by the child with no prompts from the tester/trainer was termed “spontaneous” and verbalization, in response to comments or questions by the tester/ trainer was termed “responsive”. A single coder established intra-rater reliability using intra-class correlation coefficients before coding each dataset. Intra-rater reliability of over 89% was established using 20% of the entire dataset for both coding schemes.

Statistical analyses: The Kolmogorov-Smirnov test for normality was conducted to confirm the normal distribution of data. For the TD children, the training-related changes in “percent duration of attention” were analyzed using a repeated measures ANOVA with test times (pretest, session 1, session 4, session 8, and posttest) and attention type (to robot, to tester, and to elsewhere), as the within-subjects factors. The training-related changes in “percent duration of verbalization” were analyzed using repeated measures ANOVA with test times (pretest, session 1, session 4, session 8, and posttest) and verbalization type (spontaneous and responsive) as the within-subjects factors. If the Mauchly’s test of sphericity was significant then Greenhouse-Geisser corrections were applied [47]. If there was a significant main effect and an interaction effect involving a particular factor, we conducted further post-hoc t-tests to assess the significant interaction effect only. Effect sizes are reported for all significant findings using partial-eta squared (η_p²) values [47]. Statistical significance was set at p<0.05. For the two children with ASDs, we report individual data on attention and verbalization patterns.

Training related changes in percent duration attention to robot, to tester/trainer, and to elsewhere in TD children and children with ASDs

In TD children, the ANOVA indicated a significant main effect of attention type (F (1.46,42.35)=80.24, p<0.001, η_p²=0.735) and a test time×attention type interaction (F (4.27,123.67)=10.75, p<0.001, η_p²=0.27). To further investigate our significant interaction, based on the trends seen in Figure 2, we conducted two post-hoc comparisons. First, we compared social attention between the pretest and posttest. Second, we compared social attention between training sessions 1 and 8. Post-hoc tests comparing the pretest and posttest indicated that in both tests, children spent maximum time attending to the robot followed by the trainer followed by attention to elsewhere (ps<0.001) (Figure 2). In terms of changes in attention patterns following training, children looked less at the robot during the posttest compared to the pretest (p<0.001) (Figure 2). Concurrently, there was an increase in the time spent attending to the tester and to elsewhere in the posttest compared to the pretest (ps<0.001) (Figure 2). Post hoc tests comparing training sessions 1and 8 reflected similar trends as the testing sessions in that children spent maximum time engaged with the robot, followed by the trainer followed by attention to elsewhere (ps<0.001) (Figure 2). However, in terms of training-related changes, children showed some reduction in their attention to the trainer by session 8 compared to session 1 (p<0.001) (Figure 2). In contrast, attention to elsewhere increased by session 8 compared to session 1, possibly indicating disinterest (p<0.001) (Figure 2).

The trends for social attention in the two children with ASDs suggest that both children directed maximum attention towards the robot (Figures 3A and 3B). However, both the HF and the LF child with autism showed a progressive decline or plateauing of attention towards the robot with a simultaneous increase in attention towards elsewhere over the weeks of training. There was no significant training-related increase in the attention towards the trainer across training for both children (Figures 3A and 3B).

Figure 2: Percent duration of attention to the robot, the tester/trainer, and to elsewhere across the testing and training sessions in TD children.

Training related changes in percent duration of verbalization in TD children and children with ASDs

In TD children, the ANOVA indicated a significant main effect of test time (F (3.02, 87.47)=12.76, p<0.001, η_p²=0.31), a main effect of verbalization type (F (1,29)=64.70, p<0.001, η_p²=0.69), and a test time×verbalization type interaction effect (F (2.37,68.79)=7.91, p<0.001, η_p²=0.21). To evaluate the test time×verbalization type interaction, we conducted 2 types of post-hoc tests based on Figure 4. First, we compared verbalization between the pretest and posttest. Second, we compared verbalization between training sessions 1 and 8. Post-hoc tests comparing the pretest and the posttest indicated that in both tests, children showed greater spontaneous verbalization compared to responsive verbalization (ps<0.001) (Figure 4). In addition, the TD children demonstrated greater spontaneous verbalization in the posttest compared to the pretest (p<0.001) (Figure 4). However, no significant pretest-posttest differences were found for responsive verbalizations (Figure 4). Post-hoc t-tests comparing changes across training sessions 1 and 8 showed that children demonstrated significantly greater spontaneous than responsive verbalizations across sessions 1 and 8 (ps<0.05) (Figure 4). Furthermore, children demonstrated greater spontaneous verbalization in session 8 compared to session 1 (p<0.05) (Figure 4). No training-related changes were observed for responsive verbalizations.

Figure 3: (A) Percent duration of attention to the robot, trainer, and to elsewhere across sessions in a high-functioning child with ASD. (B) Percent duration of attention to the robot, trainer and to elsewhere across sessions in a low-functioning child with ASD.

Figure 4: Percent duration of spontaneous and responsive verbalization across the testing and training conditions in TD children.

In general, both children with ASDs demonstrated greater spontaneous compared to responsive verbalization across training weeks (Figures 5A and 5B). The HF child with ASD had reduced spontaneous verbalizations in session 8, with a subsequent increase in spontaneous verbalizations during the post test. The LF child with ASD had low levels of verbal communication and did not show trainingrelated changes in the duration of spontaneous verbalizations. Both children showed a plateau of responsive verbalizations across training sessions (Figures 5A and 5B).

While robots have been used “anecdotally” to facilitate social interactions in children with ASDs, their influence on promoting social communication skills in TD children is unclear. The purpose of this study was to evaluate the effects of a systematic, 8-session robot-adultchild interaction protocol on the social attention and verbalization patterns of TD children between 4 and 8 years of age. In addition, we examined the feasibility and value of using robot-child interactions in 7 to 8-years-old children with ASDs using a case-study approach. This study has served as a foundation for an ongoing randomized controlled trial assessing the efficacy of robot-child interactions in children with ASDs. We found several interesting changes in the attention and verbalization patterns following training. First, both TD children and children with ASDs directed most of their attention to the robot during the testing and the training sessions (Figures 2, 3A, and 3B). Within the training sessions, TD children demonstrated an initial increase in their attention towards the trainer, followed by a subsequent decrease by session 8. In contrast, children with ASDs showed low levels of attention towards the trainer and did not demonstrate any significant training-related improvements in attention patterns towards the trainer; which is consistent with the social impairments of autism. Both TD children and the HF child with ASD showed a steady increase in the attention directed to elsewhere across training sessions (Figures 2 and 3A). Second, in terms of verbalization patterns, TD children showed greater spontaneous verbalization in the posttest and across training compared to the baseline pretest performance (Figure 4). In general, there were less responsive verbalizations and they plateaued across the testing and training sessions. The two children with ASDs demonstrated greater spontaneous compared to responsive verbalizations across all training sessions; however, they did not show consistent increases in spontaneous verbalizations following training (Figures 5A and 5B). We further discuss these findings and their implications in the following paragraphs.

Figure 5: (A) Percent duration of spontaneous and responsive verbalization across sessions in a high-functioning child with ASD. (B) Percent duration of spontaneous and responsive verbalization across sessions in a low-functioning child with ASD.

Changes in attention patterns across training in TD children and children with ASDs

Typically developing children and children with ASDs spent most of their time looking at the robot throughout the training compared to the tester/trainer, and to elsewhere. However, attention towards the robot reduced by the mid-training session and plateaued thereafter. Increased attention to the robot has been reported in other studies where robots were used as therapeutic tools for children with autism [2,7,8,12,48,49]. Children with ASDs demonstrated greater shared attention towards a robot mediator, Tito compared to an adult-human mediator [7]. This was attributed to the appealing characteristics, simplicity and predictability of the robot [7]. Children with ASDs showed an increase in the duration of attention directed towards a humanoid robot following repeated interactions with the robot across several months during simple imitation and turn-taking games [8]. Our results are similar to the aforementioned studies and further extend these findings to TD children. We found that both TD children and children with ASDs find robots highly engaging. Our training consisted of an imitation game that involved children copying progressively complex movement patterns performed by the robot. This required careful attention towards the robot’s limbs and body, and may have contributed to the greater attention to the robot in both TD children and children with ASDs. However, our study results suggest that across training weeks, both TD children and children with ASDs showed some decline/plateauing of attention to the robot, with a concurrent steady increase in attention towards elsewhere in the room. This is the first study to report reduced interest following an extended protocol of robotic interactions in TD children, and interestingly even in children with ASDs. The 7-inch tall, humanoid robot used in the current study had a limited motor and verbal repertoire. The use of more sophisticated robots with greater capabilities and more engaging contexts may be one way to address the boredom observed during the robot-child interactions .

Typically developing children in our study spent considerable amount of time engaged in social attention episodes during the early and mid-training sessions. By the last session, there was some decrease in attention to the trainer. The presence of shared social attention with humans within the context of robot-child interactions has also been reported in several preliminary studies [2,4,8,12,13,49]. Children frequently shared their pleasure and interest about Keepon, a simple creature-like robot, with their caregivers and teachers [49]. Similarly, children with autism spontaneously initiated and responded to the bids of the experimenter during repeated exposure to the humanoid robot, Robota across twelve weeks. Thus, the robot served as a salient object that mediated joint attention between the child and the adult experimenter [8]. In our study, TD children may have shared attention with the trainer for several reasons. First, the robot could be the common focus of attention between the child and the trainer, and children might have spontaneously initiated attention bids to share information about the robot’s appearance/movements/verbalizations with the trainer. Second, the children may have developed a rapport with the trainer over time. Finally, they may have checked back with the trainer to gain reinforcement and feedback about their imitation performance. Children with ASDs in this study also showed shared social attention with the trainer but there were clearly fewer instances compared to the TD children. Our context primarily promoted engagement between the robot and the child, with no additional social bids provided by the trainer. Hence, it is not surprising that there was no significant training-related increase in the attention directed towards the trainer in children with ASDs. Moreover, children with ASDs may have used peripheral vision to monitor the trainer, which is difficult to discern using video coding. Overall, our results suggest that to facilitate social communication skills in children with ASDs within the context of an imitation game, there is a need to develop more focused and intense protocols that are targeted towards improving verbal and non-verbal communication skills of children. Specifically, within a context of robotadult- child interactions, the adult trainer needs to play a more active role in initiating interactions with the child with autism, using a variety of social bids to the child to promote joint attention and conversations.

Changes in verbalization patterns across training in TD children and children with ASDs

On comparing verbalization types, both TD children and children with ASDs produced greater spontaneous than responsive verbalizations across all sessions. In TD children, there were training-related increases in spontaneous verbalizations until the mid-training session, with no comparable increases in responsive verbalizations with training. This suggests that the context of robot-adult-child interactions promoted spontaneous engagement and communication between the children and the trainer. Imitation-based, robot-child interactions appear to promote more spontaneous than responsive forms of communication, such as social attention, turn taking and verbalization, and would be important for several reasons. Developmentally, spontaneous social engagement is more complex and emerges later in development than responsive social engagement [50]. For example, infants begin to respond to caregiver bids several months before they spontaneously engage with their caregivers [15,51]. Moreover, children with autism have deficits in initiating spontaneous communication [30,52], and hence robot-child interactions can be used as a potential tool to encourage spontaneous verbalization in this population [2,3,7,8,11-13,53]. Spontaneous sharing of verbal information was observed in children during triadic interactions between the robotic pet, Kasha, the child, and the experimenter [53]. Similarly, low functioning children with autism began to use the robot as a mediator for their verbal interactions with adults following repeated interactions with a humanoid robot, Robota [2,8]. When children with autism realized that the mobile robot, Tito, was in fact teleoperated, this became a topic of further conversations between the child and the experimenter [7]. Thus, robotic interactions could be structured to encourage verbal and non-verbal communication skills in children with ASDs.

Similar to the findings in the above mentioned studies, we found that TD children and the HF child with ASD engaged in spontaneous conversations with the trainer about topics pertaining to the robot, such as the robot’s appearance, motoric capabilities and its vocalizations. Over time, children developed a rapport with the trainer and began to engage in general conversations. Across the last few training sessions, the complexity of movement sequences progressively increased, and this may have contributed to the decrease in the spontaneous verbalization with the trainer in session 8 compared to session 4 in TD children and children with ASDs. In TD children, the amounts of spontaneous verbalization for all training sessions were always greater than the baseline pretest level. In children with ASDs, though there was no major training-related increase in the duration of spontaneous verbalizations, it was encouraging to see that they also engaged in greater spontaneous than responsive verbalization with the trainer within the context of robot-child interactions. As stated earlier, our future studies in children with ASDs will be structured to ensure greater opportunities for promoting non-verbal communication and conversations between children with ASDs and their social partners. For example, the adult could ask the child to show him/her the actions the robot just did, or may ask the child specific questions about the robot and the activities performed.

Our preliminary study had several limitations such as the lack of a control group and limited number of training sessions. In addition, the robot used in the current study, Isobot, had limited motor and verbal capabilities. This study served as the foundation for our currently ongoing randomized controlled trial to assess the efficacy of robot-child interactions on the social and communication skills of children with ASDs. Note that we are not reporting imitation performance of the TD children and the children with ASDs in this paper because those data are part of another publication [54]. Lastly, we recognize the technical challenges to clinical implementation of robotic interventions such as operational inconsistencies, limited autonomy and capacities, software and hardware problems, as well as the high costs.

Our study is the first to systematically examine the effects of an 8-session, training protocol using robot-adult-child interactions on the social attention and verbalization patterns of TD children. We also extended this protocol to two children with ASDs as a pilot for future intensive studies in children with ASDs. We found that motor imitation games with robots help facilitate social communication skills in TD children, and to some extent, in children with ASDs. Typically developing children and children with ASDs directed greatest attention towards the robot; however, interest in the robot wore off over time. In addition, TD children and children with ASDs engaged in spontaneous conversations with the tester/trainer during robotchild interactions. These findings have implications for children with ASDs who demonstrate motor difficulties in imitation and praxis, as well as social communication impairments such as reduced social attention and spontaneous verbal communication. We propose that robot-child interactions could be a feasible tool for advancing social and communication development. However, more active participation from the trainer is needed to promote social communication skills in children with ASDs. Future studies should extend this work through systematic studies within a larger sample of children with ASDs using robots with a broader repertoire of enjoyable, educational, and functionally relevant activities.

We would like to thank the children and families who participated in this study. We thank Tim Gifford and Christian Wanamaker for their help with programming of Isobot. We thank graduate students-Kinga Palatinus and Maninderjit Kaur for their help with data collection or data analysis. We thank the undergraduate students who contributed to this project through data collection, data processing, and behavioral coding-Kathleen Lynch, Megan Lally, and Carolyn Susca. AB thanks the National Institutes of Mental Health for support of this research through grants- R21MH089441 and R33MH089441.