Children and adults with autism spectrum disorder (ASD) often suffer from difficulties regarding the ability to perceive others’ emotions and intentions [1] suggested that difficulties with understanding emotions are related to difficulties with expressing emotions and vice versa in children with autism. These difficulties have been found to be related to the development of “joint attention” [2], “theory of mind” [3], and “facial cognition” [4,5]. Thus, several aspects of the social, communicative, and behavior difficulties children with ASD experience may be associated with emotion perception differences/difficulties. One of these is non-verbal and pragmatic communication difficulties [6,7]. As specified in DSM-V (Diagnostic and Statistical Manual of Mental Disorders, fifth edition), one aspect of non-verbal and pragmatic communication difficulties among children with ASD is facial expression and perception of emotions. Difficulties with understanding emotion by seeing others’ facial expression has been investigated. Hobson studied, for example, how children with autism disorder classified human facial expressions. He found that individuals with autism disorder showed difficulties in understanding others’ emotions and they did not spontaneously classify other’s emotions by reading facial information. Other studies have also found that children with autism disorder showed error patterns in tests of identifying facial expressions [8-10].

Emotions are not only understood and expressed in facial expressions, words, and body language but also in prosody that is one aspect of non-verbal and pragmatic communication. Prosody is characterized by rhythm, intonation, tone, pitch, and stress of speech which holds extra lingual information such as the emotional state of the speaker, presence of irony or sarcasm, or if the utterance is a question or command. Emotional prosody is the emotional tone of voice, and it is significant for the understanding of others’ emotional state and intentions [11,12]. Perception of emotional prosody contributes to our understanding of what is known as the pragmatic meaning of speech, the intended ‘between-the lines’ meaning [13,14]. For example, when we say “what a fool” it can have different meanings such as: affinity to others, sympathy for someone who has made mistakes, or an insult. According to Hargrove, Banse and Scherer, affective prosody is linked to the speaker’s state of feeling which then also becomes an important component of verbal communication [15,16]. At toddler age, understanding and use of prosodic features starts developing [17,18] and at school age the development of prosody comprehension and production continues [19-21]. However, the development of prosody skills continues into adolescence. With regard to emotional prosody, Oshima et al. pointed out that there is a significant increase in skills of understanding others’ emotion from sound or writing in elementary (7-12 years old) and junior high (13-15 years old) school children [22]. Noguchi et al. further report that skills of understanding irony and joking are fully established as late in high school aged children [23].

Children with ASD more often than children without ASD show difficulties with understanding people’s intentions since they often interpret other’s words literally [24]. They may more often feel insulted by others when called “a fool” and they may feel like they are praised when they hear “big deal” even if communicated sarcastically. If a teacher in a classroom asks a student with ASD who has fallen asleep, “What are you doing?”, he or she may answer “I’m sleeping”, not understanding the teacher’s intention “Don’t sleep!”. As a consequence, the teacher may easily feel that the student is making a fool of him/her.

Previous studies of understanding and expression of emotional prosody among individuals with ASD

Difficulties with understanding prosody have been found among individuals with ASD [25,26]. Recently, Globerson et al. reported poorer performance on an emotional prosody recognition task among adults with ASD compared to a matched group without ASD. Among children Mitsuhashi and Nakamura compared children with learning disabilities and high-functioning pervasive developmental disorders (HFPDD) and found that children with HFPDD experienced more difficulties in understanding others’ emotions and intentions from speech intonation and often took words literally [27]. Other studies have found that children and adolescents with ASD show difficulties or differences with perceiving and imitating prosodic patterns (exaggerated or monotone, fast or slow) compared to typically-developing controls [28-30].

The study by Boucher et al. showed that children with autism disorder had perceptual processing difficulties for matching familiar voice and faces compared with a mixed group of children with language-delay. The same study also found that the children with autism disorder were impaired relative to typically developing children on an affect matching task of emotional sounds and emotional facial expressions presented on photographs. Oshima et al. investigated prosody characteristics of Japanese children with high functioning autism (HFA) and controls using a test which included facial expressions and context information conveyed with positive and negative words [22]. The number of correct answers given by children with HFA was significantly lower with regard to understanding of irony and jokes (incompatibility between literal meaning and emotion words) compared to the controls that were typically developing children. The study by Oshima et al. only used positive and negative words of linguistic meaning and not neutral words [22]. Takezawa et al. used neutral-word stimuli (along with positive and negative words) in a study comparing children with HFPDD and a typically developed control group to determine how they understood emotions from facial expressions and prosody [31-33]. The study reported no significant differences between the HFPDD and the control group, neither on positive words, negative words, or neutral words. The researchers underlined that the study sample was small and that they did not systematically select the words with regard to emotional valance and strength.

Previous studies of perception and expression of emotions have used pictures or video of faces [34], graphic illustrations, participant’s facial expressions or reading stories as stimuli [35]. For example, Peppé et al. reported that the HFA group performed significantly less well on several aspects of both receptive and expressive prosody in the PEPS-C (Profiling Elements of Prosody in Speech-Communication) test [36]. Another study by Loveland, et al found no significant differences between individuals (children, adolescents, and young adults) with HFPDD and individuals without HFPDD with regard to emotional understanding from prosody using facial expression and neutral words presented on video [37]. The study did not classify the emotions presented (positive, negative, or neutral) and further the number of participants was small. It was also not clear from the procedure which of the information was recognized and if the information was intricately interrelated.

First, it is necessary to carry out research focusing on the understanding and expression of prosody by use of sound stimuli only as opposed to extra stimuli such as pictures, stories, and facial expressions to evaluate children’s understanding and expression of genuine prosody. Second, most of the existing studies of prosody among individuals with HFA have investigated either understanding or expression of prosody not both [38,39]. Third, most of the existing studies do not include positive, negative, and neutral words. Neutral words are important to include being able to evaluate how participants understand and express emotional prosody without relying on linguistic meaning.

The aim of this study was therefore to investigate the characteristics of both understanding and expression of prosody in children with ASD by using six types of words in three categories: 1) “matched-words” being a) positive prosody and positive linguistic meaning, and b) negative prosody and negative linguistic meaning, 2) “mismatched-words” being a) positive prosody and negative linguistic meaning, and b) negative prosody and positive linguistic meaning, 3) “neutral-words” being a) positive prosody and neutral linguistic meaning, and b) negative prosody and neutral linguistic meaning.

Method

Participants

The ASD group included 35 children diagnosed with HFA, Asperger syndrome (AS) or pervasive developmental disorder not otherwise specified (PDD-NOS) by a paediatrician or child psychiatrist. The children were between 6 and 15 years of age (M=9.7, SD=2.23) and 28 (80%) were boys. The children were recruited from a self-help group for children with ASD and their parents. Only children with an IQ above 70 with no general language impairments were included. Information about ASD diagnoses, IQ and general language impairment was obtained from case records and it was ensured that assessments were completed by experts. In the control group 158 children between 3 and 5 years of age were included. The children were recruited from a kindergarten in Japan and they had no ASD diagnosis, intellectual or general language impairment. The kindergarten children were divided into three separate groups according to the three grades in Japanese kindergartens: 39 third-grade children 5 years of age (17 boys (44%)), 55 second grade 4 year old children (29 boys (52%)), and 64 first grade children 3 years of age (35 boys (55%)). Thus, all participants of the ASD group were older than the control groups and above the age where understanding and expression of prosody has emerged in typically developed children (cf. introduction).

Auditory word stimuli and equipment

Three words in each of six types of word stimuli were used: (1) positive prosody and positive linguistic meaning; (2) negative prosody and negative linguistic meaning; (3) negative prosody and positive linguistic meaning; (4) positive prosody and negative linguistic meaning; (5) positive prosody and neutral linguistic meaning; (6) negative prosody and neutral linguistic meaning. In total 18 words are listed in Table 1. The selected words and test procedure were confirmed to be appropriate for the age groups by two experts in psycholinguistics.

Table 1: Words used in the understanding and expression prosody test.

A female professional actress in her thirties read the words aloud into a sound recorder (Mini Disc and CD-R). Auditory word stimuli of prosody were presented using the Mini Disc player, a CD player, or a TV (without picture). The person who conducted the test (Tester) played one word at a time and the next word was played when all subjects had given their response on separate sheets prepared for each of the words.

Experiment

In the ASD group, the tester visited and completed the tests at the self-help group meetings for children with ASD and their parents. In the control group, the tester visited the kindergarten in their spare time. All tests were conducted in a group setting. The children were asked to answer which mood (emotion) the speech was in “good mood” or “bad mood” or “don’t know” after listening to each word. All of them had to check off one mood among the three categories on a prepared sheet with emoji faces. The words were presented in random order. Before conducting the test, the tester explained to the children the meaning of the emotions: “good mood means when you feel happy or you are having fun, and bad mood means when you get angry or upset at something”. All children completed the task, there were no missing data.

Statistics

Gender differences and differences between test and control groups (children with ASD, 3, 4, and 5 year old controls) for each type of word were analysed using ANOVA statistics. For post-hoc analysis LSD tests was used.

There were no significant differences found with regard to gender in any of groups. It was therefore decided to compare the groups including both genders. The analysis showed significant differences between the groups for the matched-words (F(3, 189)=8.72, p<0.001), the mismatched-words (F(3, 189=3.464, p<0.001) and the neutralwords (F(3, 189)=9.725, p<0.001). Figure 1 displays the average number of correct answers to matched-words, mismatched-words, and neutral-words for children with ASD, and 3, 4, and 5 year old controls. The post hoc tests showed, in the matched-words condition, that the ASD group performed significantly higher than the 3 year old controls and significantly lower than the 5 year old controls. The 3 year old controls performed significantly lower than the 4 and 5 year old controls. For the mismatched-words, the ASD group performed significantly lower than the 5 year old controls and the 4 year old controls performed significantly lower than the 5 year old controls. In the neutral-words condition, the ASD group performed significantly lower than the 5 year old controls and the 3 year old controls performed significantly lower than the 4 and 5 year old controls.

Figure 1: Average and standard deviation of correct answers on the understanding of prosody test.

Method

Participants

Following the understanding of prosody test the expression of prosody test was carried out. This test took a longer time to carry out than the understanding of prosody test and was therefore completed by the tester over several days at the kindergarten and the self-help group meetings. Some of the children who participated in the understanding of prosody test did not complete the expression of prosody test because they were absent on the test days.

The ASD group included 26 children with HFA, AS or PDD-NOS diagnosed by paediatrician or child psychiatrist. They were between 6 and 15 years of age (M=9.4, SD=2.42) and 21 (81%) were boys. Similarly, only children with an IQ above 70 with no general language impairments were included. In the control groups 72 children without ASD, intellectual impairment or a general language impairment between 3 and 5 years of age participated. The children were divided into three separate groups: 30 third grade children 5 years of age (15 were boys (50%)), 30 second grade children 4 years of age (12 were boys (40%)), and 17 first-grade children 3 years of age (8 were boys (47%).

Auditory word stimuli and equipment

The same 18 words as used in the understanding of prosody test were used in the expression of prosody test. The child’s prosody (voice sound) was recorded by using a Mini Disc recorder or a personal computer.

Experiment

The test was conducted in a one-to-one setting (one child and the tester recording their voices) in a separate room at the kindergarten and a play room at the self-help group center. Before running the test, the tester explained to each child the meaning of the emotions and ensured he/she understood the instruction: “good mood means when you feel happy or you are having fun, and bad mood means when you get angry or upset at something”. The tester asked each child; “please say [one of the 18 words] in a good (or bad) mood”. The tester asked them to repeat the answer if they said the word in too quiet a voice. If the participant said that they had no idea of how to say the word, the tester told them “say what you think fits best”. All of the children completed the task. Two university students (1 male and 1 female) who knew nothing about this study listened to the children’s voice (words in good or bad mood) randomly one by one and evaluated whether the prosody (voice) was proper or not with regard to mood. Reliability was evaluated by comparing the ratings of the two raters independently for words in good and bad mood, respectively. Kappa statistics with quadratic weights were used to evaluate the degree of agreement [40]. The inter-rater reliability for both word categories was above 0.60 which, according to Fleiss and Cohen, is acceptable.

Statistics

Similar to the understanding of prosody experiment, analysis of gender and group differences (ASD, 3-, 4- and 5 year old controls) for each type of word was analysed using ANOVA statistics. For post-hoc analysis LSD tests were used.

Results

There were no significant differences found with regard to gender in any of the groups. It was therefore decided to compare the groups including both genders. The analysis revealed a significant difference between the groups in matched-words (F(3, 94)=19.96, p<0.001), mismatched-words (F(3, 94=3.769, p<0.001) and neutral-words (F(3, 94=2.187, p<0.001). Figure 2 displays the average number of correct answers of matched-words, mismatched-words and neutral-words for children with ASD, and 3-, 4-, and 5-years old controls.

Figure 2: Average and standard deviation of correct answers on the expression of prosody test.

In the matched-words condition, post-hoc tests showed that the ASD group performed significantly lower than 3-, 4- and 5-year old controls. In the mismatched-words condition the 3-year old controls performed significantly lower than the 5-year-old controls. In the neutral-words condition the ASD group was found to perform significantly lower than the 3-, 4- and 5-year old controls.

The overall finding was, in both the understanding and expression of prosody test, that the ASD group experienced more difficulties compared to the 5 year old controls. Overall, the abilities of prosody for children with ASD seem to be at the level of 3- and 4-year old typically developed children. For children with ASD compared to 3- and 4-year old controls, there were only significant differences for the matched-words and neutral-words condition in the expression of prosody test.

In the understanding of prosody test with neutral-words, the ASD group had a lower score than the 5 year old controls and 3 year old controls scored lower than 4- and 5-year old controls. It may reflect how and when development of understanding prosody in typically developed children and children with ASD takes place. Because the neutral words lack emotional information, the participants need to understand the speaker’s emotion from prosody only and not the linguistic meaning. The ability to understand prosody seems to not be developed fully among typically developed 3 year old children.

Looking at the score of mismatched-words in the understanding of prosody test, the average of correct answers was under 2 (out of 6) for all control groups. This may reveal that the ability to understand prosody with mismatched-words is not fully developed at this age. This is supported in a study by MacWhinney et al. which used a sentence understanding test among children above 6 years of age. The children performed similar to adults [41]. In contrast, functional magnetic resonance studies [42] find that 3 to 7 month old infants are able to discriminate emotional vocalizations (positive, neutral and negative). These results together may indicate that following language acquisition children tend to vacillate between emotional prosody and linguistic meaning. Subsequently, at primary school age children become progressively better at reading others’ emotions according to information from the context, facial expression, body language and gaze.

In the expression of prosody test the ASD group showed general difficulties regardless of word condition compared to the controls. The difficulties were more prominent in the expression of prosody test compared to the understanding of prosody test. In the expression of prosody test, there were no differences for the control groups for matched words and neutral words and with an average score around 4 (out of 6). This indicates that this ability starts to develop before the age of three for children without ASD. In contrast, the average of correct answers was 3 or below (out of 6) for the mismatched words for all groups. All groups were relatively low on the ability to express prosody when they had to express their emotion through the prosody with different linguistic meaning.

In the Loveland et al. study no overall differences between people with and without ASD in understanding emotion from facial expression and prosody were found [37]. However, the study sample was small and the participants were not divided into subgroups according to age. It has been found, that four year-old children rely on the verbal content to understand a speaker’s feeling when the cues conflict, while adults rely on para-language [43]. Next, a study by Ashwin et al. and Wang et al. found that the ability to understand emotions among individuals with HFPDD depended on the type of emotions [44,45]. The present study combined both age and type of emotion, which revealed significant differences, thus indicating that both are important to include in studies of prosody in children with ASD.

This study also indicates that typically developed children’s development of understanding and expression of prosody of matchedwords develops earlier than mismatched- and neutral-words. When young children understand prosody, they may be more likely to rely on linguistic meaning. It takes time to develop pragmatic language skills. Imaizumi, et al found that junior high school (13-15 years of age) children had better skills in understanding others’ emotions from sound or writing compared to elementary school (7-12 years of age) children [46]. Noguchi et al. additionally concluded that the skills of understanding irony and joking are first fully established in high school age [23].

A number of cognitive, social, and communicative characteristics among children with ASD may be involved in and linked to the difficulties of understanding and expressing prosody: Delayed and different emergence and order of joint attention in early childhood compared to typical children [47,48], difficulties with executive functions [49], syntactic language difficulties [50], Theory of Mind abilities, and pragmatic language abilities in general [51]. All these associations need to be further studied.

The limitation of this study was first of all the small sample size of the ASD group. Future studies should compare different age groups of children with ASD matched with age equivalent controls to gain a better understanding of the development of prosody skills. This study controlled for gender bias, however, due to the few number of girls (n=5 and n=7) it was difficult to conclude if boys and girls had challenges with understanding and expressing emotional prosody to the same degree and with the same characteristics. In future studies with a bigger sample size, especially girls, it is recommended to analyse separate for each gender.

This study excluded children with intellectual impairment (IQ<70), however, future studies should include full IQ scores in the analysis. Finally, future studies should control for socio-demographic variables such as parents’ educational level and income, which might also affect the child’s abilities in expression and understanding prosody.

Neuropsychology studies might add important knowledge to this study’s findings. Significant differences in brain response (by use of ERPs) between children with and without ASD processing emotional prosodic changes in natural word stimuli have been found [52]. Eigsti, et al reported significant differences in neural activation (by use of fMRI) between adolescents with and without ASD processing affective and grammatical prosody stimuli [53].

Still more research on the development of understanding and expression of prosody among children with ASD and typical children is needed to obtain more solid knowledge of these important human abilities. Knowledge about the development of understanding and expression of prosody might help in the planning of communicative support for each individual child. It might, for instance, be social skills and speech training programs which emphasize prosody skills.