The Centers for Disease Control estimates that 1 in 68 [1], children is affected by autism spectrum disorder (ASD), a neurological disorder that disrupts early development in cognition and communication. Approximately two-thirds of children with ASD grow up to have a significant cognitive and social impairment, and difficulty in acquiring new adaptive behaviors [2]. There is broad scientific consensus that early and intensive behavioral intervention can result in sizeable gains in cognitive, communication, social, academic, and adaptive skills, and has the greatest chance of significantly improving outcomes, sometimes even resulting in a complete loss of diagnosis [3-5]. Accordingly, the American Academy of Pediatrics (AAP) recommends universal screening of 18 and 24 month old children for ASD, and also that individuals diagnosed with ASD begin to receive no less than 25 hours per week of treatment within 60 days of identification [6]. Despite the AAP recommendation, two-thirds of US children on the autism spectrum under the age of 8 fail to get even the minimum recommended treatment [7] because of major problems with the availability, quality, and general funding for early intervention programs [8-10]. Since the AAP’s 2007 recommendation of universal, early screening, there has been a sharp increase in demand for ASD-related services (58% on average, [11]). However, according to a recent study, most states have reported an enormous shortage of ASD-trained personnel, including behavioral therapists (89%), speechlanguage pathologists (82%), and occupational therapists (79%) [11]. In many states children are getting less than 5 hours per week of service, and this immense shortage disproportionally affects African American and Latino children [11]. Families of newly diagnosed children often face lengthy waitlists for therapy, leaving children without treatment during the most critical, early period of development. The failure to provide adequate early intervention services ends up costing society (USA) an estimated $126 billion per year [12].

The use of technology in the delivery of much needed therapy promises to be effective for shrinking the gap between the therapy that is recommended for children with ASD and the amount they receive.

Recently, there has been a trend towards computer-based interventions which capitalize on the often-observed preference those children with ASD show for flat screen information [13-16] as well as evidence suggesting that tablets specifically have been a great learning tool for children on the spectrum [17-19]. However, since there is scant research on computer-based interventions for children with ASD, many questions remain to be answered: at what age are children diagnosed with ASD capable of being fully engaged with computer-based interventions? Critically, can the neediest population of newly diagnosed children as young as 2 with severe ASD symptoms engage with and benefit from such an intervention? Does the severity of ASD symptoms influence engagement?

In order to answer these questions, we developed a tablet-based therapeutic application for children with ASD called Mental Imagery Therapy for Autism (MITA) [20]. In this report, we describe data from a feasibility study of MITA that included 1,514 children with ASD who worked with the application for four to twelve months. We discuss MITA usability, feasibility, fidelity of implementation and promise of outcomes. To our knowledge, this report is the first to study tablet-based cognitive exercises administered to 2-year-old toddlers with autism and it is also the largest study of an early parent-administered intervention tool to toddlers with autism.

MITA’s tasks and overall curriculum are grounded in some of the best scientifically supported and established, evidence-based therapies for ASD [21], drawing most heavily from language therapy, Applied Behavioral Analysis (ABA) and Pivotal Response Training (PRT). MITA focuses on one of the four key or “pivotal” areas of development targeted by PRT that in turn affects a wide range of behaviors. Using the technique of conditional discrimination MITA targets a child’s ability to notice and respond to multiple cues presented simultaneously, a pivotal skill that is deficient in individuals with ASD and which can have a profound effect on virtually every area of functioning. To understand this ability, imagine that you are instructed to pick up a red crayon that is under the table. This may seem like a trivial task, but in order to accomplish it successfully, you need to notice three different features, or “cues” of the object: its color (red), its shape (crayon) and its location (under the table). You must then mentally integrate all three pieces of information into a new mental image, a red crayon under the table, in order to take the correct action. The ability to integrate multiple cues (called mental synthesis [22,23] is highly developed in individuals not afflicted by ASD well before the age of 6, but it is known to be a common challenge for children on the spectrum [24]. As a consequence, ASD symptoms often include a phenomenon called stimulus overselectivity, whereby an individual focuses on only one aspect of an object or environment while ignoring others [25-27]. When asked to pick up a red crayon under the table, a child with ASD may hyper-attend to the cue “crayon” and ignore both its location and the fact that it should also be red, therefore picking up any available crayon. It is often said that individuals with ASD “can't see the forest for the trees”: they pay too much attention to specific parts, get lost in the details and miss the whole picture (or Gestalt) [28]. The consequences of attempting to navigate the world with an impaired ability to respond to multiple cues can be profound and can affect virtually every area of functioning [24]. However, using PRT to develop responsively to multiple cues has been shown to reduce stimulus overselectivity and, most importantly, to lead to improvements in general learning [1,29]. In addition to developing a child’s ability to respond to multiple cues, MITA also aims to train receptive language, starting with simple vocabulary, and progressing towards higher forms of language, such as adjectives, verbs, pronouns, and syntax.

In this manuscript we report data from a feasibility study of parentadministered tablet-assisted therapy for children with ASD. We compare the performance of 1,514 children of different ages and varying ASD severities who have been working with MITA for four to twelve months, between February of 2016 and February of 2017.

We present data from an observational study of a tablet-based therapeutic application called Mental Imagery Therapy for Autism (MITA) administered by parents to 1,514 young children with ASD over the course of four to twelve months. MITA was developed by ImagiRation from 2013 to 2016 and made available for free at all major app stores in February of 2016. In the first year, MITA was downloaded 70,325 times, indicating a significant interest in supplemental therapy for ASD.

Once MITA was downloaded, the user was asked to register and provide demographic details, including the child’s diagnosis as well as month and year of birth. During twelve months (from February of 2016 to February of 2017) MITA was registered 41,690 times (59% of downloads). Table 1 shows the distribution of registration over operating systems and device types.

Table 1: MITA download statistics.

Subjects

From the pool of registered MITA users, we selected subjects based on the following criteria:

The subject’s parent must have self-reported the diagnosis as ASD: Since our primary interest is early intervention for ASD, only data from ASD subjects were analyzed for this report.

The subject’s parent must have filled out at least two ATEC questionnaires at least three months apart: Since our goal was to compare performance as a function of age and ASD severity, we needed to administer a standardized measure of ASD symptom severity. For this study, we chose to use the Autism Treatment Evaluation Checklist (ATEC), which is designed to be completed by parents, teachers, or caretakers [30]. Approximately one month after the first use of MITA and no sooner than 100 puzzles had been solved, parents were required to complete the 77 question ATEC. Subsequently, parents were asked to complete ATEC at three month intervals in order to continue their use of MITA. Since our application was available for free to the general public, we expected a large volume of downloads by people of widely ranging commitment. We used the completed ATEC evaluations not only as a means of tracking severity but also as a benchmark to discern those who had serious intentions of working with a therapeutic application and those who did not. Parents who invested the time to faithfully fillout an extensive ATEC questionnaire on two separate occasions not only gave us the requisite severity information but also demonstrated a commitment to using the application. This automatically selected subjects who used MITA for at least four months. The number of subjects who completed two or more evaluations was 2,525. Out of the 2,525 parents who completed at least two questionnaires, 1,839 (73%) self-reported their child’s diagnosis as ASD. Other subjects reported diagnoses of various other neurodevelopmental disorders or that they were not yet diagnosed. Some subjects chose not to report a diagnosis since this was not a required field.

The subject must have been 12 years of age or younger at the time of registration: Since we are interested in the effects of early intervention, we decided to limit our analysis to subjects who were 12 years of age or younger at the time of the first questionnaire. Therefore, we excluded another 323 subjects because of age. Thus, the total number of subjects included for analysis was reduced to 1,514 (5% of all potential subjects).

The number of subjects in each age group is indicated in Table 2. Seventy nine percent of subjects were male.

Table 2: The number of subjects in each age group.

Experimental interventions

All subjects in this study used a tablet-based therapeutic application called Mental Imagery Therapy for Autism (MITA) developed by ImagiRation. MITA consists of nine different developmental activities, described in detail in Supplemental Materials. All of the activities follow a systematic approach to train the skill of multiple cues responding, and eight of the activities provide this training outside of the verbal domain [20]. This unique feature allows the activities to be within reach of those individuals who are either nonverbal or minimally verbal. While these children may not be able to follow an explicit verbal command (such as “pick up the red crayon under the table”), results from our studies have demonstrated that they can follow a command implicit in the visual setup of the puzzle.

To teach children to follow implicit visual commands that require attending to multiple cues, the MITA program starts with puzzles that require attending to one cue, such as color (Figure 1A) or shape (Figure 1B). Once a child shows adequate proficiency in attending to a variety of single cues, MITA activities progress in difficulty by requiring attention to two cues simultaneously, such as both color and shape (Figure 1C) and eventually to three or more cues.

Figure 1: Examples of puzzles with increasing complexity. (a) A child must notice one cue (color) in order to correctly match the car at the top of the screen. (b) A child must notice one cue (shape) in order to correctly match the car. (c) A child must notice two cues (color and shape) in order to correctly match the car (note: the overlaid text and arrows are not seen by the child). Notice that hyper-attending to a single cue (only the shape or only the color of the vehicle) and ignoring the other cues is insufficient for finding the correct match. (d) A default daily session consists of 4 activities followed by Playtime. Each activity contains 5 puzzles marked by a circle on the diagram. See Supplemental Materials for the description of each activity.

In addition to color, shape and size, MITA activities sometimes require attention to an object’s orientation in space, the number of objects, as well as minor visual details. The choice of these particular stimuli was made to reflect those commonly used by PRT behavioral therapists who intentionally structure the therapeutic environment to include objects of various color, shape and size and then ask the child to find an object based on two (or more) of these features [14].

Each of the nine MITA activities consists of multiple levels, starting with easier levels that require attending to a single cue, moving on to intermediate levels that require attending to two cues and culminating in challenging levels that require attending to three or four cues at a time. Most activities have as many as 50 levels which range from easy to difficult in a gradual and systematic manner. Please refer to the Supplemental Materials for a detailed discussion and a variety of examples of MITA activities and levels.

Progression from one level to the next is adaptive. A child must demonstrate a high level of proficiency before being advanced to the next level by the adaptive algorithm. In order to progress to the next level of difficulty for any particular activity, a child needs to answer correctly on the first attempt for 7 out of 10 consecutive puzzles in that activity. This strict algorithm avoids the undesirable situation in which a child progresses to the next level by simply using trial-and-error without mastering a particular skill. Progression through levels is not always upward; a child is demoted down a level if he or she makes 12 or more errors in 10 consecutive puzzles. A child who is showing only moderate achievement may stay at a single level until mastery is demonstrated. The gradual increase in difficulty coupled with a strict assessment of mastery before progressing to the next step is ideal for training multiple cue responding.

Daily session: MITA activities are organized into daily sessions (Figure 1D). The minimum length of the daily session is determined by the number of activities presented on the screen as well as the number of puzzles in each activity. The minimum length of the daily session automatically adjusts based on age and performance, with predefined benchmarks based on recommendations from ABA specialists. The initial number of activities in each session was set to four activities for children younger than 6 years of age, and six activities for children older than 6. The number of activities automatically increased by one (until the maximum of six) once the child had shown progress by completing at least half of the difficulty levels in any two activities. The initial number of puzzles within each activity was set to five puzzles. In addition to these automated settings, the minimal number of activities could be manually adjusted to anywhere from one to six activities per session, and the number of puzzles within each activity could be manually adjusted to 5, 10, 15 or 20 puzzles. Once children complete all the puzzles in each of the daily session’s activities, they are rewarded with Playtime. Note that the maximum number of puzzles per session is not limited: children can reenter any activity and continue working with puzzles for as long as they want.

Following the recommendations of consulting ABA specialists, we advised parents to exercise with MITA on a daily basis for approximately 10 minutes each day until the child is able to reach the top difficulty level in each activity, which is likely to take as long as two to three years.

Outcome measurements

We were interested in looking at how subjects performed at their best, as well as their average performance over time. Thus, we created two measurements of performance:1) The Highest Achievement score and 2). The Average Performance score.

The highest achievement score: As discussed above, MITA uses adaptive algorithms to advance a subject through the levels over time. Because inconsistent patterns of attention, intellectual function and behavior are a hallmark of children on the autistic spectrum, we expected to see many subjects fluctuate in their level in any particular activity. In fact, our data show exactly such vacillations. We have observed drops of five or more difficulty levels in 342 subjects (23%) and drops in three or more difficulty levels in 732 subjects (49%).These drops were intermittent and unpredictable. For example, a subject who was doing well for months and who had reached and sustained level 20 in a particular activity may then experience a downswing and descend to level 12 over the course of weeks before beginning to rebound. Using the subject’s current level (12) would therefore not be an adequate gauge of that child’s overall achievement in this activity since he had previously succeeded in a more challenging level. To account for possible downswings, the Highest Achievement score was calculated based on the highest level reached (and sustained over the course of at least three daily session) instead of the current level. Specifically, the Highest Achievement score was calculated as a sum of sustained maximum levels reached by the child in each of the nine activities normalized by the maximum possible level in each activity. Reaching the highest possible level of difficulty in all activities corresponds to the Highest Achievement score of 100. A score of 1 corresponds to not progressing beyond the easiest level in any activity.

Since MITA consists of puzzles which incrementally increase in difficulty, it is in some sense analogous to an IQ test [22], and therefore, the Highest Achievement Score is analogous to an IQ score. Of course, this analogy only works after the subject has worked with MITA long enough for the adaptive algorithm to have sufficient data to match the subject’s actual ability level with the activity’s difficulty level. Furthermore, as the subject ages and his intellectual faculties improve, we would expect his Highest Achievement score/“MITA IQ score” to increase as well.

The average performance score: Whereas the Highest Achievement score captures a child’s best performance and disregards fluctuations, the Average Performance score encapsulates a child’s overall performance over all daily sessions. Performance is assessed after the completion of every puzzle with a score inversely proportional to the number of errors and normalized by the number of answer choices. For example, in a puzzle with one task (e.g. find the matching animal) and three answer choices (one correct and two decoys), the performance score could be 100% (subject found the correct answer on the first try), 50% (subject found the correct answer on the second try) or 0% (subject found the correct answer only after exhausting all possible options). Making more than three errors in a puzzle with only three answer choices corresponds to a performance score of 0%. Accidental drags and drops did not count as incorrect answers because we did not want to penalize subjects for poor fine motor skills. The performance scores for all the puzzles solved in a daily session are averaged into a daily performance score, which are in turn averaged into the Average Performance score.

Since the Average Performance score is an average of all daily sessions, it can be easily affected by aberrant days when the child is cognitively disengaged or going through a downswing due to sickness or a period of seizures. To avoid factoring in such anomalous activity into the score, we established a minimum threshold of 20 completed puzzles since this number corresponds to our pre-set minimum daily session (4 activities with 5 puzzles in each activity). Days when a subject was unable to solve the minimum of 20 puzzles were excluded from analysis because the subject failed to demonstrate the minimum level of commitment to engaging with the app on that day.

Autism severity measurements

To gauge symptom severity, we chose to use the Autism Treatment Evaluation Checklist (ATEC). The ATEC is a parent-administered measure which has been shown to correlate significantly with the professionally administered Childhood Autism Rating Scale (CARS) [31,32]. While we realize that a professional assessment of ASD severity may have been more reliable, the nature of the study prohibits such an approach. Clearly, it would have been logistically unfeasible to professionally assess 1,514 individuals located all over the globe. In our opinion, the advantages of the large sample size outweigh the limitations. Furthermore, as we only wanted an approximate measure of ASD severity for the purpose of classifying subjects into general categories of mild, moderate or severe ASD symptom-groups, the ATEC total score measurements was sufficient for our purposes.

Each subject was assigned their autism severity group mild, moderate, or severe based on their 77-question ATEC score [30]. Parents were asked to complete ATEC at three month intervals in order to continue their use of MITA. In addition, parents were able to complete ATEC as often as they wanted at their own discretion. As a result, over the time span of 12 months parents completed 12,912 ATEC evaluations.

During the data analysis stage, each ATEC was evaluated for internal consistency. Inconsistent ATEC evaluations (1,169 out of 12,912, or 9.1%) were likely the result of users who wanted access to the app but did not want to spend time to carefully complete 77 questions.

We assessed the consistency of each evaluation by looking for irregular patterns of subscale scores. The four subscales of ATEC reverse their questions in contiguous subscales so that a person with severe autism, for example, will likely answer “A. not true” to most of the questions in subscale 1, “C. very descriptive” in subscale 2, “A. not descriptive” in subscale 3, and “D. serious problem” in subscale 4. An insincere responder rushing through the questions may choose to always select the answers displayed either on the top or on the bottom of the list resulting in subscale I to subscale IV scores of 28, 0, 36, 0 or 0, 40, 0, 75, respectively. In either case, the inconsistent seesaw pattern of subscale scores is easily caught by an algorithm. This algorithm, however, would not be able to identify a responder who always selects the answer in the middle of the list. These responders were identified by calculating the standard deviation in each subscale. An insincere responder who always selects the same answer in the middle of the list will result in a standard deviation of zero for that section. Inconsistent ATEC evaluations with unusually low standard deviation and/or seesaw pattern of subscale scores were deleted from further analysis.

Autism severity was assigned using the average of all ATEC scores completed by a subject and the norms shown in Table 3. The decision to use the average ATEC score instead of the initial ATEC score was based on our observation of high intra-subject variability of ATEC scores. The standard deviation of subjects’ total ATEC score was on average 14%.

Table 3: ATEC autism severity norms [33].

Statistical analysis

Normally distributed data were expressed as mean ± standard deviation. Non-parametric data were presented as median with interquartile ranges (IRQ). Comparisons of normally distributed data were undertaken with unpaired Student's t-tests. A two-sided p

In this manuscript we report data from a feasibility study of parent-administered tablet-assisted therapy for children with ASD. We were especially interested in comparing the performance of subjects of different ages and varying ASD severities. We have measured and analyzed the performance of 1,514 children who have been working with the Mental Imagery Therapy for Autism (MITA) application for four to twelve months, between February of 2016 and February of 2017. We evaluated the application along four parameters: usability, feasibility, fidelity of implementation, and promise of outcomes.

Usability: Are children able to understand and physically use the application? Does usability vary between the different age groups and ASD severity groups?

When assessing the usability of MITA, we were particularly interested in evaluating the performance of the most vulnerable population: the newly-diagnosed 2 year old children who may not yet be receiving adequate therapy and those with the most severe symptoms, all of whom have the most to benefit from early and intensive therapy (Table 4).

Table 4: The number of subjects (and the percent of the total) in each age and severity category.

To assess MITA usability quantitatively we calculated the following daily averages by age and severity group:

1) The number of puzzles solved (Figure 2 and Table 5).

Figure 2: The average number of puzzles per day across the nine age and severity groups. The error bars show standard deviation. T-test was used to compare groups with the same age and the same severity. Group pairs with statistically significant difference are indicated with a horizontal bar and a corresponding P-value. There was no statistically significant difference between other groups.

Table 5: Average ± SD number of puzzles per day by age and severity group.

2) The amount of time spent engaging with the MITA app (Figure 3 and Table 6).

Figure 3: The median duration of MITA exercises per day in minutes for the nine ages and severity groups. The error bars show quartile 1 and quartile 3. There was no statistically significant difference between any of the groups.

Table 6: The median (IRQ) duration of MITA exercises per day for the nine ages and severity groups.

Average number of puzzles per day: As described above, for children younger than 6 years of age, the MITA daily session is automatically pre-set to five puzzles in each of four activities, for a total of 20 puzzles, but children can continue solving puzzles for as long as they want. Our data show that children between the ages of 2 and 6 solved, on average, 54 ± 30 puzzles per day, nearly three times the initial settings, which is a good indication of children’s proclivity for solving puzzles. Furthermore, pair-wise age group comparisons within each severity category showed no significant difference in the number of daily puzzles between the 2 to 3 year old age group (54 ± 31 puzzles on average, Table 5) and the 4 to 5 year old group (54 ± 28 puzzles on average; P-values>0.3), indicating that children as young as 2 can successfully use a therapeutic application regardless of ASD severity. In addition, pair-wise severity group comparisons within each age category showed no significant difference among any of the severity groups (P-values>0.1), which means that subjects with severe symptoms were solving just as many puzzles as those with mild symptoms in every age group.

For children who are 6 and older, the MITA daily session is automatically pre-set to 30 puzzles (five puzzles in each of six activities). Our data show that the average number of puzzles solved per day by 6 to 12-year-olds is 71 ± 42 (Table 5) which is more than twice the initial settings. As a result of the higher automated app settings for the older age group, pair-wise age comparisons within each severity category showed statistical significance in the number of puzzles solved per day by the 6 to 12 year old group compared to all other age groups, (P-values are indicted in the graph) (Figure 2).

The propensity of all subjects to continue solving puzzles well past the minimum requirement necessary to earn the reward matches the anecdotal evidence gathered from parents who have reported their child’s fondness for the application.

Average number of minutes per day: The median (IQR) amount of time spent with the MITA exercises per day was 13.3 (9.9-18.2) minutes, (Figure 3 and Table 6), which fell above our recommendation of 10 minutes per day, with no statistically significant difference among any of the age or severity groups (P-values>0.1). The small increase in the duration of MITA exercises in older children is, once again, the result of the automated app settings which require older children to complete a greater number of puzzles before earning Playtime. Notably, 19% of MITA sessions lasted more than 20 min, and 6% lasted more than 30 minutes, indicating that it is possible for kids to be engaged with a therapeutic app for a considerable amount of time.

The quantitative data allows us to conclude that children as young as 2 with a variety of ASD severities are able to physically use, understand and engage with the activities contained in a therapeutic app for at least 10 minutes per day, and are able to complete and exceed the length of a recommended daily session of activities.

Feasibility: Can the application be used over an extended period of time thus making it feasible as a therapeutic application? Is it equally feasible for every age group and severity level?

Over the span of our study, in which participants were able to exercise with MITA for four to twelve months, the median (IRQ) number of days each child engaged with MITA was 20 (14-42) and the median (IRQ) total number of puzzles solved was 1,317 (703-2,646). MITA usage indicates that children were able to come back to MITA day after day over the course of months. However, MITA exercises are expected to have their greatest effect in those who exercise consistently over several years.

To gauge MITA dropout rate, we wanted to focus on subjects who were serious about working through MITA therapy and exclude those who were just playing with the free app. The first significant hurdle for the parents is the first 77-quuestion ATEC evaluation, which was administered approximately one month after the first use of MITA and after the child had solved at least 100 puzzles. Accordingly, only these subjects (about 20% of all MITA downloads) were included into dropout rate calculations. Figure 4 shows the percent of subjects remaining in the study 3 months and 6 months after filling out the first evaluation. The data indicate a significant dropout rate, with only 12% of subjects remaining 6 months after the first evaluation.

Figure 4: Percent of subjects remaining in the study as a function of time after the first evaluation.

We were interested in checking whether weaker subjects (2 to 3 year olds with severe ASD) who showed less interest in MITA and whose parents would therefore be less motivated to fill out the questionnaire dropped out of the study at a greater rate than stronger, more engaged subjects, thus creating a positive selection process for stronger children. To address this question, we looked at the proportion of subjects who dropped out after filling out the first evaluation and before filling out the second evaluation across the nine age and severity groups (Figure 5 and Table 7). Interestingly, among the three age groups, 2 to 3 year olds were less likely to discontinue using MITA. Among 2 to 3 year olds (but not among 4 to 5 and 6 to 12 year olds), those with more severe ASD symptoms were more likely to stop using MITA. Overall, however, weaker subjects (2 to 3 year olds with severe ASD) were dropping out at a lower rate than the strongest subjects (6 to 12 year olds with mild symptoms). These data allow us to conclude that despite a handful of anecdotal examples of weaker subjects dropping out of the study, there is little evidence of an overall trend of positive selection of stronger children into the MITA study.

Figure 5: Percent of subjects who dropped out after filling out the first evaluation and before filling out the second evaluation across the nine ages and severity groups.

Table 7: Number of subjects who dropped out after filling out the first evaluation and before filling out the second evaluation across the nine ages and severity groups.

One of the main reasons cited by parents for dropping out of the study is that their child lost interest in Playtime, MITA’s primary reward structure. We are currently working to improve our incentive/reward structure by making it more varied and time-limited as well as allowing parents to personalize it to reflect their child’s unique interests, with the goal of maintaining engagement for a variety of kids over the course of years.

MITA usage data indicates it is feasible for a therapeutic application to be used over the course of months by kids as young as 2 years of age with any ASD severity, but there is likely to be significant dropout over time. For now, we have insufficient data to gauge the feasibility of multi-year usage. We are continuing our study and will be able to report in the near future.

To assess the fidelity of implementation we compared usage recommendations with actual practice data. Our two chief recommendations were:

1) That a MITA daily session should consist of at least 20 puzzles, and

2) Children should engage with MITA on a daily basis.

Our recommended session length of 20 puzzles was implicit in our pre-set minimum settings. However, as we have mentioned parents have the ability to override the recommendation to as little as 5 puzzles per day, and in addition kids could quit MITA at any time. Our data show that subjects across all age and severity groups adhered to the recommendation at least 64 ± 19% of the time (Table 8). Kids older than 6 were able to adhere to recommendations significantly better than 2 to 3-year-olds in every severity group (Figure 6), but this is again likely the result of the automated app settings which present older children with a greater number of puzzles before earning Playtime. Most notably, there was no significant difference in adherence to recommended session length among any of the severity groups (P-values>0.1).

Table 8: Days in which subjects completed a daily session as a percent of all days they started MITA.

Figure 6: Days in which subjects completed a daily session as a percent of all days they started MITA. The error bars show standard deviation. T-test was used to compare groups with the same age and the same severity. Group pairs with statistically significant difference are indicated with a horizontal bar and a corresponding P-value. There was no statistically significant difference between other groups.

To encourage the maximum impact of exercises, we recommended that MITA be used daily. Although there was one subjects who worked with MITA every day for over ten months, confirming that it is possible to engage with MITA on a daily basis, the actual median (IQR) number of days children engaged with MITA per week was 0.8 (0.5-1.5), significantly less than our recommendation. Specifically, only 200 subjects(13%) were engaged with MITA more than twice per week on average, only 72 subjects (5%) were engaged with MITA more than 3 times per week; only 28 subjects (2%) were engaged with MITA more than 4 times per week; and only 7 subjects (0.5%) were engaged with MITA more than 5 times per week. Despite these low numbers, it is interesting to note that pair-wise age and severity group comparisons showed no significant difference between the amount of usage in any of the groups, (P-values>0.2), indicating that age and severity of symptoms does not influence implementation (Figures 7 and Table 9). Even more telling, among the subjects who engaged with MITA at least twice per week, the highest proportion was among 2 to 3 year olds with severe ASD (Figure 8 and Table 10). A likely explanation for this may be parents’ motivation to work with children who do not yet receive sufficient therapy. Since the vast majority of participants were from the United States where a plethora of services are available to older children diagnosed with ASD, it is possible that this pattern of use may be explained by enrollment in other therapies. Since there are fewer services available for younger children [7] parents may be more likely to fill-in the gap by working with kids on their own. When services do become available, parents may be less motivated to administer MITA on a regular basis.

Figure 7: The median number of days per week engaged with MITA exercises for the nine ages and severity groups. The error bars show quartile 1 and quartile 3. There was no statistically significant difference between groups.

Table 9: The median (IRQ) number of days per week engaged with MITA exercises for the nine ages and severity groups.

Table 10: Subjects engaged with MITA more than twice per week as a percent of all subjects in that group.

Figure 8: Subjects engaged with MITA more than twice per week as a percent of all subjects in that group.

The quantitative data indicates that while the application is being used as it was intended as far as the daily session length, with more than 64% of daily sessions adhering to our recommendation across all ages and severity levels; it is not being used on a daily basis as recommended. Encouragingly, however, our youngest subjects with the most severe ASD symptoms constitute the highest proportion of MITA’s “most frequent users.” It will be interesting to see if improvements in MITA’s reward structure will encourage children to engage with the application more frequently.

Promise of outcomes: Can a child show improvement within an activity? Does the extent of improvement vary between the different age groups and ASD severity groups?

A child’s ability to improve was assessed by the Highest Achievement score and the Average Performance score, which measure a child’s best performance and average performance, respectively. In addition, we looked at and compared the number of errors per puzzle made by all nine age and severity groups as well as the daily change in performance score.

Highest achievement score: As described in methods, the Highest achievement score is a measure of a child’s best performance in MITA and is equal to the sum of sustained maximum levels reached by the child in each of the nine activities normalized by the maximum possible level in each activity. The Highest Achievement score, therefore, can be thought of as a child’s “MITA IQ” as it is a gauge of overall abilities. Since subjects worked with MITA for variable duration (from 4 to 12 months) and at variable intensity (from 0.1 to 7 days a week), making a direct comparison of the Highest Achievement score is not appropriate. It is clear, for example, that reaching a Highest Achievement score of 50 in 10 days is qualitatively different than reaching the same score in 300 days. Consequently, we present the Highest Achievement score as a function of days engaged with MITA.

Each red square in the nine graphs in Figure 8 represent one subject’s Highest Achievement score plotted versus the number of days the child was engaged with MITA. Since this study is ongoing, the data in Figure 9 is a snapshot of achievement as of February 2017; most of the individuals have continued to climb up the achievement ladder. Thus, Highest Achievement score at an arbitrary date is not as telling as the speed at which a child reached that score.

Figure 9: Highest achievement score vs. number of days engaged with MITA.

Figure 10 and Table 11 compare the speed of achievement (average Highest Achievement score increase per daily session) for all nine age and severity groups. All age groups and severity levels were able to increase their Highest Achievement score over time, but 6 to 12 year olds and 4 to 5 year olds increased their score at a significantly faster rate than 2 to 3 years old (Figure 10) and faster than 4 to 5 year olds in the moderate and severe categories. In other words, older children working with MITA climbed along MITA difficulty levels quicker than younger children, regardless of ASD severity. This result is not surprising, since we would generally expect older children to have higher overall achievement speed. However, we were surprised to see that there was no significant difference in the speed of achievement within any of the age groups (P-values>0.06). In other words, 2 to 3 year olds had similar speeds of achievement regardless of whether their symptoms are mild, moderate or severe; the same is true for every other age group in our study. This data is very encouraging as it suggests that MITA exercises are just as accessible and beneficial to the neediest population (individuals with severe symptoms) as it is to those with milder symptoms.

Figure 10: The speed of achievement, calculated as the average increase in highest achievement score per daily session. A speed of achievement of 1 approximately corresponds to the cumulative increment of six levels per day across all activities. The error bars show standard deviation. T-test was used to compare groups with the same age and the same severity. Group pairs with statistically significant difference are indicated with a horizontal bar and a corresponding P-value. There was no statistically significant difference between other groups.

Table 11: The speed of achievement, calculated as the average increase in highest achievement score per daily session.

Average performance score: Unlike the Highest Achievement score which considers a child’s best performance, the Average Performance score is a measure of overall performance over all daily sessions. The Average Performance score, therefore, can be thought of as a child’s cumulative “MITA grade.” As educators, we expected to see a normal Gaussian distribution with an average in the C range (70-80%), and we were hoping that none of our “students” were failing the MITA course. The Average Performance score data met our expectations (Figure 11 and Table 12). All nine age and severity groups had average scores between 70 and 78 but 6 to 12-year-olds with mild symptoms did significantly better than all comparable groups and older kids generally did better within each severity group (Figure 11). However, just as with the Highest Achievement score, we were surprised that there were no significant differences within the 2 to 3 and 4 to 5 year old age groups, meaning that ASD severity did not affect the overall performance scores (P-values>0.3).

Figure 11: The Average Performance score across the nine ages and severity groups. The error bars show standard deviation. T-test was used to compare groups with the same age and the same severity. Group pairs with statistically significant difference are indicated with a horizontal bar and a corresponding P-value. There was no statistically significant difference between other groups.

Table 12: The average performance score across the nine ages and severity groups.

Number of errors per puzzle. While the Average Performance Score accounts for the number of errors made per puzzle normalized by the number of answer choices, we wanted to look directly at error rate as an additional measure of performance. The quantitative data shows that, in general, the youngest subjects make a significantly greater number of mistakes than older subjects (Figure 12 and Table 13). Within the 6 to 12 year old age group, kids with more severe symptoms also made more mistakes. However, just as with the other performance data, we were surprised and encouraged that there was no significant differences within the 2 to 3 and the 4 to 5 year old age groups (P-values>0.3), meaning that the severity of symptoms did not affect the error rate.

Figure 12: The average number of errors per puzzle across the nine age and severity groups. The error bars show standard deviation. T-test was used to compare groups with the same age and the same severity. Group pairs with statistically significant difference are indicated with a horizontal bar and a corresponding P-value. There was no statistically significant difference between other groups.

Table 13: The Average number of errors per puzzle across the nine age and severity groups.

Arguably the most important parameter to assess the promise of outcome is the change of the Average Performance score per daily session. On average, subjects improved their Average Performance score by 0.15 ± 0.58 per day, which translates into an impressive improvement of 4.5% over 30 days of MITA use. All groups but the oldest children with severe ASD showed some improvement in their Average Performance score. Younger subjects showed significantly more improvement than 6 to 12 year-olds in the moderate and severe categories (Figure 13 and Table 14).

Figure 13: The change in the average performance score per daily session for all nine ages and severity groups. The error bars show standard deviation. T-test was used to compare groups with the same age and the same severity. Group pairs with statistically significant difference are indicated with a horizontal bar and a corresponding P-value. There was no statistically significant difference between other groups.

Table 14: The change in the Average Performance score per daily session for all nine age and severity groups

Though it is true that older kids were generally doing more difficult puzzles, it is important to note that the Average Performance score is a normalized score that should not be influenced by puzzle difficulty. To confirm this, we plotted the Average Performance score as a function of puzzle difficulty for each MITA activity (data not shown). We found no correlation between the score and puzzle difficulty confirming optimal normalization. Thus, we can conclude that the improvement of the Average Performance score over time represents a genuine improvement in performance.

In this manuscript we report data from a feasibility study of parent-administered tablet-assisted therapy for children with ASD. To our knowledge, it is the first study of tablet-based cognitive exercises intended as an early intervention for subjects with ASD as young as 2 years of age, and the largest of its kind [32]. By looking at the performance of 1,514 children who worked with the Mental Imagery Therapy for Autism (MITA) computerized learning application for four to twelve months, we conclude that

1) MITA works as designed;

2) Parents are capable of administering tablet-based therapy;

3) Children with ASD were able to engage with the MITA application independent of ASD severity;

4) Children as young as 2 years of age (as well as older children) were able to engage with MITA exercises.

Although all outcome measures in all age groups were inversely proportional to ASD severity, the effects of severity on performance were never statistically significant for 2 to 5 year-olds and only some outcome measures were statistically different in 6-12 year-olds. Children of all ASD severity levels:

1) Were climbing along difficulty levels at a comparable rate, no statistically significant difference (Figure 10 and Table 11),

2) Had comparable Average Performance scores; statistically significant difference only for 6 to 12 year olds) (Figure 11 and Table 12),

3) Had comparable number of errors per puzzle; statistically significant difference only for 6 to 12 year olds), (Figure 12 and Table 13),

4) Had comparable daily improvements in their Average Performance score; no statistically significant difference (Figure 13 and Table 14).

However, we were very encouraged by data which indicate that the severity of symptoms did not influence the performance in younger age groups. In other words, 2 to 3 and 4 to 5 year olds showed promise of outcomes regardless of whether their ASD symptoms were mild, moderate or severe. This suggests that MITA exercises are just as accessible and beneficial to the neediest population (individuals with severe symptoms) as it is to those with milder symptoms, as long as they start exercising at a young age.

In general, the effect of age on performance was much stronger than the effect of ASD severity. Younger children:

1) showed a significantly slower increase of the Highest Achievement score (Figure 10 and Table 11);

2) had significantly worse Average Performance scores (Figure 11 and Table 12); and

3) had a significantly greater number of errors per puzzle (Figure 12 and Table 13).

However younger kids also showed a significantly greater improvement in the Average Performance score each day they were engaged with MITA (Figure 13 and Table 14), underlying a considerable promise of outcomes for younger children.

It is also noteworthy that children with more severe ASD solved slightly more puzzles per day, no statistically significant difference (Figure 2 and Table 5) and spend slightly more time with MITA per day, no statistically significant difference (Figure 3 and Table 6), which is likely an indication of greater parental motivation in administering therapeutic exercises to this population.

The quantitative data gathered in this study allows us to make the following conclusions about usability, feasibility, fidelity of implementation and promise of outcomes:

a) Usability: Children as young as 2 with any ASD severity are able to physically use, understand and engage with the activities contained in the MITA therapeutic app for at least 10 min per day, and are able to complete more than 50 cognitive exercises presented in puzzle form.

b) Feasibility: It is feasible for a therapeutic application to be used over the course of months by kids as young as 2 years of age with any ASD severity, but there is likely to be significant dropout over time which may be curbed by a varied and timelimited reward structure that can be personalized by parents to reflect the unique interests of their child. For now, we have insufficient data to gauge the feasibility of multi-year usage.

c) Fidelity of implementation: Subjects are capable of using the application as intended once they have sat down for a daily session, but more work needs to be done to ensure that subjects are using the application on a frequent enough basis to result in a therapeutic benefit.

d) Promise of outcomes: Subjects of all ages and ASD severity were able to show some improvement within MITA by progressing through MITA difficulty levels and increasing their Highest Achievement Score over time. The greatest promise of outcome, however, is for the youngest subjects, who demonstrated the greatest improvements in their Performance Score as compared to older subjects.

MITA effectiveness is currently being tested in a longitudinal observational clinical trial, the goal of which is to test the hypothesis that regular practice with MITA will result not only in a greater ability to attend to multiple cues, but also in vast improvements on transfer tasks measuring visuo-spatial as well as communicative skills. Furthermore, we hope to show that MITA, coupled with an effective vocabulary training program, can lead to improvements in the realm of language comprehension. We predict that children who begin training at an early age, and who make consistent progress over the course of training, will see drastic improvements in their language function. Since many kids diagnosed with ASD are already receiving ample vocabulary training, what's missing is the skill to attend to various syntactic combinations of learned words, i.e. true flexible syntactic language comprehension. For example, a child who has learned the words “fish” “ate” and “cat” but who cannot mentally synthesize those words into a novel scene, would struggle to understand the difference between “a fish ate a cat” and “a cat ate a fish”, while a similar child who has learned mental synthesis will be successful with the task. We believe that combining the mental synthesis ability with vocabulary knowledge will result in an understanding of a full syntactic language, which will eventually lead to a significant reduction of the severity of the ASD diagnosis and ultimately to a more productive and independent life.

This observational study is exempted from IRB and informed consent according to Code of Federal Regulations, Title 45, Public Welfare, Department Of Health And Human Services, Part 46, Protection Of Human Subjects, Subpart A, Basic HHS Policy for Protection of Human Research Subjects, §46.101. “Research conducted in established or commonly accepted educational settings, involving normal educational practices, such as

(i) research on regular and special education instructional strategies, or

(ii) research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods.”