Towards Robot-Assisted Therapy for Children With Autism—The Ontological Knowledge Models and Reinforcement Learning-Based Algorithms

1 Introduction

About 15% of the world’s population lives with some form of disability, according to the World Health Organization (WHO) (World Health Organization (WHO), 2020). Of these, 2–4% have significant functional difficulties and need assistive technology such as low-vision devices, wheelchairs, and or hearing aids. They are especially critical when a person’s quality of life decreases and their independence is reduced.

In recent years, understanding disability has shifted from a physical or medical perspective to one that considers a person’s physical, social, and ethnic political context (Altman, 2014). Today, disability is understood to be the result of the interaction between a person’s state of health or impairment and the multitude of factors influencing their environment (Field et al., 2007). Significant progress has been made in making the world more accessible to people with disabilities; however, much more work is required to meet their needs especially in psychology; because of the important role in diagnosing ASD, that can be done frequently through careful and proficient examination of medical reports, behavioral history, and interviews with friends and family to obtain more detailed and valuable information about interests, activities, and attitudes, as well as intolerances of the child or adolescent.

Advances in technology can play an important role (Du Toit et al., 2018), as they can enable people with disabilities to receive the health care necessary to lead full and independent lives. Integrated artificial intelligence (AI) researches have allowed us to explore the use of robots in different health care settings, where robots can be deployed as tools for intervention and rehabilitation programs. Robots represent a growing clinical interest in mental health services (Georgescu et al., 2019). With innovations ranging from “virtual psychotherapists” (Riva and Gaggioli, 2008; Pdxscholar and Thompson, 2016) to social robots in the treatment of dementia and autism disorders (Chu et al., 2017; Bagheri et al., 2020), artificially intelligent virtual and robotic agents are becoming increasingly popular. High-level therapeutic interventions (Fiske et al., 2020), which were previously offered exclusively by highly qualified health professionals are now performed by robots. The ethical and social implications of the increasing use of incorporated AI in mental health need to be identified and addressed to enable responsible clinical implementation (Burr et al., 2020).

There has been much interest in using social robotics to improve joint attention in children with autism spectrum disorder (ASD). In a pioneering research study, children diagnosed with ASD showed improvements in their social behaviors when interacting with robots, and this was reflected by an increase in both imitation and shared attention (Cao et al., 2020). A key aspect of their use in therapy is to involve the children in extended therapy sessions and thus keeping them focused on specific physical and social tasks (Wainer et al., 2014). A reason for which children with ASD will typically found engagement with robots stimulating is that they are simple (both in appearance and behavior), predictable, and nonintimidating agents, as humans can be in social complexity (Marchetti et al., 2020). An additional pilot study (Conti et al., 2015) involving three children who had ASD and an intellectual disability explored the efficiency of an NAO robot intervention to enhance children’s body imitation.

This paper proposes a social robot NAO that will help therapists diagnose children with autism spectrum disorders and prescribe a set of therapies according to their particular needs. This robot can deduce if a child suffers from autism by analyzing his picture, using a convolutional neural network which is the most used algorithm in the field of image processing, this algorithm will be applied to a database containing pictures of children with ASD. Then, screening tests are performed to assign an appropriate therapy plan. The robot is autonomously supervised, this means that it is independently working to reach a given therapeutic goal under the supervision of a human therapist who can, if necessary, modify the robot’s actions before they are executed to ensure that only valid therapeutic actions are performed. An ontological knowledge base is used to annotate all the information about the patient, the therapies and, and all the processes. This study applies the convolutional neural network-based algorithm on an ontological knowledge base containing data from different case studies. We start with a review of the literature and follow a discussion of the framework of the methodology we used to equip the humanoid robot to learn skills autonomously and interact with struggling children. These skills can be used to help the robot act as a therapist and prescribe therapy depending on the severity of the disease. Then we moved on to the results, discussions, and the deployment environment to test and validate our algorithms. And finally, we conclude some recommendations for future studies.

2 Related Works

The increasing availability of innovative advances in assistive technology can significantly improve people’s quality of life with disabilities. This section discusses the state of the art in the use of Social Assistant Robots for developmental and clinical purposes, considering all major phases of psychological treatment of ASD. We will present the role of these technologies according to the world classification of functioning. Next, we will present some related work on social robotics and its use for assisting children with autism spectrum disorders.

2.1 Autism Syndrome Disorder

Autism Spectrum Disorder (ASD) is a group of neurodevelopmental disorders that are present at birth and whose first manifestations may appear in early childhood (Ghosn, 2021). They affect the acquisition, assimilation, and or application of skills. They may involve dysfunction in attention, memory, perception, language, and social interaction (Hyman et al., 2020). These symptoms can be more or less present and even evolve. The concept of the spectrum allows us to integrate all the disorders and to define the combination of characteristics that the patient possesses since cases differ from one person to another (Mottron and Bzdok, 2020). The goal is to identify the strengths and weaknesses of each individual and recommend appropriate interventions. ASD is not curable, as it is not a disease, but a neurodevelopmental condition (Bölte, 2014). Nevertheless, there are ways to help the affected child. While difficulties in social interactions and communication characterize autism spectrum disorders, early interventions such as speech therapy, and behavior therapy can improve young people’s development with autism (Landa, 2018; Kasilingam et al., 2019; Kolaski, 2020). These interventions are often expensive, but social assistance robots could help the therapists, making treatment more affordable (McCreadie and Tinker, 2005; Elbeleidy et al., 2021).

2.2 Assistive Technologies

Assistive technologies refer to any article, piece of equipment, software program, or product system used to increase, maintain, or improve people with disabilities (Carver et al., 2016; Federici et al., 2017). it enhances the ability of a person with a disability to participate in daily living activities and perform tasks that would otherwise be difficult or impossible for them to complete (World Health Organisation, 2007; Giesbrecht et al., 2017). The principle of enhanced capacity includes an increased level of independent action, reduced time spent on daily living activities (Portugal et al., 2019), more choice of activities, and greater satisfaction in participating in activities (ISO/TC 173, 2016).

According to the International Classification of Functioning (Pazzaglia and Molinari, 2016), which uses disability as a term covering activity limitations, impairments, and participation restrictions, assistive technology aims to reduce limitations and impairments and promote full participation in the activities of life. In this context, technical assistance devices, or so-called Embodied assistive technology. They include those that improve structure and function (Livingstone et al., 2020), such as mobility devices (e.g., prosthetic legs, cochlear implants, and electronic implants for bladder control) (Azocar et al., 2018; Aiordachioae et al., 2020). And those that improve activity performance (e.g., voice input systems, wheelchairs for riding stairs, and communication panels) (Sasaki et al., 2020; Smith, 2020; Patel and Parmar, 2020). Specialized assistance systems (e.g., vision, hearing, cognition, and communication) (Bubar, 2020; Desideri et al., 2020; Kisanga and Kisanga, 2020; Nganji et al., 2011), as well as hardware and software resources that are software and hardware, that facilitate access to information technology (e.g., computers, mobile devices). Environmental modifications or so-called Assistive environments (e.g., automatic door openers, walk-in entrances, and accessible bathrooms) (Guay et al., 2020; Martinez-Martin et al., 2020; Hu and Hu, 2021), which reduce or eliminate restrictions on participation, are also considered types of assistive technology.

The expression of disability changes with the affected person’s environment; therefore, assistive technology devices are considered part of the environment, reducing the term of disability. They can be used, for example, to build accessibility, to increase communication, to allow access to a computer, to enable environmental control of electronic devices, to modify houses to access them, to facilitate activities personal care and family activities, to improve mobility, to stabilize seats, and to modify workplaces and schools. Nevertheless, this type of technology does not support an independent life when the user has chronic or degenerative motor limitations or cognitive abilities (Hu et al., 2020).

Assistive robotics (AR) made its appearance by integrating robotics with artificial intelligence and machine learning techniques to respond to this problem and with the primary objective of successfully promoting the well-being and autonomy of people with disabilities, taking into account societal needs. Ongoing research has concluded that Robots can assist the majority of tasks and activities of daily living, as is the case with household robots (Qian and Bi, 2015), and rehabilitation robots (Feil-Seifer and Matarić, 2005).

2.3 Social Assistive Robots

One of the main challenges in accepting assistive technology is the way it is perceived. In this sense, the interaction between the robot and the user is a crucial issue. This social interaction led to the development of social assistance robotics (SAR). In (Martins et al., 2020; Narasima Venkatesh, 2020), SAR can be defined as the intersection of AR and socially interactive robotics (SIR), whose main task is the interaction with human individuals. In recent years, social assistance robots have been tested. In 2016, IBM created the MERA (Multi-Purpose Eldercare Robot Assistant) prototype (SoftBank Mobile, 2020), based on the Pepper robot developed by SoftBank Robotics in Japan in 2014 (Arsovski et al., 2019). MERA can analyze videos of a person’s face and measure vital signs like heart and respiratory rates. It can also answer basic health-related questions and determine if an individual has had a fall (Detect Autism from a Facial Image, 2021).

3 Research Methodology

The robot should have task-oriented social behavior to reach the therapeutic goals, it should be supervised by human therapists to ensure safe and ethical behaviors, and it should provide and analyze data (e.g., user performance and performance history, robot operation) recorded in structured formats to different parties. In this section, we first present a therapeutic scenario led by a robot to diagnose a child with ASD. Then, we will model the knowledge database that the robot will use to react in each therapy session. Subsequently, we will present the method of classifying children, selecting therapy, and planning the therapy.

3.1 Modeling of General Knowledge

The representation of Knowledge consists, in a first approach, in the definition and hierarchization of the components of said Knowledge, which itself can be attached to an individual, a group (a community), or to the entire population to pass from raw information, of which the web constitutes, and an inexhaustible source to Knowledge. It is necessary to go through cognitive processing of sanitization (attribution of the meaning) and synthesis. The second approach, oriented towards practice, is schematized by the Semantic Graph: a set of elements of which specific pairs, or multiples, are directly linked by so-called semantic links. The links are oriented and qualified, in particular by a tree structure and a predefined ontological definition. A Profile element, which represents the fundamental element to which a set forming a “Knowledge” is attached, determines the origin (manual, synthetic), as well as the editing levels authorized on the links. Overall, the design (General) consists of documentation work (research, collection, organization of information). In this sense, we have tried to give a general presentation of a situation, where a robot therapist is in direct contact with a child (Figure 1), it can record all the information about him by invoking a conversation, and then store this information in an ontological database explained in Section 3.3. The first step is when the robot will detect if the child is ASD positive by analyzing facial features in his picture using an ASD detection algorithm, this is mentioned in Section 3.4. The used algorithm is trained on the database presented in Section 3.2. The second step consists of doing screening tests to better detect disorders the child suffers from, and then choose the priority list of therapies that can be performed to improve his condition, a detailed description is in Section 3.5. Finally, the third step is the planning of therapies followed by their validation to give feedback to the human therapist who will make the final diagnosis of the child’s condition.

FIGURE 1. Overview of the therapy procedure with robots for children with ASD.

3.2 Database Used: Structures and Metadata

The objective is to provide a primary diagnostic that can help the robot therapist decide whether the child has ASD. For this, we used version 9 of the Kaggle database “Detect Autism from a facial image,” (de Mello and de Souza, 2019) containing 2,536 images divided equally between two classes, either autistic and non-autistic; this is in three folders, learning, test, and validation. The validation images are located in a valid directory. It is also separated into 50 images of autistic children and 50 photos of non-autistic children in the same format as for the training set. The second way the data is provided is in the consolidated directory. This directory also contains the two subdirectories Autistic and Non_Autistic. It represents the consolidation of the train’s files, test, and valid directories into a single set. Users can then partition the consolidated data into their train, test, and validation sets. The pictures vary in size and do not show the child’s face—the children in the age distribution of about 2 years to 14 years. Gender relations are close to their respective populations. Men are diagnosed with autism 3 times more than women. Thus, the distribution of male to female images in the autistic class is close to 3 to 1. In the non-autistic class, the ratio is much closer to 1 to 1.

3.3 Ontological Knowledge Database for Therapy Automation

Psychotherapy helps patients become aware of and change their behavior in the face of immediate emotional conflict and initiate the transformation process through listening, observation, awareness, and interventions (Janssen et al., 2019). And with the rapid development of new types of sensors, memory banks, computers, and electronic control devices, automation has been introduced in many diverse human activity fields (Ogg and Rašticová, 2020). In industrial production, its use brings about a radical change in the existing patterns of working life (Lippi and Da Rin, 2019). It completely changed the scope and speed of mathematical analysis and information research. Attempts are currently underway to apply automation to certain aspects of medical diagnosis and hence to psychotherapy procedures (Cameron et al., 1964; Goldberg et al., 2020; Casas-Bocanegra et al., 2020).

The ontological model of the assistive robot therapist is shown in Figure 2. Ontology tries to combine all clinical concepts to the elements of interaction to provide a model containing both parts; patient and robot; in therapy. The ontology represents all the information contained in the basic knowledge, about the patient information, the screening tests, the set of interventions, and therapies; used in the architecture. We consider all the actions that the robot can perform in each exercise, and therefore in each therapy, and any unexpected situations. These actions can be gestural or sound. The robot control engine is the decision support component. The latter uses the domain and receives the problem with the adapted exercise set up to provide a valid action plan that meets the learning objectives.

FIGURE 2. The ontological model of a robot therapist.

3.4 Classification of Patients Model and Detection of Symptoms

When the robot is in direct contact with the child, it will need information on its profile, some of which can be collected from the interaction protocol. This information is stored in three databases. These are databases of questionnaires, behaviors (images, videos), and physiological signals. The robot will need a machine learning mechanism to decide if the child has ASD.

A recent study about autism has shown that children with autism have a distinct facial structure that differs from that of typically developing controls; We chose to use convolutional neural networks because they are the most efficient models to classify images. This algorithm is different from a neural network because it works on an input volume. It is not necessary to extract features. In this case, the system learns to do feature extraction and uses image convolution and filters to generate invariant features passed to the next layer. Each layer tries to find a useful pattern or information about the data. Figure 3 shows what the architecture of a CNN (Lin et al., 2021) looks like.

FIGURE 3. Feature extraction and classification.

In the symptom detection part, it will be essential for us to choose the appropriate therapy for each child. To do this, we will use a community detection algorithm (McGuire, 1983), which we have developed in our related work. The goal is to classify patients with autism into three communities (Figure 4) that’ll designate the disease’s severity (low, medium, and high).

FIGURE 4. The child classification and symptom discovery model.

3.5 Therapy Selection Model

The selection can be made based on the three resulting communities; therefore, we can predict individual responses to different therapy options using various patient data and recommend which therapy is believed to provide the best result. For this, we will need a clinical database integrating patients suffering from ASD disease and the therapies proposed for each. This phase will also allow us to adjust therapies to suit the patient’s needs (Gutierrez and Sequeda, 2021), as shown in one of our previously published papers. And therefore, to offer therapy to a child. You just have to look for the community to which he belongs to decide which therapy will benefit him. Another method based on screening tests can be performed; based on the routine tests that therapists do to facilitate diagnosis. Another method based on screening tests can be performed (Albawi et al., 2018); based on the routine tests that therapists do; to facilitate diagnosis. This method was designed after extensive research work on screening tests and prescribed interventions for each set of disorders.

3.5.1 Psychometric Tests and Functional Assessment Tools

There are screening and assessment tools that can be used to identify warning signs in children with autism (Gutierrez and Sequeda, 2021). The best known and most widely used is the M-Chat test (Modified Checklist for Autism in Toddlers) (Javed et al., 2018). This questionnaire focuses on the child’s behavior and can be administered as early as 18 months of age. The tests and assessments to establish a diagnosis will come later. Functional assessment tests and tools help establish a diagnosis of Autism Spectrum Disorder. Each test provides an assessment of communication, social and behavioral skills, depending on the age of the candidates, whether they are children, adolescents, or adults. Table 1 represents all psychometric tests by age group. They are used to examine behaviors and skills.

TABLE 1. Psychometric tests for all age ranges.

3.5.2 Interventions Used for ASD

As soon as an autism diagnosis is made, children, and adults must be able to benefit from adapted interventions. These interventions aim to improve the quality of life of people with ASD through actions that promote communication, learning, and socialization while reducing behavioral problems. Table 2 shows us the list of therapies used for ASD.

TABLE 2. Interventions Used for ASD based on troubles type.

3.5.3 An Algorithm of Description the Set of Therapies

After having compared the two preceding tables, we were able to construct a matrix of correspondence between Disorder and therapies, that would allow us to deduce the therapies that must be prescribed for each autistic patient according to his case (Table 3). The columns of the matrix represent screening tests by age group; for children, for adults and adolescents, and tests for all ages. the lines of the matrix represent the existing therapies. If a test is positive, we prescribe the patient the therapies that have the value 1. For example, if the patient is a child, and if the ADI-R test is positive, we will prescribe the therapies: developmental, behavioral, and communication. Thus, we developed a therapy selection algorithm which process is described simply by Algorithme1 below in Figure 5:

TABLE 3. Disorder/therapy correspondence matrix.

FIGURE 5. Therapy selection algorithm.

This algorithm takes as input the child class CC, the age range of patient A and the correspondence matrix M, and returns as output a Sorted Map of the prescribed therapies PTM which is sorted in descending order of peer values. Each pair represents a therapy and its weight which the weight of the therapy is calculated by the number of times a positive screening test STS corresponds to this therapy the priority of therapy is given by its weight which indicates how many screening tests are required to do this specific therapy. we have as output a list of prioritized therapies. the decision will be up to the therapist to choose if the robot will proceed with the first therapy, follow all therapies or cancel a therapy.

3.5.4 Therapy Planning and Scheduling Model

As already mentioned in the previous sections, each therapy consists of a sequence of previously planned exercises (Gbedawo, 2020). However, the output of the high-level planner does not indicate the order of the exercises to be performed (Vivanti, 2020). To control the therapy, the robot uses a low-level scheduling mechanism. Figure 6 represents the robot therapy flow (Billeci et al., 2020).

FIGURE 6. The flow of therapy by the robot.

Indeed, once the robot is running, it goes into a detection position. After the patient has been detected, he will move on to the phase of downloading his profile to generate the therapy plan and the progress thereof in the latter. As mentioned in the previous sections, therapy consists of a set of exercises that constitute a sequence of perceptions to be performed. The robot controller turns on to execute its perceptions one after the other, and check if they are produced correctly by the patient, otherwise, it corrects them until the end of the sequence.

4 Results and Discussion

In this section, we present some of the results obtained by following the methodology described in Section 3. Figure 7 summarizes in some way the path that we have followed.

FIGURE 7. Input, outputs, and processing levels used by the robot therapist’s platform.

4.1 The Construction of the Classification Model Based on the CNN and the TensorFlow Framework.

In this section, we have tried to build the classification part of the children. We used the convolutional neural networks of the TensorFlow Framework for Deep Learning. The test database is “Detect Autism from a facial image” (de Mello and de Souza, 2019) which we described in the data structure section. Figure 8 shows some of the images contained in the database.

FIGURE 8. Part of the images from the database. Detect Autism from a facial image (Guay et al., 2020).

We used 100 epochs to train our model. Figure 9 shows the results of the training. Precision was used to measure the performance of the model during training. And we can see that the model achieves 86% accuracy on the validation set in epoch 100. After that, we plotted the model loss and accuracy data. Figure 10 shows the curves plotted with the designation of the best epoch. We see that our model’s best epoch is number 91, with 88% accuracy and 38% loss. We can see that we need to add epochs to know where the pattern will start to readjust.

FIGURE 9. Part of the training results (from epoch 93 to 100).

FIGURE 10. The loss and precision data curves for 100 epochs.

4.2 The Simulator Used for Model Deployment

In our research, we used the humanoid robot Nao (Figure 11) as a development platform and retrieve medical data from patients. Nao is an interactive, autonomous, and fully programmable humanoid robot manufactured by the company Aldebaran Robotics. It is mainly used for research in universities and laboratories, in education, and also with autistic children and Alzheimer’s patients. In our work, we used Nao in its 6th version which has the following features: Embedded OS: OpenNAO (embedded GNU/Linux based on Gentoo), a Middleware: NAOqi 2.8.6. The NAOqi environment is designed for simple voice interaction and includes a dialog engine, an emotional engine, and an autonomous life. We used also a Programming tool: Choregraphe (graphical programming interface to create and edit movements and edit interactive movements and behaviors) and a Programming language: Drag&Drop, C++, Python, and Java.

FIGURE 11. Nao specifications.

4.3 Integration and Tests

The contextualization of the process from the detection of autism to the selection of therapies has been put in the form of a Choregraphe program which is shown in Figure 12. This behavior is defined in solitary mode and starts only when a person is in the field of view of the robot for at least 3 s and/or it is a preferred moment defined by the patient.

FIGURE 12. Choregraphe program for the contextualization of the process.

The organization of the boxes in the Choregraphe program follows the flowchart shown in Figure 12.

- First of all, the robot must be in “Basic Awareness” mode, which means that it can establish and maintain eye contact. It is then able to react to stimuli such as sounds, movements, face detection, or touch. From this mode, the robot can follow a face, and recognize it.

- The robot checks that the time is appropriate for the examination (e.g., it is not at night; it is not an unavailable time defined by the patient).

- The robot recognizes the face of the person it has in its visual area and checks that it is the patient. If the person is not recognized, it takes a picture of him and adds it to its database. At the same time, he asks questions to complete the patient chart and if the robot has waited 7 s without anyone appearing in its view, the application stops.

- If the patient is recognized, the robot retrieves its photo and classifies it using the CNN model (which we discussed in Section 4.1) that has been trained beforehand and stored in the robot.

- If the result of the classification is “NoAutistic” then the robot informs us that the patient does not have an autism spectrum disorder, otherwise it starts the screening tests to deduce the troubles the patient has.

- For each disorder, the interventions that can be prescribed are recorded in the form of the disorder/therapy correspondence matrix in the corresponding SQL database to define and recommend an “effective treatment”. Once the treatment is prescribed, the robot records it in the patient’s chart.

5 Conclusion

Diagnosing autism is a complicated and expensive process. Fortunately, the correlation between facial features and autism means that a model can be trained to detect it. Using this model, the results can be responsibly communicated by a robot therapist. This article has proposed a general architecture of a robot therapist that can detect if a child has autism spectrum disorder, and offered the majority of modules that this one will need to prescript therapy. The detection of children with ASD was done through the Deep Learning classification algorithm, based on convolutional neural networks, which gave an accuracy of 88% in addition, we have implemented a robot that will be used later to implement our algorithms, and interact with autistic children. In our future work, we want to readjust our model to have more precision. This will be carried out in one of the collaborating laboratories within the framework of the H2020 project of which our laboratory is part. Moreover, real clinical tests on ASD patients will be done by using the matrix of correspondence to select the set of therapies, that will allow us to collect a new database about ASD patient therapies. We will apply fuzzy logic to the correspondence matrix to have more precision. After that, we will apply a deep learning algorithm to the new database to select therapies in a very efficient way.

Data Availability Statement

Publicly available datasets were analyzed in this study. This data can be found here: https://www.kaggle.com/gpiosenka/autistic-children-data-settraintestvalidate/metadata.

Ethics Statement

Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author Contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Funding

This research has been partially supported by the project “CybSPEED: Cyber-Physical Systems for Pedagogical Rehabilitation in Special Education” H2020-MSCA-RISE-2017. This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 777720.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor is currently co-organizing a Research Topic with one of the authors (VK), and confirms the absence of any other collaboration.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References