Interdisciplinary approach for the diagnosis and correction of the emotional sphere disorders of children using artificial intelligence methods: a cross-cultural study
Emotions play a vital role in human life. Emotions are manifested in behavior, facial expressions, voice and speech. Adequate manifestation and recognition of emotions is determined by the cultural environment and depends on the conditions of the child's upbringing. The range of manifestations of emotional states in children with atypical development and developmental disorders may be limited, due to the disease, atypical, reduced, inverted, do not correspond to age. Tests for assessing the emotional sphere of children are few and, as a rule, adapted to a certain age of children or to developmental characteristics. When diagnosis, the specialists takes into account the child’s emotional disorders, but there is no single standardized approach based on objective instrumental methods for assessing the manifestation and recognition of emotions. The use of a standardized approach to the assessment and diagnosis of the emotional sphere disorders of children, based on objective qualitative and quantitative data, can be applied in clinic, and pedagogical practice. Artificial intelligence methods related to recognizing emotions from a child’s voice and facial expressions can serve as an assistant to a doctor in the express diagnosis of developmental disorders. Considering the urgent needs of the medical and pedagogical community, using the latest achievements of science, we have developed the interdisciplinary methodological approach Child’s Emotional Development Method (CEDM) and used artificial intelligence methods-automatic classification of emotional speech and facial expression of children. Interdisciplinary approach using physiological, psychophysiological, psychological methods and artificial intelligence methods considers the diagnosis, psychoneurological state, and the age of the child. Different algorithms for automatic assessment of children's emotional states based on the characteristics of speech, facial expression, multimodal (audio+video) were used. For our emotion classification model, we employ common state-of-the-art techniques and base models, such as a 3D modification of ResNet CNN for video, and focus on optimizing the use of multiple modalities by employing attention-based modality fusion. The data on automatic recognition and using algorithms are the basis for the creation of an automatic system for express diagnostics of emotional disorders in children, taking into account their gender, age, and the type of developmental disorders.
Biotechnology And National Health
Technologies for human health
5 years: 2019-2024 - for Russian team, 6 years: 2019-2025 - for Indian team
Currently, the practice is being implemented in Russia and India, but in the future it may cover all BRICS countries.
The relevance of the study is caused by the theoretical significance of the problem of the emotions formation in ontogenesis in the typical and atypical development of children - on the one hand, on the other - the problem of recognizing the emotional state by the characteristics of voice and speech, facial expression. The fundamental task of the project is the development, testing and standardization of a unified approach for the diagnosis and correction of emotional disorders in children, taking into account the age, gender, cultural and linguistic affiliation of the child and the severity of psychoneurological disorders and developmental disorders. The theoretical significance of the study is to reveal the cultural features of the formation of the emotional sphere of children and the reflection of the emotional state in the characteristics of speech, voice, and facial expression of children on the material of two different language families - Indo-European (Slavic group) and Dravidian languages (Tamil group) and two different cultures – Russian and Indian. The applied aspect is to obtain new data on the reflection of emotional states in the characteristics of voice, speech, and facial expression in a sample of typically developing children and children with developmental disabilities. Many developmental disorders or atypical development of children are accompanied by a violation of the emotional sphere, which makes it difficult, and in some cases, makes it impossible social adaptation of a child. The developed standardized approach to assessing and diagnosing the emotional sphere of children, based on objective qualitative and quantitative data, can be used in clinic, speech therapy and pedagogical practice. The use different algorithms for automatic assessment of emotional states based on the characteristics of speech and facial expressions, which were previously used in the analysis of a particular language, is relevant. This approach allows us to apply data on automatic recognition of emotional states when developing interfaces and systems for teaching children with atypical development and developmental disorders, accompanied by the emotional sphere disorders. Cooperation between our two countries, India and Russia, has a long history in various fields of science, industry, medicine, and technology. In national level many pediatricians and speech therapists are working on cognitive linguistic and communication disorder classification. Our approach is distinguished by the use of a set of test tasks for children in a wide age range, taking into account the age of 5-16 years, psychoneurological status, diagnosis and upbringing conditions.
Grants of teams for current 5 years: Project of the Russian Science Foundation RSF-DST № 22-45-02007 "Development of an interdisciplinary approach for the diagnosis and correction of the emotional sphere disorders of children using artificial intelligence methods: a cross-cultural study" - 2022-2024. Russian Foundation for Basic Research №19-57-45008. 2020-2021 IND-а "Emotion classification in children’s speech by humans and machines: cross-cultural study". A publication on the implementation of the practice is being prepared in the media Chennai, India.
Lyakso E.E., Frolova O.V., Kleshnev E.A., Ruban N., Mekala A.M., Arulalan K.V. Approbation of the Child's Emotional Development Method (CEDM). ICMI' 2022. P. 201–210 Matveev A., Matveev Y., Frolova O., Nikolaev A., Lyakso E. A neural network architecture for children’s audio–visual emotion recognition. Mathematics. 2023. Vol. 11, N 22: 4573. Lyakso E., Ruban N., Frolova O., Mekala A. M. The children’s emotional speech recognition by adults: Cross-cultural study on Russian and Tamil language. PLoS ONE. 2023. Vol. 18, N. 2. Roshan M., Rawat M., Aryan K., Lyakso E., Mary Mekala A., Ruban N. Linguistic based emotion analysis using softmax over time attention mechanism. PLOS ONE. 2024
