m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children
Challenges associated with autism spectrum disorder (ASD) include deficits in interpersonal communication, social interaction skills, and behavior. Autistic children experience difficulties in recognizing emotions and expressing emotions, along with intense emotional upheavals called meltdowns. Thes...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10535108/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841556990668570624 |
|---|---|
| author | Asha Kurian Shikha Tripathi |
| author_facet | Asha Kurian Shikha Tripathi |
| author_sort | Asha Kurian |
| collection | DOAJ |
| description | Challenges associated with autism spectrum disorder (ASD) include deficits in interpersonal communication, social interaction skills, and behavior. Autistic children experience difficulties in recognizing emotions and expressing emotions, along with intense emotional upheavals called meltdowns. These outbreaks lead to immense physical and emotional distress in children with autism. Generalized emotion recognition classifiers cannot handle the variations in the prototypical display of affect experienced by ASD children. This paper looks at developing a personalized multimodal neural framework, m_AutNet, that can effectively identify the emotions of autistic children by combining data from their facial and vocal expression modalities. The proposed network includes a personalized facial feature extraction module (that incorporates a distance metric to cluster embeddings with similar labels together and marginalizes dissimilar embeddings), and an audio modality CNN feature extractor that works on speech expression samples of autistic children. Domain adaptation of the multimodal features is achieved through a generative adversarial network tuned with the Wasserstein metric to form a domain-invariant distribution alignment of the feature vectors. A classifier performs emotion classification on this domain space following adaptation. The proposed algorithm shows higher performance than state-of-the-art affect recognition classifiers for autistic children, with an accuracy of 88.25%. |
| format | Article |
| id | doaj-art-16e054bf7dfc4c17a3f6be289505a9fe |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-16e054bf7dfc4c17a3f6be289505a9fe2025-01-07T00:01:35ZengIEEEIEEE Access2169-35362025-01-01131651166210.1109/ACCESS.2024.340308710535108m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic ChildrenAsha Kurian0https://orcid.org/0009-0003-6909-5247Shikha Tripathi1https://orcid.org/0000-0001-8123-5570Department of Computer Science and Engineering, PES University, EC Campus, Bengaluru, Karnataka, IndiaDepartment of Electronics and Communication Engineering, PES University, RR Campus, Bengaluru, Karnataka, IndiaChallenges associated with autism spectrum disorder (ASD) include deficits in interpersonal communication, social interaction skills, and behavior. Autistic children experience difficulties in recognizing emotions and expressing emotions, along with intense emotional upheavals called meltdowns. These outbreaks lead to immense physical and emotional distress in children with autism. Generalized emotion recognition classifiers cannot handle the variations in the prototypical display of affect experienced by ASD children. This paper looks at developing a personalized multimodal neural framework, m_AutNet, that can effectively identify the emotions of autistic children by combining data from their facial and vocal expression modalities. The proposed network includes a personalized facial feature extraction module (that incorporates a distance metric to cluster embeddings with similar labels together and marginalizes dissimilar embeddings), and an audio modality CNN feature extractor that works on speech expression samples of autistic children. Domain adaptation of the multimodal features is achieved through a generative adversarial network tuned with the Wasserstein metric to form a domain-invariant distribution alignment of the feature vectors. A classifier performs emotion classification on this domain space following adaptation. The proposed algorithm shows higher performance than state-of-the-art affect recognition classifiers for autistic children, with an accuracy of 88.25%.https://ieeexplore.ieee.org/document/10535108/Autismaffective computingdomain adaptationdata fusiongenerative adversarial networkmultimodal neural network |
| spellingShingle | Asha Kurian Shikha Tripathi m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children IEEE Access Autism affective computing domain adaptation data fusion generative adversarial network multimodal neural network |
| title | m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children |
| title_full | m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children |
| title_fullStr | m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children |
| title_full_unstemmed | m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children |
| title_short | m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children |
| title_sort | m autnet x2013 a framework for personalized multimodal emotion recognition in autistic children |
| topic | Autism affective computing domain adaptation data fusion generative adversarial network multimodal neural network |
| url | https://ieeexplore.ieee.org/document/10535108/ |
| work_keys_str_mv | AT ashakurian mautnetx2013aframeworkforpersonalizedmultimodalemotionrecognitioninautisticchildren AT shikhatripathi mautnetx2013aframeworkforpersonalizedmultimodalemotionrecognitioninautisticchildren |