A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing
Finger joint angle (FJA) estimation, as a dynamic and fine-grained decoding mode, can support intuitive and natural human–machine interactions. This study is pioneering work dedicated to achieving accurate and clinically friendly FJA estimation. In this context, a novel surface electromyography and...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-01-01
|
| Series: | Alexandria Engineering Journal |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1110016824011499 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841553828904697856 |
|---|---|
| author | Zhouping Chen Mohamed Amin Gouda Longcheng Ji Hong Wang |
| author_facet | Zhouping Chen Mohamed Amin Gouda Longcheng Ji Hong Wang |
| author_sort | Zhouping Chen |
| collection | DOAJ |
| description | Finger joint angle (FJA) estimation, as a dynamic and fine-grained decoding mode, can support intuitive and natural human–machine interactions. This study is pioneering work dedicated to achieving accurate and clinically friendly FJA estimation. In this context, a novel surface electromyography and force myography-based multimodal multistream multilevel fusion network (MMMFNet) is proposed. MMMFNet utilises gate-recurrent units (GRU) networks to capture spatial dependencies among features and adopts a multilevel fusion strategy to extract more comprehensive representations from information sources than a single-level fusion. A cross-stream interaction (CSI) block is added between the GRU networks to focus on important information in the prediction and optimise the learning process and representation capability by facilitating information interaction among branch networks. The experimental results suggest that the MMMFNet has higher estimation accuracy (RMSE: 6.225±0.276 vs. 6.953±0.282, R2: 0.879±0.019 vs 0.846±0.021), shorter training time (4.146±0.074 min vs. 16.536±0.430 min), and lower computational complexity (5.4×106 FLOPs vs 8.8×106 FLOPs) than state-of-the-art fusion methods. Moreover, MMMFNet, which quantifies feature importance using attention weights, is the first explainable multimodal model for hand movement intention decoding. In summary, compared with state-of-the-art methods, MMMFNet can not only estimate FJAs more accurately but is also more clinically friendly. |
| format | Article |
| id | doaj-art-7dbbccba973d4e10900d1faaede100a6 |
| institution | Kabale University |
| issn | 1110-0168 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Alexandria Engineering Journal |
| spelling | doaj-art-7dbbccba973d4e10900d1faaede100a62025-01-09T06:13:20ZengElsevierAlexandria Engineering Journal1110-01682025-01-01110923A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensingZhouping Chen0Mohamed Amin Gouda1Longcheng Ji2Hong Wang3Department of Mechanical Engineering and Automation, Northeastern University, Shenyang, ChinaDepartment of Mechanical Engineering and Automation, Northeastern University, Shenyang, ChinaDepartment of Mechanical Engineering and Automation, Northeastern University, Shenyang, ChinaCorresponding author.; Department of Mechanical Engineering and Automation, Northeastern University, Shenyang, ChinaFinger joint angle (FJA) estimation, as a dynamic and fine-grained decoding mode, can support intuitive and natural human–machine interactions. This study is pioneering work dedicated to achieving accurate and clinically friendly FJA estimation. In this context, a novel surface electromyography and force myography-based multimodal multistream multilevel fusion network (MMMFNet) is proposed. MMMFNet utilises gate-recurrent units (GRU) networks to capture spatial dependencies among features and adopts a multilevel fusion strategy to extract more comprehensive representations from information sources than a single-level fusion. A cross-stream interaction (CSI) block is added between the GRU networks to focus on important information in the prediction and optimise the learning process and representation capability by facilitating information interaction among branch networks. The experimental results suggest that the MMMFNet has higher estimation accuracy (RMSE: 6.225±0.276 vs. 6.953±0.282, R2: 0.879±0.019 vs 0.846±0.021), shorter training time (4.146±0.074 min vs. 16.536±0.430 min), and lower computational complexity (5.4×106 FLOPs vs 8.8×106 FLOPs) than state-of-the-art fusion methods. Moreover, MMMFNet, which quantifies feature importance using attention weights, is the first explainable multimodal model for hand movement intention decoding. In summary, compared with state-of-the-art methods, MMMFNet can not only estimate FJAs more accurately but is also more clinically friendly.http://www.sciencedirect.com/science/article/pii/S1110016824011499 |
| spellingShingle | Zhouping Chen Mohamed Amin Gouda Longcheng Ji Hong Wang A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing Alexandria Engineering Journal |
| title | A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing |
| title_full | A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing |
| title_fullStr | A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing |
| title_full_unstemmed | A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing |
| title_short | A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing |
| title_sort | multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid semg and fmg sensing |
| url | http://www.sciencedirect.com/science/article/pii/S1110016824011499 |
| work_keys_str_mv | AT zhoupingchen amultimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing AT mohamedamingouda amultimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing AT longchengji amultimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing AT hongwang amultimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing AT zhoupingchen multimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing AT mohamedamingouda multimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing AT longchengji multimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing AT hongwang multimodalmultistreammultilevelfusionnetworkforfingerjointangleestimationwithhybridsemgandfmgsensing |