Research on the robustness of neural machine translation systems in word order perturbation
Pre-trained language model is one of the most important models in the natural language processing field, as pre-train-finetune has become the paradigm in various NLP downstream tasks.Previous studies have proved integrating pre-trained language models (e.g., BERT) into neural machine translation (NM...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
POSTS&TELECOM PRESS Co., LTD
2023-10-01
|
| Series: | 网络与信息安全学报 |
| Subjects: | |
| Online Access: | http://www.cjnis.com.cn/thesisDetails#10.11959/j.issn.2096-109x.2023078 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841529598527930368 |
|---|---|
| author | Yuran ZHAO Tang XUE Gongshen LIU |
| author_facet | Yuran ZHAO Tang XUE Gongshen LIU |
| author_sort | Yuran ZHAO |
| collection | DOAJ |
| description | Pre-trained language model is one of the most important models in the natural language processing field, as pre-train-finetune has become the paradigm in various NLP downstream tasks.Previous studies have proved integrating pre-trained language models (e.g., BERT) into neural machine translation (NMT) models can improve translation performance.However, it is still unclear whether these improvements stem from enhanced semantic or syntactic modeling capabilities, as well as how pre-trained knowledge impacts the robustness of the models.To address these questions, a systematic study was conducted to examine the syntactic ability of BERT-enhanced NMT models using probing tasks.The study revealed that the enhanced models showed proficiency in modeling word order, highlighting their syntactic modeling capabilities.In addition, an attacking method was proposed to evaluate the robustness of NMT models in handling word order.BERT-enhanced NMT models yielded better translation performance in most of the tasks, indicating that BERT can improve the robustness of NMT models.It was observed that BERT-enhanced NMT model generated poorer translations than vanilla NMT model after attacking in the English-German translation task, which meant that English BERT worsened model robustness in such a scenario.Further analyses revealed that English BERT failed to bridge the semantic gap between the original and perturbed sources, leading to more copying errors and errors in translating low-frequency words.These findings suggest that the benefits of pre-training may not always be consistent in downstream tasks, and careful consideration should be given to its usage. |
| format | Article |
| id | doaj-art-35eb7d1af33546f2a2252bb13dbb2a9d |
| institution | Kabale University |
| issn | 2096-109X |
| language | English |
| publishDate | 2023-10-01 |
| publisher | POSTS&TELECOM PRESS Co., LTD |
| record_format | Article |
| series | 网络与信息安全学报 |
| spelling | doaj-art-35eb7d1af33546f2a2252bb13dbb2a9d2025-01-15T03:17:02ZengPOSTS&TELECOM PRESS Co., LTD网络与信息安全学报2096-109X2023-10-01913814959581483Research on the robustness of neural machine translation systems in word order perturbationYuran ZHAOTang XUEGongshen LIUPre-trained language model is one of the most important models in the natural language processing field, as pre-train-finetune has become the paradigm in various NLP downstream tasks.Previous studies have proved integrating pre-trained language models (e.g., BERT) into neural machine translation (NMT) models can improve translation performance.However, it is still unclear whether these improvements stem from enhanced semantic or syntactic modeling capabilities, as well as how pre-trained knowledge impacts the robustness of the models.To address these questions, a systematic study was conducted to examine the syntactic ability of BERT-enhanced NMT models using probing tasks.The study revealed that the enhanced models showed proficiency in modeling word order, highlighting their syntactic modeling capabilities.In addition, an attacking method was proposed to evaluate the robustness of NMT models in handling word order.BERT-enhanced NMT models yielded better translation performance in most of the tasks, indicating that BERT can improve the robustness of NMT models.It was observed that BERT-enhanced NMT model generated poorer translations than vanilla NMT model after attacking in the English-German translation task, which meant that English BERT worsened model robustness in such a scenario.Further analyses revealed that English BERT failed to bridge the semantic gap between the original and perturbed sources, leading to more copying errors and errors in translating low-frequency words.These findings suggest that the benefits of pre-training may not always be consistent in downstream tasks, and careful consideration should be given to its usage.http://www.cjnis.com.cn/thesisDetails#10.11959/j.issn.2096-109x.2023078neural machine translationpre-training modelrobustness, word order |
| spellingShingle | Yuran ZHAO Tang XUE Gongshen LIU Research on the robustness of neural machine translation systems in word order perturbation 网络与信息安全学报 neural machine translation pre-training model robustness, word order |
| title | Research on the robustness of neural machine translation systems in word order perturbation |
| title_full | Research on the robustness of neural machine translation systems in word order perturbation |
| title_fullStr | Research on the robustness of neural machine translation systems in word order perturbation |
| title_full_unstemmed | Research on the robustness of neural machine translation systems in word order perturbation |
| title_short | Research on the robustness of neural machine translation systems in word order perturbation |
| title_sort | research on the robustness of neural machine translation systems in word order perturbation |
| topic | neural machine translation pre-training model robustness, word order |
| url | http://www.cjnis.com.cn/thesisDetails#10.11959/j.issn.2096-109x.2023078 |
| work_keys_str_mv | AT yuranzhao researchontherobustnessofneuralmachinetranslationsystemsinwordorderperturbation AT tangxue researchontherobustnessofneuralmachinetranslationsystemsinwordorderperturbation AT gongshenliu researchontherobustnessofneuralmachinetranslationsystemsinwordorderperturbation |