Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization
Despite the inherent complexity of Abstractive Text Summarization, which is widely acknowledged as one of the most challenging tasks in the field of natural language processing, transformer-based models have emerged as an effective solution capable of delivering highly accurate and coherent summarie...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Sakarya University
2024-12-01
|
| Series: | Sakarya University Journal of Computer and Information Sciences |
| Subjects: | |
| Online Access: | https://dergipark.org.tr/en/download/article-file/4019375 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841556361166454784 |
|---|---|
| author | Sevinç İlhan Omurca Nihal Zuhal Kayalı |
| author_facet | Sevinç İlhan Omurca Nihal Zuhal Kayalı |
| author_sort | Sevinç İlhan Omurca |
| collection | DOAJ |
| description | Despite the inherent complexity of Abstractive Text Summarization, which is widely acknowledged as one of the most challenging tasks in the field of natural language processing, transformer-based models have emerged as an effective solution capable of delivering highly accurate and coherent summaries. In this study, the effectiveness of transformer-based text summarization models for Turkish language is investigated. For this purpose, we utilize BERTurk, mT5 and mBART as transformer-based encoder-decoder models. Each of the models was trained separately with MLSUM, TR-News, WikiLingua and Fırat_DS datasets. While obtaining experimental results, various optimizations were made in the summary functions of the models. Our study makes an important contribution to the limited Turkish text summarization literature by comparing the performance of different language models on existing Turkish datasets. We first evaluate ROUGE, BERTScore, FastText-based Cosine Similarity and Novelty Rate metrics separately for each model and dataset, then normalize and combine the scores we obtain to obtain a multidimensional score. We validate our innovative approach by comparing the summaries produced with the human evaluation results. |
| format | Article |
| id | doaj-art-7aed6e8c5b074a5bb30bce3e71ba5f6a |
| institution | Kabale University |
| issn | 2636-8129 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Sakarya University |
| record_format | Article |
| series | Sakarya University Journal of Computer and Information Sciences |
| spelling | doaj-art-7aed6e8c5b074a5bb30bce3e71ba5f6a2025-01-07T09:08:00ZengSakarya UniversitySakarya University Journal of Computer and Information Sciences2636-81292024-12-017334636010.35377/saucis...150438828Evaluation-Focused Multidimensional Score for Turkish Abstractive Text SummarizationSevinç İlhan Omurca0https://orcid.org/0000-0003-1214-9235Nihal Zuhal Kayalı1https://orcid.org/0000-0002-6545-173XKOCAELİ ÜNİVERSİTESİ, MÜHENDİSLİK FAKÜLTESİ, BİLGİSAYAR MÜHENDİSLİĞİ BÖLÜMÜTÜRK-ALMAN ÜNİVERSİTESİDespite the inherent complexity of Abstractive Text Summarization, which is widely acknowledged as one of the most challenging tasks in the field of natural language processing, transformer-based models have emerged as an effective solution capable of delivering highly accurate and coherent summaries. In this study, the effectiveness of transformer-based text summarization models for Turkish language is investigated. For this purpose, we utilize BERTurk, mT5 and mBART as transformer-based encoder-decoder models. Each of the models was trained separately with MLSUM, TR-News, WikiLingua and Fırat_DS datasets. While obtaining experimental results, various optimizations were made in the summary functions of the models. Our study makes an important contribution to the limited Turkish text summarization literature by comparing the performance of different language models on existing Turkish datasets. We first evaluate ROUGE, BERTScore, FastText-based Cosine Similarity and Novelty Rate metrics separately for each model and dataset, then normalize and combine the scores we obtain to obtain a multidimensional score. We validate our innovative approach by comparing the summaries produced with the human evaluation results.https://dergipark.org.tr/en/download/article-file/4019375natural language processingabstractive summarizationtransformersevaluation metricsrouge |
| spellingShingle | Sevinç İlhan Omurca Nihal Zuhal Kayalı Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization Sakarya University Journal of Computer and Information Sciences natural language processing abstractive summarization transformers evaluation metrics rouge |
| title | Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization |
| title_full | Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization |
| title_fullStr | Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization |
| title_full_unstemmed | Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization |
| title_short | Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization |
| title_sort | evaluation focused multidimensional score for turkish abstractive text summarization |
| topic | natural language processing abstractive summarization transformers evaluation metrics rouge |
| url | https://dergipark.org.tr/en/download/article-file/4019375 |
| work_keys_str_mv | AT sevincilhanomurca evaluationfocusedmultidimensionalscoreforturkishabstractivetextsummarization AT nihalzuhalkayalı evaluationfocusedmultidimensionalscoreforturkishabstractivetextsummarization |