Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays
ChatGPT, an innovative large language model that has impressed worldwide audiences with its exceptional generative capabilities, is now positioned to significantly transform the field of education. The purpose of this exploratory study is to investigate how accurately ChatGPT generates feedback on t...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Sakarya University
2024-12-01
|
| Series: | Sakarya University Journal of Education |
| Subjects: | |
| Online Access: | https://dergipark.org.tr/en/download/article-file/3893319 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841556417756004352 |
|---|---|
| author | Berkan Çelik Behice Ceyda Cengiz Amine Hatun Ataş |
| author_facet | Berkan Çelik Behice Ceyda Cengiz Amine Hatun Ataş |
| author_sort | Berkan Çelik |
| collection | DOAJ |
| description | ChatGPT, an innovative large language model that has impressed worldwide audiences with its exceptional generative capabilities, is now positioned to significantly transform the field of education. The purpose of this exploratory study is to investigate how accurately ChatGPT generates feedback on the content and organization components of EFL compare and contrast essays and the extent to which the feedback length provided by ChatGPT differs from that of the human teacher.To address these questions, a ChatGPT prompt incorporating evaluation criteria for content and organization components was developed, generating feedback on 10 compare and contrast student essays using the ChatGPT 3.5 version. The ChatGPT feedback and teacher feedback were assessed quantitatively and qualitatively according to the predetermined evaluation criteria. Furthermore, two types of feedback were compared descriptively and by conducting the Wilcoxon Sign Rank Test. The findings revealed that ChatGPT produced highly accurate feedback for both content and organization components, surpassing the teacher in the length of feedback provided. While the accuracy rate of the generated feedback was high, issues such as holistic assessment of the essay, false positives, failure to provide feedback where needed, and discrepancies in the depth of feedback compared to teacher feedback were identified. The results suggest that while ChatGPT shows promise in providing educational feedback, teacher-AI collaboration in giving feedback for EFL compare and contrast essays is important for delivering feedback that optimally benefits learners. |
| format | Article |
| id | doaj-art-54082efab3f3470db8bb9cba6ce917f7 |
| institution | Kabale University |
| issn | 2146-7455 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Sakarya University |
| record_format | Article |
| series | Sakarya University Journal of Education |
| spelling | doaj-art-54082efab3f3470db8bb9cba6ce917f72025-01-07T08:59:44ZengSakarya UniversitySakarya University Journal of Education2146-74552024-12-0114351553810.19126/suje.147547428Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast EssaysBerkan Çelik0https://orcid.org/0000-0002-7068-8918Behice Ceyda Cengiz1https://orcid.org/0000-0002-9963-434XAmine Hatun Ataş2https://orcid.org/0000-0001-6325-353XYuzuncu Yil University, Başkale Vocational School, Department of Computer TechnologiesZonguldak Bulent Ecevit University, Faculty of Humanities and Social Sciences, Department of Translation and InterpretingGalatasaray University, Distance Education Application and Research CenterChatGPT, an innovative large language model that has impressed worldwide audiences with its exceptional generative capabilities, is now positioned to significantly transform the field of education. The purpose of this exploratory study is to investigate how accurately ChatGPT generates feedback on the content and organization components of EFL compare and contrast essays and the extent to which the feedback length provided by ChatGPT differs from that of the human teacher.To address these questions, a ChatGPT prompt incorporating evaluation criteria for content and organization components was developed, generating feedback on 10 compare and contrast student essays using the ChatGPT 3.5 version. The ChatGPT feedback and teacher feedback were assessed quantitatively and qualitatively according to the predetermined evaluation criteria. Furthermore, two types of feedback were compared descriptively and by conducting the Wilcoxon Sign Rank Test. The findings revealed that ChatGPT produced highly accurate feedback for both content and organization components, surpassing the teacher in the length of feedback provided. While the accuracy rate of the generated feedback was high, issues such as holistic assessment of the essay, false positives, failure to provide feedback where needed, and discrepancies in the depth of feedback compared to teacher feedback were identified. The results suggest that while ChatGPT shows promise in providing educational feedback, teacher-AI collaboration in giving feedback for EFL compare and contrast essays is important for delivering feedback that optimally benefits learners.https://dergipark.org.tr/en/download/article-file/3893319artificial intelligencecompare and contrast essaychatgpteflfeedback |
| spellingShingle | Berkan Çelik Behice Ceyda Cengiz Amine Hatun Ataş Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays Sakarya University Journal of Education artificial intelligence compare and contrast essay chatgpt efl feedback |
| title | Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays |
| title_full | Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays |
| title_fullStr | Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays |
| title_full_unstemmed | Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays |
| title_short | Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays |
| title_sort | evaluating chatgpt in generating feedback on content and organization components of efl compare and contrast essays |
| topic | artificial intelligence compare and contrast essay chatgpt efl feedback |
| url | https://dergipark.org.tr/en/download/article-file/3893319 |
| work_keys_str_mv | AT berkancelik evaluatingchatgptingeneratingfeedbackoncontentandorganizationcomponentsofeflcompareandcontrastessays AT behiceceydacengiz evaluatingchatgptingeneratingfeedbackoncontentandorganizationcomponentsofeflcompareandcontrastessays AT aminehatunatas evaluatingchatgptingeneratingfeedbackoncontentandorganizationcomponentsofeflcompareandcontrastessays |