Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With Feedback
Automatic short answer scoring (ASAS) helps reduce the grading burden on educators but often lacks detailed, explainable feedback. Existing methods in ASAS with feedback (ASAS-F) rely on fine-tuning language models with limited datasets, which is resource-intensive and struggles to generalize across...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10771759/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846123655500660736 |
|---|---|
| author | Menna Fateen Bo Wang Tsunenori Mine |
| author_facet | Menna Fateen Bo Wang Tsunenori Mine |
| author_sort | Menna Fateen |
| collection | DOAJ |
| description | Automatic short answer scoring (ASAS) helps reduce the grading burden on educators but often lacks detailed, explainable feedback. Existing methods in ASAS with feedback (ASAS-F) rely on fine-tuning language models with limited datasets, which is resource-intensive and struggles to generalize across contexts. Recent approaches using large language models (LLMs) have focused on scoring without extensive fine-tuning. However, they often rely heavily on prompt engineering and either fail to generate elaborated feedback or do not adequately evaluate it. In this paper, we propose a modular retrieval augmented generation (RAG) based ASAS-F system, utilizing RAG as a few-shot selection method to score answers and generate feedback in zero-shot and few-shot learning scenarios. We design our system to be adaptable without extensive prompt engineering using an automatic prompt generation framework. Results show an improvement in scoring accuracy by 9% on unseen questions compared to fine-tuning, offering a scalable and cost-effective solution. |
| format | Article |
| id | doaj-art-c781c278b6ec4fc9ba4aabca1cb5cae5 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-c781c278b6ec4fc9ba4aabca1cb5cae52024-12-14T00:01:13ZengIEEEIEEE Access2169-35362024-01-011218537118538510.1109/ACCESS.2024.350874710771759Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With FeedbackMenna Fateen0https://orcid.org/0000-0002-2892-1202Bo Wang1https://orcid.org/0000-0001-7587-5141Tsunenori Mine2https://orcid.org/0000-0002-7462-8074Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, JapanGraduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, JapanFaculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, JapanAutomatic short answer scoring (ASAS) helps reduce the grading burden on educators but often lacks detailed, explainable feedback. Existing methods in ASAS with feedback (ASAS-F) rely on fine-tuning language models with limited datasets, which is resource-intensive and struggles to generalize across contexts. Recent approaches using large language models (LLMs) have focused on scoring without extensive fine-tuning. However, they often rely heavily on prompt engineering and either fail to generate elaborated feedback or do not adequately evaluate it. In this paper, we propose a modular retrieval augmented generation (RAG) based ASAS-F system, utilizing RAG as a few-shot selection method to score answers and generate feedback in zero-shot and few-shot learning scenarios. We design our system to be adaptable without extensive prompt engineering using an automatic prompt generation framework. Results show an improvement in scoring accuracy by 9% on unseen questions compared to fine-tuning, offering a scalable and cost-effective solution.https://ieeexplore.ieee.org/document/10771759/Automatic short answer scoringlarge language modelsretrieval-augmented generation |
| spellingShingle | Menna Fateen Bo Wang Tsunenori Mine Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With Feedback IEEE Access Automatic short answer scoring large language models retrieval-augmented generation |
| title | Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With Feedback |
| title_full | Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With Feedback |
| title_fullStr | Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With Feedback |
| title_full_unstemmed | Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With Feedback |
| title_short | Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring With Feedback |
| title_sort | beyond scores a modular rag based system for automatic short answer scoring with feedback |
| topic | Automatic short answer scoring large language models retrieval-augmented generation |
| url | https://ieeexplore.ieee.org/document/10771759/ |
| work_keys_str_mv | AT mennafateen beyondscoresamodularragbasedsystemforautomaticshortanswerscoringwithfeedback AT bowang beyondscoresamodularragbasedsystemforautomaticshortanswerscoringwithfeedback AT tsunenorimine beyondscoresamodularragbasedsystemforautomaticshortanswerscoringwithfeedback |