Examining the responsible use of zero-shot AI approaches to scoring essays
Abstract The promise of AI to alleviate the burdens of grading and potentially enhance writing instruction is an exciting prospect. However, we believe it is crucial to emphasize that the accuracy of AI is only one component of its responsible use in education. Various governmental agencies, such as...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-12-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-024-79208-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846137039607562240 |
|---|---|
| author | Matthew Johnson Mo Zhang |
| author_facet | Matthew Johnson Mo Zhang |
| author_sort | Matthew Johnson |
| collection | DOAJ |
| description | Abstract The promise of AI to alleviate the burdens of grading and potentially enhance writing instruction is an exciting prospect. However, we believe it is crucial to emphasize that the accuracy of AI is only one component of its responsible use in education. Various governmental agencies, such as NIST in the US, and non-governmental agencies like the UN, UNESCO, and OECD have published guidance on the responsible use of AI, which we have synthesized to come up with our principles for the responsible use of AI in assessments at ETS. Our principles include fairness and bias mitigation; privacy & security; transparency, explainability, and accountability; educational impact & integrity; and continuous improvement. The accuracy of AI-scoring is one component of our principles related to educational impact & integrity. In this work, we share our thoughts on fairness & bias mitigation, and transparency & explainability. We demonstrate an empirical evaluation of zero-shot scoring using GTP-4o, with an emphasis on fairness evaluations and explainability of these automated scoring models. |
| format | Article |
| id | doaj-art-370b87a18bd7490a8c33a91fa1a8d24d |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-370b87a18bd7490a8c33a91fa1a8d24d2024-12-08T12:31:12ZengNature PortfolioScientific Reports2045-23222024-12-0114111010.1038/s41598-024-79208-2Examining the responsible use of zero-shot AI approaches to scoring essaysMatthew Johnson0Mo Zhang1Educational Testing ServiceEducational Testing ServiceAbstract The promise of AI to alleviate the burdens of grading and potentially enhance writing instruction is an exciting prospect. However, we believe it is crucial to emphasize that the accuracy of AI is only one component of its responsible use in education. Various governmental agencies, such as NIST in the US, and non-governmental agencies like the UN, UNESCO, and OECD have published guidance on the responsible use of AI, which we have synthesized to come up with our principles for the responsible use of AI in assessments at ETS. Our principles include fairness and bias mitigation; privacy & security; transparency, explainability, and accountability; educational impact & integrity; and continuous improvement. The accuracy of AI-scoring is one component of our principles related to educational impact & integrity. In this work, we share our thoughts on fairness & bias mitigation, and transparency & explainability. We demonstrate an empirical evaluation of zero-shot scoring using GTP-4o, with an emphasis on fairness evaluations and explainability of these automated scoring models.https://doi.org/10.1038/s41598-024-79208-2AIScoringFairnessExplainabilityEducational Measurement |
| spellingShingle | Matthew Johnson Mo Zhang Examining the responsible use of zero-shot AI approaches to scoring essays Scientific Reports AI Scoring Fairness Explainability Educational Measurement |
| title | Examining the responsible use of zero-shot AI approaches to scoring essays |
| title_full | Examining the responsible use of zero-shot AI approaches to scoring essays |
| title_fullStr | Examining the responsible use of zero-shot AI approaches to scoring essays |
| title_full_unstemmed | Examining the responsible use of zero-shot AI approaches to scoring essays |
| title_short | Examining the responsible use of zero-shot AI approaches to scoring essays |
| title_sort | examining the responsible use of zero shot ai approaches to scoring essays |
| topic | AI Scoring Fairness Explainability Educational Measurement |
| url | https://doi.org/10.1038/s41598-024-79208-2 |
| work_keys_str_mv | AT matthewjohnson examiningtheresponsibleuseofzeroshotaiapproachestoscoringessays AT mozhang examiningtheresponsibleuseofzeroshotaiapproachestoscoringessays |