Retracting ChatGPT: completeness and relevance of academic references

Abstract This two-part study describes a learning exercise including a video and a worksheet designed to raise students’ awareness of the need to evaluate the completeness of references generated by ChatGPT. The first part of the study assesses the completeness and relevance of academic references g...

Full description

Saved in:
Bibliographic Details
Main Authors: Ana Lidia Franzoni Velázquez, Esperanza Huerta, Scott Jensen
Format: Article
Language:English
Published: Springer 2024-11-01
Series:Discover Education
Subjects:
Online Access:https://doi.org/10.1007/s44217-024-00333-1
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1846164970442588160
author Ana Lidia Franzoni Velázquez
Esperanza Huerta
Scott Jensen
author_facet Ana Lidia Franzoni Velázquez
Esperanza Huerta
Scott Jensen
author_sort Ana Lidia Franzoni Velázquez
collection DOAJ
description Abstract This two-part study describes a learning exercise including a video and a worksheet designed to raise students’ awareness of the need to evaluate the completeness of references generated by ChatGPT. The first part of the study assesses the completeness and relevance of academic references generated by ChatGPT using four prompts and three versions of ChatGPT. Content analysis using a priori coding revealed that most of the references generated are hallucinations or only partially complete. The results from the first part of the study were utilized in the second part of the study where 66 students evaluated the completeness of a sample of the references that had been generated and evaluated in the first part of the study. Although most students participating in the case study correctly answered questions about the content of the videos, only some students were able to correctly evaluate the completeness of the generated references.
format Article
id doaj-art-57d0c40ac85b436d959205cbe76a9ba9
institution Kabale University
issn 2731-5525
language English
publishDate 2024-11-01
publisher Springer
record_format Article
series Discover Education
spelling doaj-art-57d0c40ac85b436d959205cbe76a9ba92024-11-17T12:43:22ZengSpringerDiscover Education2731-55252024-11-013111110.1007/s44217-024-00333-1Retracting ChatGPT: completeness and relevance of academic referencesAna Lidia Franzoni Velázquez0Esperanza Huerta1Scott Jensen2Computer Science, Instituto Tecnológico Autónomo de MéxicoAccounting and Finance, San José State UniversityInformation Systems and Technology, San José State UniversityAbstract This two-part study describes a learning exercise including a video and a worksheet designed to raise students’ awareness of the need to evaluate the completeness of references generated by ChatGPT. The first part of the study assesses the completeness and relevance of academic references generated by ChatGPT using four prompts and three versions of ChatGPT. Content analysis using a priori coding revealed that most of the references generated are hallucinations or only partially complete. The results from the first part of the study were utilized in the second part of the study where 66 students evaluated the completeness of a sample of the references that had been generated and evaluated in the first part of the study. Although most students participating in the case study correctly answered questions about the content of the videos, only some students were able to correctly evaluate the completeness of the generated references.https://doi.org/10.1007/s44217-024-00333-1ChatGPTLarge language modelsReferencesCompletenessRelevance
spellingShingle Ana Lidia Franzoni Velázquez
Esperanza Huerta
Scott Jensen
Retracting ChatGPT: completeness and relevance of academic references
Discover Education
ChatGPT
Large language models
References
Completeness
Relevance
title Retracting ChatGPT: completeness and relevance of academic references
title_full Retracting ChatGPT: completeness and relevance of academic references
title_fullStr Retracting ChatGPT: completeness and relevance of academic references
title_full_unstemmed Retracting ChatGPT: completeness and relevance of academic references
title_short Retracting ChatGPT: completeness and relevance of academic references
title_sort retracting chatgpt completeness and relevance of academic references
topic ChatGPT
Large language models
References
Completeness
Relevance
url https://doi.org/10.1007/s44217-024-00333-1
work_keys_str_mv AT analidiafranzonivelazquez retractingchatgptcompletenessandrelevanceofacademicreferences
AT esperanzahuerta retractingchatgptcompletenessandrelevanceofacademicreferences
AT scottjensen retractingchatgptcompletenessandrelevanceofacademicreferences