The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writing
Generative AI (GenAI) use in research writing is growing fast. However, it is unclear how peer reviewers recognize or misjudge AI-augmented manuscripts. To investigate the impact of AI-augmented writing on peer reviews, we conducted a snippet-based online survey with 17 peer reviewers from top-tier...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-08-01
|
| Series: | Computers in Human Behavior: Artificial Humans |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2949882124000550 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846141651331842048 |
|---|---|
| author | Hilda Hadan Derrick M. Wang Reza Hadi Mogavi Joseph Tu Leah Zhang-Kennedy Lennart E. Nacke |
| author_facet | Hilda Hadan Derrick M. Wang Reza Hadi Mogavi Joseph Tu Leah Zhang-Kennedy Lennart E. Nacke |
| author_sort | Hilda Hadan |
| collection | DOAJ |
| description | Generative AI (GenAI) use in research writing is growing fast. However, it is unclear how peer reviewers recognize or misjudge AI-augmented manuscripts. To investigate the impact of AI-augmented writing on peer reviews, we conducted a snippet-based online survey with 17 peer reviewers from top-tier HCI conferences. Our findings indicate that while AI-augmented writing improves readability, language diversity, and informativeness, it often lacks research details and reflective insights from authors. Reviewers consistently struggled to distinguish between human and AI-augmented writing but their judgements remained consistent. They noted the loss of a “human touch” and subjective expressions in AI-augmented writing. Based on our findings, we advocate for reviewer guidelines that promote impartial evaluations of submissions, regardless of any personal biases towards GenAI. The quality of the research itself should remain a priority in reviews, regardless of any preconceived notions about the tools used to create it. We emphasize that researchers must maintain their authorship and control over the writing process, even when using GenAI's assistance. |
| format | Article |
| id | doaj-art-1568ac3b809c4bd08d973fc8a2225dca |
| institution | Kabale University |
| issn | 2949-8821 |
| language | English |
| publishDate | 2024-08-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers in Human Behavior: Artificial Humans |
| spelling | doaj-art-1568ac3b809c4bd08d973fc8a2225dca2024-12-04T05:15:05ZengElsevierComputers in Human Behavior: Artificial Humans2949-88212024-08-0122100095The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writingHilda Hadan0Derrick M. Wang1Reza Hadi Mogavi2Joseph Tu3Leah Zhang-Kennedy4Lennart E. Nacke5Stratford School of Interaction Design and Business, University of Waterloo, Waterloo, Ontario, N2L 3G1, CanadaStratford School of Interaction Design and Business, University of Waterloo, Waterloo, Ontario, N2L 3G1, CanadaStratford School of Interaction Design and Business, University of Waterloo, Waterloo, Ontario, N2L 3G1, CanadaStratford School of Interaction Design and Business, University of Waterloo, Waterloo, Ontario, N2L 3G1, CanadaStratford School of Interaction Design and Business, University of Waterloo, Waterloo, Ontario, N2L 3G1, CanadaCorresponding author. Stratford School of Interaction Design and Business, University of Waterloo, 125 St Patrick St, Stratford, ON, N5A 0C1, Canada.; Stratford School of Interaction Design and Business, University of Waterloo, Waterloo, Ontario, N2L 3G1, CanadaGenerative AI (GenAI) use in research writing is growing fast. However, it is unclear how peer reviewers recognize or misjudge AI-augmented manuscripts. To investigate the impact of AI-augmented writing on peer reviews, we conducted a snippet-based online survey with 17 peer reviewers from top-tier HCI conferences. Our findings indicate that while AI-augmented writing improves readability, language diversity, and informativeness, it often lacks research details and reflective insights from authors. Reviewers consistently struggled to distinguish between human and AI-augmented writing but their judgements remained consistent. They noted the loss of a “human touch” and subjective expressions in AI-augmented writing. Based on our findings, we advocate for reviewer guidelines that promote impartial evaluations of submissions, regardless of any personal biases towards GenAI. The quality of the research itself should remain a priority in reviews, regardless of any preconceived notions about the tools used to create it. We emphasize that researchers must maintain their authorship and control over the writing process, even when using GenAI's assistance.http://www.sciencedirect.com/science/article/pii/S2949882124000550Artificial intelligenceGenerative AIReviewer perceptionResearch writingAI writing augmentation |
| spellingShingle | Hilda Hadan Derrick M. Wang Reza Hadi Mogavi Joseph Tu Leah Zhang-Kennedy Lennart E. Nacke The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writing Computers in Human Behavior: Artificial Humans Artificial intelligence Generative AI Reviewer perception Research writing AI writing augmentation |
| title | The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writing |
| title_full | The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writing |
| title_fullStr | The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writing |
| title_full_unstemmed | The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writing |
| title_short | The great AI witch hunt: Reviewers’ perception and (Mis)conception of generative AI in research writing |
| title_sort | great ai witch hunt reviewers perception and mis conception of generative ai in research writing |
| topic | Artificial intelligence Generative AI Reviewer perception Research writing AI writing augmentation |
| url | http://www.sciencedirect.com/science/article/pii/S2949882124000550 |
| work_keys_str_mv | AT hildahadan thegreataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT derrickmwang thegreataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT rezahadimogavi thegreataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT josephtu thegreataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT leahzhangkennedy thegreataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT lennartenacke thegreataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT hildahadan greataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT derrickmwang greataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT rezahadimogavi greataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT josephtu greataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT leahzhangkennedy greataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting AT lennartenacke greataiwitchhuntreviewersperceptionandmisconceptionofgenerativeaiinresearchwriting |