A randomized controlled trial on anonymizing reviewers to each other in peer review discussions.
Many peer-review processes involve reviewers submitting their independent reviews, followed by a discussion between the reviewers of each paper. A common question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this que...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2024-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0315674 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841555535196848128 |
|---|---|
| author | Charvi Rastogi Xiangchen Song Zhijing Jin Ivan Stelmakh Hal Daumé Kun Zhang Nihar B Shah |
| author_facet | Charvi Rastogi Xiangchen Song Zhijing Jin Ivan Stelmakh Hal Daumé Kun Zhang Nihar B Shah |
| author_sort | Charvi Rastogi |
| collection | DOAJ |
| description | Many peer-review processes involve reviewers submitting their independent reviews, followed by a discussion between the reviewers of each paper. A common question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this question by conducting a randomized controlled trial at the Conference on Uncertainty in Artificial Intelligence (UAI) 2022 conference where reviewer discussions were conducted over a typed forum. We randomly split the reviewers and papers into two conditions-one with anonymous discussions and the other with non-anonymous discussions. We also conduct an anonymous survey of all reviewers to understand their experience and opinions. We compare the two conditions in terms of the amount of discussion, influence of seniority on the final decisions, politeness, reviewers' self-reported experiences and preferences. Overall, this experiment finds small, significant differences favoring the anonymous discussion setup based on the evaluation criteria considered in this work. |
| format | Article |
| id | doaj-art-d186643eb94847729e274c98be631e55 |
| institution | Kabale University |
| issn | 1932-6203 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-d186643eb94847729e274c98be631e552025-01-08T05:32:24ZengPublic Library of Science (PLoS)PLoS ONE1932-62032024-01-011912e031567410.1371/journal.pone.0315674A randomized controlled trial on anonymizing reviewers to each other in peer review discussions.Charvi RastogiXiangchen SongZhijing JinIvan StelmakhHal DauméKun ZhangNihar B ShahMany peer-review processes involve reviewers submitting their independent reviews, followed by a discussion between the reviewers of each paper. A common question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this question by conducting a randomized controlled trial at the Conference on Uncertainty in Artificial Intelligence (UAI) 2022 conference where reviewer discussions were conducted over a typed forum. We randomly split the reviewers and papers into two conditions-one with anonymous discussions and the other with non-anonymous discussions. We also conduct an anonymous survey of all reviewers to understand their experience and opinions. We compare the two conditions in terms of the amount of discussion, influence of seniority on the final decisions, politeness, reviewers' self-reported experiences and preferences. Overall, this experiment finds small, significant differences favoring the anonymous discussion setup based on the evaluation criteria considered in this work.https://doi.org/10.1371/journal.pone.0315674 |
| spellingShingle | Charvi Rastogi Xiangchen Song Zhijing Jin Ivan Stelmakh Hal Daumé Kun Zhang Nihar B Shah A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. PLoS ONE |
| title | A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. |
| title_full | A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. |
| title_fullStr | A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. |
| title_full_unstemmed | A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. |
| title_short | A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. |
| title_sort | randomized controlled trial on anonymizing reviewers to each other in peer review discussions |
| url | https://doi.org/10.1371/journal.pone.0315674 |
| work_keys_str_mv | AT charvirastogi arandomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT xiangchensong arandomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT zhijingjin arandomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT ivanstelmakh arandomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT haldaume arandomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT kunzhang arandomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT niharbshah arandomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT charvirastogi randomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT xiangchensong randomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT zhijingjin randomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT ivanstelmakh randomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT haldaume randomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT kunzhang randomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions AT niharbshah randomizedcontrolledtrialonanonymizingreviewerstoeachotherinpeerreviewdiscussions |