Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects.
Recent large-scale replication projects (RPs) have estimated concerningly low reproducibility rates. Further, they reported substantial degrees of shrinkage of effect size, where the replication effect size was found to be, on average, much smaller than the original effect size. Within these RPs, th...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2025-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0327799 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849335669705408512 |
|---|---|
| author | Rachel Heyard Leonhard Held |
| author_facet | Rachel Heyard Leonhard Held |
| author_sort | Rachel Heyard |
| collection | DOAJ |
| description | Recent large-scale replication projects (RPs) have estimated concerningly low reproducibility rates. Further, they reported substantial degrees of shrinkage of effect size, where the replication effect size was found to be, on average, much smaller than the original effect size. Within these RPs, the included original-replication study-pairs can vary with respect to aspects of study design, outcome measures, and descriptive features of both original and replication study population and study team. This often results in between-study-pair heterogeneity, i.e., variation in effect size differences across study-pairs that goes beyond expected statistical variation. When broader claims about the reproducibility of an entire field are based on such heterogeneous data, it becomes imperative to conduct a rigorous analysis of the amount and sources of shrinkage and heterogeneity within and between included study-pairs. Methodology from the meta-analysis literature provides an approach for quantifying the heterogeneity present in RPs with an additive or multiplicative parameter. Meta-regression methodology further allows for an investigation into the sources of shrinkage and heterogeneity. We propose the use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and heterogeneity (represented by the scale). This provides valuable insights into drivers and factors associated with high or low reproducibility rates and therefore contextualises results of RPs. The proposed methodology is illustrated using publicly available data from the Replication Project Psychology and the Replication Project Experimental Economics. All analysis scripts and data are available online. |
| format | Article |
| id | doaj-art-51a239c85e814e97a919ba4c19b4d36c |
| institution | Kabale University |
| issn | 1932-6203 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-51a239c85e814e97a919ba4c19b4d36c2025-08-20T03:45:11ZengPublic Library of Science (PLoS)PLoS ONE1932-62032025-01-01208e032779910.1371/journal.pone.0327799Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects.Rachel HeyardLeonhard HeldRecent large-scale replication projects (RPs) have estimated concerningly low reproducibility rates. Further, they reported substantial degrees of shrinkage of effect size, where the replication effect size was found to be, on average, much smaller than the original effect size. Within these RPs, the included original-replication study-pairs can vary with respect to aspects of study design, outcome measures, and descriptive features of both original and replication study population and study team. This often results in between-study-pair heterogeneity, i.e., variation in effect size differences across study-pairs that goes beyond expected statistical variation. When broader claims about the reproducibility of an entire field are based on such heterogeneous data, it becomes imperative to conduct a rigorous analysis of the amount and sources of shrinkage and heterogeneity within and between included study-pairs. Methodology from the meta-analysis literature provides an approach for quantifying the heterogeneity present in RPs with an additive or multiplicative parameter. Meta-regression methodology further allows for an investigation into the sources of shrinkage and heterogeneity. We propose the use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and heterogeneity (represented by the scale). This provides valuable insights into drivers and factors associated with high or low reproducibility rates and therefore contextualises results of RPs. The proposed methodology is illustrated using publicly available data from the Replication Project Psychology and the Replication Project Experimental Economics. All analysis scripts and data are available online.https://doi.org/10.1371/journal.pone.0327799 |
| spellingShingle | Rachel Heyard Leonhard Held Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. PLoS ONE |
| title | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. |
| title_full | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. |
| title_fullStr | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. |
| title_full_unstemmed | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. |
| title_short | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. |
| title_sort | meta regression to explain shrinkage and heterogeneity in large scale replication projects |
| url | https://doi.org/10.1371/journal.pone.0327799 |
| work_keys_str_mv | AT rachelheyard metaregressiontoexplainshrinkageandheterogeneityinlargescalereplicationprojects AT leonhardheld metaregressiontoexplainshrinkageandheterogeneityinlargescalereplicationprojects |