Evaluation of excess significance bias in animal studies of neurological diseases.
Animal studies generate valuable hypotheses that lead to the conduct of preventive or therapeutic clinical trials. We assessed whether there is evidence for excess statistical significance in results of animal studies on neurological disorders, suggesting biases. We used data from meta-analyses of i...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2013-07-01
|
| Series: | PLoS Biology |
| Online Access: | https://doi.org/10.1371/journal.pbio.1001609 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849332407125147648 |
|---|---|
| author | Konstantinos K Tsilidis Orestis A Panagiotou Emily S Sena Eleni Aretouli Evangelos Evangelou David W Howells Rustam Al-Shahi Salman Malcolm R Macleod John P A Ioannidis |
| author_facet | Konstantinos K Tsilidis Orestis A Panagiotou Emily S Sena Eleni Aretouli Evangelos Evangelou David W Howells Rustam Al-Shahi Salman Malcolm R Macleod John P A Ioannidis |
| author_sort | Konstantinos K Tsilidis |
| collection | DOAJ |
| description | Animal studies generate valuable hypotheses that lead to the conduct of preventive or therapeutic clinical trials. We assessed whether there is evidence for excess statistical significance in results of animal studies on neurological disorders, suggesting biases. We used data from meta-analyses of interventions deposited in Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies (CAMARADES). The number of observed studies with statistically significant results (O) was compared with the expected number (E), based on the statistical power of each study under different assumptions for the plausible effect size. We assessed 4,445 datasets synthesized in 160 meta-analyses on Alzheimer disease (n = 2), experimental autoimmune encephalomyelitis (n = 34), focal ischemia (n = 16), intracerebral hemorrhage (n = 61), Parkinson disease (n = 45), and spinal cord injury (n = 2). 112 meta-analyses (70%) found nominally (p≤0.05) statistically significant summary fixed effects. Assuming the effect size in the most precise study to be a plausible effect, 919 out of 4,445 nominally significant results were expected versus 1,719 observed (p<10⁻⁹). Excess significance was present across all neurological disorders, in all subgroups defined by methodological characteristics, and also according to alternative plausible effects. Asymmetry tests also showed evidence of small-study effects in 74 (46%) meta-analyses. Significantly effective interventions with more than 500 animals, and no hints of bias were seen in eight (5%) meta-analyses. Overall, there are too many animal studies with statistically significant results in the literature of neurological disorders. This observation suggests strong biases, with selective analysis and outcome reporting biases being plausible explanations, and provides novel evidence on how these biases might influence the whole research domain of neurological animal literature. |
| format | Article |
| id | doaj-art-bca865a5a5cf43c1bf8387fb018d8c16 |
| institution | Kabale University |
| issn | 1544-9173 1545-7885 |
| language | English |
| publishDate | 2013-07-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Biology |
| spelling | doaj-art-bca865a5a5cf43c1bf8387fb018d8c162025-08-20T03:46:12ZengPublic Library of Science (PLoS)PLoS Biology1544-91731545-78852013-07-01117e100160910.1371/journal.pbio.1001609Evaluation of excess significance bias in animal studies of neurological diseases.Konstantinos K TsilidisOrestis A PanagiotouEmily S SenaEleni AretouliEvangelos EvangelouDavid W HowellsRustam Al-Shahi SalmanMalcolm R MacleodJohn P A IoannidisAnimal studies generate valuable hypotheses that lead to the conduct of preventive or therapeutic clinical trials. We assessed whether there is evidence for excess statistical significance in results of animal studies on neurological disorders, suggesting biases. We used data from meta-analyses of interventions deposited in Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies (CAMARADES). The number of observed studies with statistically significant results (O) was compared with the expected number (E), based on the statistical power of each study under different assumptions for the plausible effect size. We assessed 4,445 datasets synthesized in 160 meta-analyses on Alzheimer disease (n = 2), experimental autoimmune encephalomyelitis (n = 34), focal ischemia (n = 16), intracerebral hemorrhage (n = 61), Parkinson disease (n = 45), and spinal cord injury (n = 2). 112 meta-analyses (70%) found nominally (p≤0.05) statistically significant summary fixed effects. Assuming the effect size in the most precise study to be a plausible effect, 919 out of 4,445 nominally significant results were expected versus 1,719 observed (p<10⁻⁹). Excess significance was present across all neurological disorders, in all subgroups defined by methodological characteristics, and also according to alternative plausible effects. Asymmetry tests also showed evidence of small-study effects in 74 (46%) meta-analyses. Significantly effective interventions with more than 500 animals, and no hints of bias were seen in eight (5%) meta-analyses. Overall, there are too many animal studies with statistically significant results in the literature of neurological disorders. This observation suggests strong biases, with selective analysis and outcome reporting biases being plausible explanations, and provides novel evidence on how these biases might influence the whole research domain of neurological animal literature.https://doi.org/10.1371/journal.pbio.1001609 |
| spellingShingle | Konstantinos K Tsilidis Orestis A Panagiotou Emily S Sena Eleni Aretouli Evangelos Evangelou David W Howells Rustam Al-Shahi Salman Malcolm R Macleod John P A Ioannidis Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biology |
| title | Evaluation of excess significance bias in animal studies of neurological diseases. |
| title_full | Evaluation of excess significance bias in animal studies of neurological diseases. |
| title_fullStr | Evaluation of excess significance bias in animal studies of neurological diseases. |
| title_full_unstemmed | Evaluation of excess significance bias in animal studies of neurological diseases. |
| title_short | Evaluation of excess significance bias in animal studies of neurological diseases. |
| title_sort | evaluation of excess significance bias in animal studies of neurological diseases |
| url | https://doi.org/10.1371/journal.pbio.1001609 |
| work_keys_str_mv | AT konstantinosktsilidis evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT orestisapanagiotou evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT emilyssena evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT eleniaretouli evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT evangelosevangelou evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT davidwhowells evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT rustamalshahisalman evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT malcolmrmacleod evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases AT johnpaioannidis evaluationofexcesssignificancebiasinanimalstudiesofneurologicaldiseases |