Background Around the start of the pandemic of Sars-Cov-2 in March 2020, the rapid emergence of this new research field seemed to boost a frantic production of studies and related publications. In January 2021, the volume of research papers addressing any issue regarding Covid-19 amounts to more than 200.000 studies, leaving the healthcare professionals overwhelmed by an overload of information of mixed quality. To easily seek the needed pieces of information, tools based on text mining, machine learning, etc., were created and rapidly gained popularity. One of them is the Covid-19 Database by “Dimensions” including all COVID-related publications, updated daily. It is integrated with Altmetric, a platform that tracks down any online mention and share on the web and yields a score that is proportional to how intense the online activity is around any scientific work. To evaluate the solidity and replicability of the articles that have been disseminated the most during the first year of the pandemic, we aim to replicate a seminal paper by Ioannidis, showing that the most cited articles between 1990 and 2003 were contradicted with subsequent research. Aim This study aims to evaluate whether online top-trending articles investigating interventions against COVID-19 have been either contradicted by the following research or have claimed stronger results than what subsequent studies did replicate. The characteristics of these studies, including the Altmetric score, will be analysed to understand if any is associated with refutation. Methods Original clinical research studies will be included if they made their first appearance online between 31st December 2019 and - 1st January 2021 before being published in an indexed journal, had an Altmetric score equal to or greater than 99 percent of those considered by Dimensions’ extracted batch, were in English, and addressed the efficacy of pharmaceutical interventions against COVID-19 on humans. The studies, named online trending articles, will be examined, and compared with other concurrently or subsequently published research on the same question that have similar or larger sample size than that of the online trending original study, or used a theoretically better-controlled design (meta-analyses and RCT or meta-analyses for RCT and non-RCT respectively). Based on this comparison, the online trending articles will be coded as Unchallenged, when no subsequent clinical research of eligible design and the sample size was available to validate the claimed efficacy; Contradicted, when subsequent research showed the intervention to be ineffective or detrimental; Initially stronger effects, when the relative risk reduction for the main outcome in the subsequent research was half or less compared with what had been proposed by the original online trending study, or when the subsequent research showed that the estimated benefit was of short duration or its applicability and generalizability was limited; and Replicated effects, when the study finding was evaluated as replicable based on results of other studies, included with the previously defined criteria. A control group of articles will be assembled from the same batch extracted from “Dimensions” to evaluate if there is a significant difference between the online trending original articles, defined as before, and articles with a lower Altmetric in their characteristics, findings, and probability of being contradicted. For each control article included, a search will be performed to find subsequent research, as described for the online trending articles, followed by a similar article selection. The control group articles will be classified adopting the same coding system. Two researchers will conduct study classification independently; disagreements will be resolved by discussion among the research group. A descriptive analysis of online trending articles and control articles will be performed. The differences between the unchallenged or replicated online trending studies and the contradicted or initially stronger studies will be examined. The Mann-Whitney U test will be used to compare continuous variables and the Fisher exact test for binary variables. Considering the control articles, a comparison with the online trending articles will be performed through conditional logistic regression to account for matching.
Association between the Altmetric score and subsequent refutation among COVID-19 interventional studies published in 2020: a study protocol.
Erica De Vita;Guglielmo Arzilli;Virginia Casigliani;Filippo Quattrone;
2021-01-01
Abstract
Background Around the start of the pandemic of Sars-Cov-2 in March 2020, the rapid emergence of this new research field seemed to boost a frantic production of studies and related publications. In January 2021, the volume of research papers addressing any issue regarding Covid-19 amounts to more than 200.000 studies, leaving the healthcare professionals overwhelmed by an overload of information of mixed quality. To easily seek the needed pieces of information, tools based on text mining, machine learning, etc., were created and rapidly gained popularity. One of them is the Covid-19 Database by “Dimensions” including all COVID-related publications, updated daily. It is integrated with Altmetric, a platform that tracks down any online mention and share on the web and yields a score that is proportional to how intense the online activity is around any scientific work. To evaluate the solidity and replicability of the articles that have been disseminated the most during the first year of the pandemic, we aim to replicate a seminal paper by Ioannidis, showing that the most cited articles between 1990 and 2003 were contradicted with subsequent research. Aim This study aims to evaluate whether online top-trending articles investigating interventions against COVID-19 have been either contradicted by the following research or have claimed stronger results than what subsequent studies did replicate. The characteristics of these studies, including the Altmetric score, will be analysed to understand if any is associated with refutation. Methods Original clinical research studies will be included if they made their first appearance online between 31st December 2019 and - 1st January 2021 before being published in an indexed journal, had an Altmetric score equal to or greater than 99 percent of those considered by Dimensions’ extracted batch, were in English, and addressed the efficacy of pharmaceutical interventions against COVID-19 on humans. The studies, named online trending articles, will be examined, and compared with other concurrently or subsequently published research on the same question that have similar or larger sample size than that of the online trending original study, or used a theoretically better-controlled design (meta-analyses and RCT or meta-analyses for RCT and non-RCT respectively). Based on this comparison, the online trending articles will be coded as Unchallenged, when no subsequent clinical research of eligible design and the sample size was available to validate the claimed efficacy; Contradicted, when subsequent research showed the intervention to be ineffective or detrimental; Initially stronger effects, when the relative risk reduction for the main outcome in the subsequent research was half or less compared with what had been proposed by the original online trending study, or when the subsequent research showed that the estimated benefit was of short duration or its applicability and generalizability was limited; and Replicated effects, when the study finding was evaluated as replicable based on results of other studies, included with the previously defined criteria. A control group of articles will be assembled from the same batch extracted from “Dimensions” to evaluate if there is a significant difference between the online trending original articles, defined as before, and articles with a lower Altmetric in their characteristics, findings, and probability of being contradicted. For each control article included, a search will be performed to find subsequent research, as described for the online trending articles, followed by a similar article selection. The control group articles will be classified adopting the same coding system. Two researchers will conduct study classification independently; disagreements will be resolved by discussion among the research group. A descriptive analysis of online trending articles and control articles will be performed. The differences between the unchallenged or replicated online trending studies and the contradicted or initially stronger studies will be examined. The Mann-Whitney U test will be used to compare continuous variables and the Fisher exact test for binary variables. Considering the control articles, a comparison with the online trending articles will be performed through conditional logistic regression to account for matching.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
