Most Covid-19 studies posted to a government-run database aren’t generating meaningful evidence necessary to make treatment decisions, according to a study from Stanford and Yale researchers.
Only 29% of more than 1,500 studies listed on ClinicalTrials.gov met a threshold of high quality, based on an Oxford University Centre for Evidence-Based Medicine scale, according to the study published Monday in JAMA Internal medicine.
The report highlights an unintended consequence of so many researchers pivoting to the virus. The high volume of information being generated can make it harder to figure out which studies are the best ones to follow. That is especially true since the need to share discoveries rapidly pushed more researchers to post their papers before they have been peer reviewed.
Fewer than half of the studies were randomized controlled trials, which are considered the gold standard. Just 75 randomized controlled trials included a placebo, more than 100 participants, and more than one research institution. Nearly 1,200 of the studies took place at just one site, which previous research shows can exaggerate the effectiveness of a treatment.
Bad Data Drives Bad Policy
“We have to really be careful with rapid dissemination of small studies that are going to project low quality evidence,” Mintu Turakhia, a Stanford University researcher and corresponding author of the JAMA research letter, said in an interview. “We’ve already seen how observational studies—sometimes not even the best studies—can influence public opinion and policy.”
For example, White House officials pushed the antimalarial drug hydroxychloroquine based largely on anecdotal evidence, but clinical trials later showed it didn’t work against SARS-Cov-2. The Food and Drug Administration ultimately pulled its emergency authorization for the drug.
“It is a cautionary tale. And the more studies you have that are very low quality, the more noise you produce in the public domain,” Turakhia said.
That makes it harder for physicians to stay abreast of the latest and best findings, said Alison Bateman-House, a New York University bioethicist who specializes in access to investigational medicines.
The best approach would be a series of rigorous clinical trials rather than individual experiments with varying levels of reliability, Bateman-House said.
Low quality studies can get in the way of high quality studies, and they prevent scientists from doing the right study, Vinay K, Prasad, a hematologist-oncologist who studies clinical trials and better decision making, said. That’s led doctors to prescribe treatments with dubious value based on little information.
If somebody came into the hospital with a pneumonia of unknown origin a year ago, hospitals would provide the medical care and hope for the best.
“We wouldn’t be reaching in the back pocket and just throwing [anti-inflammatory] tocilizumab and hydroxychloroquine in and a full dose blood thinner,” Prasad said. “But something is different about the zeitgeist, as the mentality has shifted.”
With few exceptions, all NIH-funded and FDA-regulated clinical trials must be registered on ClinicalTrials.gov. Turakhia said their findings intend to provide a barometer for what kind of expectations to have for the first wave of Covid-19 research. Some of those studies may capture the early experience when little information existed, “but we’re past that now,” he said.
Going forward, he said it’s important to focus on clinical trials that can answer questions with a high level of certainty, such as the study showing dexamethasone can improve patient survival.
“There aren’t that many studies like that so far that are registered. And maybe that’s OK,” he said. “But we really should not sacrifice well-designed clinical trials just because we’re facing a crisis.”