FPI / September 22, 2020
Analysis by R. Clinton Ohlers
During the past several weeks, major studies have claimed the drug hydroxychloroquine had no effect on COVID-19 patients. Some even claimed it was harmful.
However, major studies out of New York, Spain, Switzerland, Michigan, Belgium and elsewhere have provided compelling evidence that the drug, particularly in combination with azithromycin and zinc, has been saving lives all along.
Perhaps most surprising among these is a pivotal study that appeared in the Journal of the American Medical Association (JAMA) online on May 11. It reported finding no evidence of the drug’s having any positive effect on patients who were treated in New York during the early weeks of the outbreak.
In reality, however, the study’s data actually confirms the positive findings of these recent, thoroughgoing international studies that are reporting on hydroxychloroquine’s dramatic effectiveness. It even affirms their precise statistics.
When the JAMA study’s data is re-analyzed in a manner congruent with recommended treatment, that data demonstrates that hydroxychloroquine significantly and dramatically improved patient survival during the early and chaotic weeks of the pandemic in New York. Survival rates for hospitalized patients who received the drug approached 85%. When hydroxychloroquine was combined with azithromycin the survival rate rose as high as 90%. However, when hospitalized patients received neither drug, their survival fell to levels as low as 53%.
The corresponding author of the JAMA study has not yet responded to requests for comment.
The author, Eli S. Rosenberg, PhD, is at the Department of Epidemiology and Biostatistics, University at Albany School of Public Health in Albany, New York.
Rosenberg’s webpage at the University of Albany states: “Since March 2020, Rosenberg has been providing technical assistance to the New York State Department of Health COVID-19 response across a broad portfolio of studies aimed at understanding critical aspects of SARS-CoV-2 transmission, surveillance, and treatment.”
When historians revisit the early days of the pandemic in the United States, they will record the importance of the May 11 JAMA study as the tipping-point study that effectively terminated interest in hydroxychloroquine as a potential therapy in the larger medical research community. It was this study that President Donald Trump looked to in mid-March to confirm hopes for hydroxychloroquine as a sound therapy against COVID-19.
Instead, its authors asserted precisely the opposite.
They reported no significant difference in survival rates among patients who received hydroxychloroquine, whether alone or in combination with azithromycin, versus those who did not receive the drugs.
The impact was immediate.
No less a figure than Vanderbilt University Medical School’s Dr. William Schaffner, an advisor to the CDC and a leader in the field of infectious diseases, announced on CNN, “the nail has virtually been put in the coffin of hydroxychloroquine.” Schaffner previously had been hopeful about the drug. A general consensus against hydroxychloroquine followed both in the United States and internationally.
This consensus was further solidified by negative assessments in the Lancet (later retracted), by the World Health Organization, and Oxford’s Center for Evidence Based Medicine.
Just how the same data in a single study may produce such diametrically opposed evidence is the story that follows.
Any study on hydroxychloroquine coming out of New York in the early weeks of the pandemic would have faced a daunting challenge. The reason has to do with how patients in the state were treated under the guidelines laid out by Dr. Anthony Fauci, the FDA, and Governor Andrew Cuomo’s executive order of March 23. Hydroxychloroquine was to be administered only to those already hospitalized as part of a clinical trial or as “compassionate care” for the most severely ill. Therefore, patients treated with hydroxychloroquine were generally well advanced in the progress of the disease or were among the most severely ill, the most aged, and possessing the most comorbidities, such as diabetes, obesity, or heart disease.
Not surprisingly, death rates among patients who received the drug were actually much higher than for those who did not. Among the hydroxychloroquine + azithromycin group, the death rate was 25.7%. For those receiving only hydroxychloroquine, the death rate was 19.9%. For those given neither drug, it was 12.7%. Those, however, are unadjusted death rates.
How a study of this kind handles such disparities has to do with very powerful and effective statistical tools. Cox adjusted modeling, for example, helped correct for differences among patient groups such as age, sex, and greater numbers of comorbidities. Calculating standard deviations enabled researchers to determine how much difference between groups simply might be the result of chance or factors other than whether or not the drug worked.
In medicine, the threshold at which the effectiveness of a drug is considered significant, all else being equal, begins at two standard deviations. If the differences were to rise as high as three standard deviations, the efficacy of the drug would be very strongly confirmed.
An analogy here is a coin toss. If two players flip a coin, how does the losing player know something else has not affected the outcome, such as the coin being weighted? The losing player might demand that the coin be tossed 300 times and note the numbers of heads versus the number of tails. Ideally, the ratio would be 50/50. However, in only 300 tosses, that ideal is unlikely to be achieved exactly. One might easily get more than 150 heads. Nonetheless, the results can still render a conclusion. By calculating the percentage of heads to tails against the number of tosses one can test the outcome against the normal plausible range of results. In medicine, that range is two standard deviations or less.
Presumably, the Cox adjusted modeling and standard deviations accounted sufficiently for the significant differences in base-line health and severity of illness, bringing the authors to the conclusion that hydroxychloroquine had no discernible effect.
An initial concern arises when one examines just how much more severely ill or compromised the hydroxychloroquine patients were, compared with the control group. Among those receiving the drug in combination with azithromycin, for example, 30.7% entered the intensive care unit (ICU) at some point in their stay. Less than half, only 12.2%, of patients given no drugs did so.
This suggests a rate of greater severity of illness of 2.5x. By contrast the rate of death of the paired-drug group was only 2x that of the control. In other words, a group that had 2.5x the rate of severely ill patients had only 2x the rate of death, but was somehow brought to parity using a statistical tool designed to account for that group’s greater (not lesser) incidence of comorbidities. On the surface, at least, this would appear to imply the drug had imparted some benefit.
The first major red flag appears when the authors discuss the study’s limitations. They explain that “the rapidity with which patients entered the ICU and underwent mechanical ventilation, often concurrently with initiating hydroxychloroquine and azithromycin, rendered these outcomes unsuitable for efficacy analyses.”
Nevertheless, these “unsuitable” patients were included in the study as it attempted to analyze efficacy. No attempt to understand how their presence might adversely influence the study’s conclusions was presented.
This disclosure is profound for several reasons. The first of these has to do with how anti-viral drugs function.
By mid-March, proponents of hydroxychloroquine universally agreed that for the drug to be effective it had to be administered very soon after the first onset of symptoms. This was true of the doctors in Wuhan and Shanghai, such as Dr. George F. Gao, Director General of the Chinese Center for Disease Control and Prevention and his colleagues who pioneered the treatment, those in Costa Rica such as Dr. Mario Ruiz, Director of the Caja Costarricense de Seguro Social, and his colleagues, who adopted their advice, Dr. Didier Raoult, Director of the Unité de Recherche sur les Maladies Infectieuses et Tropicales Emergentes at Aix-Marseille University, and his colleagues in France, and Dr. Vladimir Zelenko in New York, who later teamed with doctors in Germany, as well as many other physicians in the United States.
In his White House press briefing on March 19, FDA Chairman Dr. Stephen Hahn could hardly have stated more clearly what was needed to discover a real treatment for COVID-19:
“We need to make sure that the sea of new treatments will get the right drug to the right patient at the right dosage at the right time.”
With antiviral therapies early treatment is important. The best known example of this is the over-the-counter drug Abreva. Cold sore sufferers are instructed to apply Abreva as soon as they sense a tingling, before a cold sore actually appears and the virus has achieved its florid state. A further reason for early treatment in COVID-19 is to win the race against the cytokine storm that emerges in response to the virus, when the immune system actually attacks the patient’s own cells. This is why proponents of the drug universally recommended its use early, before hospitalization, rather than late when hospitalization is required.
Dr. Steven Hatfill, a veteran epidemiologist who teaches mass casualty medicine at the George Washington University Medical Center, identifies this as an ideal “seven-day window” after the onset of first symptoms. This window had been well identified by April.
In late May, after this study appeared, Yale epidemiologist Dr. Harvey Risch wrote with deep concern in the American Journal of Epidemiology, emphasizing the importance of early treatment, which major studies like this one had overlooked:
“Early outpatient [COVID-19] illness is very different than later hospitalized florid disease and the treatments differ,” he warned.
By analogy, to study the effectiveness of an antiviral medication like hydroxychloroquine by assessing patients treated so late that they were already in the ICU would be like testing the effectiveness of air bags by looking at cases where the air bag had only deployed after the driver made impact with the steering wheel or was thrown through the windshield.
At the same time, one of the most interesting elements of the studies out of Europe has been the effectiveness of hydroxychloroquine among patients who were already hospitalized and which also appears when the data of the JAMA study is reanalyzed. Apparently, what appears to be occurring with hydroxychloroquine, particularly when it is given with azithromycin (and even more so with zinc) is that the ideal time to initiate treatment is that seven-day window. After that point benefits still occur, but appear increasingly to decline the later in the progress of the disease that the drug therapy is begun. Where the cut-off area occurs, that zone within the progression of the disease that the window of opportunity closes, is an open question for science to address in the future.
For our purposes of analysis here, it is enough simply to assume the same cut-off that the authors of the JAMA study noted as too late to be suitable for efficacy: when a patient has already entered the ICU before or at the same time as starting treatment.
The second reason the red flag is so significant has to do with the sizable number of the unsuitable cases.
Patients who received hydroxychloroquine +azithromycin represent the largest group in the study. Of them, 17.1% were transferred to ICU in less than one day. That number is equal to 66.5% of the paired-drug group’s death rate.
This is particularly troubling because other data in the report indicates that mortality among those rapidly transferred to ICU was also extremely high. Possibly approaching 100%, for example, in the no-drug control group. In such a scenario, these unsuitable cases contributed as much as 66.5% of the deaths in that group.
A third reason the red flag is so profound is that it is reasonable to believe this large number of unsuitable patients were placed in the wrong test group.
It is reasonable to view such late treatment as equal to receiving no drug at all. That is, in essence, what doctors from Wuhan and Shanghai reported when in March they shared their frontline treatment recommendations with doctors in Costa Rica. If such late treatment is equivalent to no treatment at all, then these patients may have belonged in the no-drug control group, not in the hydroxychloroquine test groups.
To use the airbag analogy, the purpose of the study was to determine whether the airbag (hydroxychloroquine) had the capability of cushioning passengers from fatal impact. The unsuitable cases are equivalent to passengers who received no airbag protection because the airbag was not deployed, not because airbags that did deploy had no effect.
To include in the analysis passengers for whom the airbag was deployed only after the crash occurred does much more than merely muddy the results. It actually lessens the appearance of the risk of driving without an air bag. It does so at the very same time that it attributes an increased number of lethal crashes to airbags.
Alternatively, researchers might remove these unsuitable cases from consideration altogether. However, that would also distort the data. That too would make it appear that crashes without airbags resulted in death less often than actually occurred.
Given that this scenario comprises such a large portion of the JAMA study, the inclusion of rapid ICU transfers in the hydroxychloroquine group drastically inclined the study’s conclusion toward a negative assessment.
It is fair to ask, then, what happens to the data when these patients are instead included where arguably they belong, in the no-drug control group. This is not the only way to view this data, rather it is a highly relevant and necessary one.
Doing so could also improve the integrity of the data. The authors report that the study’s final sampling included fewer patients in the no-drug group than the authors originally intended for their proposed ratio. Such a transfer moves the treated to non-treated ratio closer to the original proposal. Doing so may also distribute more evenly patients with severe illness and comorbidities between the two groups. One risk, of course, is that if the drugs were beneficial, death rates among the no-drug group would be artificially lowered. However, if hydroxychloroquine truly provides no benefit, as the authors report, then no such concern exists.
What happens when this transfer is made? The unadjusted death rates favor the effectiveness of the paired drugs to a stunning degree.
The realistic death rate for patients receiving hydroxychloroquine + azithromycin drops as low as 10.3%. By contrast, the death rate among the control group rises as high as 44.4%. That is a difference of up to 77% lower mortality when hydroxychloroquine and azithromycin were taken together. This increase is significant: the difference between the outcomes with and without treatment stands at 12 standard deviations.
A study from Spain published on July 29 examined 9,644 patients who received only hydroxychloroquine showed similar results. That study found “no serious side effects,” and when adjusted odds ratios were applied, it “shows a 66 percent reduction in COVID mortality in patients taking hydroxychloroquine.”
Therefore, one might expect that application of adjusted odds ratios, to account for comorbidities in the JAMA study, could demonstrate greater results for hydroxychloroquine + azithromycin, as occurred in the Spanish study.
Without access to the study’s raw data, the actual death rates among patients who rapidly transferred to the ICU can only be approximated. However, even when tested against significantly lower rates of death than those suggested by the published data, the results remain significant. When the actual death rates that future, detailed re-examinations of the study’s data may render, the evidence of significant effect will hold and will continue to affirm the Spanish study.
As a note to future researchers who may pursue this question, it is important to point out that the May 11 JAMA study tracked patient outcomes more comprehensively than did an earlier JAMA study published on April 24. That earlier study made a projection concerning death rates among ventilated patients. However, the May 11 study demonstrates that those projections did not bear out.
What about the results among patients in the JAMA study who received only hydroxychloroquine?
The unsuitable cases here comprised 8.1% of the group. When transferred, the hydroxychloroquine death rate falls as low as 12.9% compared with 20.6% for the no-drug group. These results also meet the threshold of significance at 2.85 standard deviations. These numbers are similar to the unadjusted death rates of 14.7% and 29.2% in the Spanish study. They are also of larger importance.
On August 24, a group of Belgian researchers published a nationwide study in the International Journal of Antimicrobial Agents. It included 8075 patients, compared with 1482 in the JAMA study, and was also a randomized retrospective observational study. The Belgian team examined hydroxychloroquine alone versus treatment without the drug. Importantly, they studied patients across the vast majority of Belgian hospitals in order to compare like groups of patients with similar levels of comorbidities and other risk factors.
The Belgian team reported a rate of death among patients who received hydroxychloroquine in the hospital as 17.7% versus 27.1% for those who did not. When compared, the results of the Belgian study are virtually identical to those found in this reanalysis.
These numbers also support the findings of a Henry Ford Health System study of 2541 patients that was released on July 2. That study found a death rate of only 13% for patients treated with hydroxychloroquine alone, compared with 26.4% among patients not treated with the drug. The Henry Ford rates also support the Belgian study and are virtually identical to those of the Spanish study.
The most dramatic effect is found when all patients in the JAMA study receiving hydroxychloroquine, whether with or without azithromycin, are considered once the unsuitable cases are transferred. These results demonstrate a rate of death as low as 11% versus 47.7% in the no-drug group. This difference was confirmed well above three standard deviations.
These results far exceed those of the Belgian study. With mortality approaching less than one-quarter the mortality of the no-drug group, the JAMA study’s data further demonstrates the enormous advantage of adding azithromycin to hydroxychloroquine—something proponents of the drug asserted in March.
One more finding remains.
Here benefit of hydroxychloroquine + azithromycin is revealed by examining survival rates among patients transferred to the ICU and also their length of stay in the hospital. Although these data do not contain enough detail to allow analysis by transferring out the unsuitable cases, they are revealing nonetheless. In other words, in spite of the distortion created by including those severely ill patients “unsuitable for efficacy analyses” there was still evidence hydroxychloroquine + azithromycin had a significant effect.
The signal still broke through the noise if one knows where to look.
In a hospital setting, most patients who die of COVID-19 do so only after first transferring to the ICU. A small number may expire too quickly, unexpectedly, or en route to ICU. Therefore, comparing death rates to ICU transfer rates can be telling, as can the length of the hospital stay of patients who nonetheless died.
Only patients treated with hydroxychloroquine and azithromycin showed any improved rate of survival when compared with the number who transferred to ICU. Among patients receiving no drugs, for example, the number who required transfer to the ICU was 12.2%. The death rate for the no-drug group was 12.7%. This indicates that a small number of patients died outside of ICU and that scant few survived after entering ICU. By contrast, among hydroxychloroquine + azithromycin patients 30.7% entered ICU and only 25.7% of the total group died. That is a 16.4% greater rate of survival compared against ICU transfer, in spite of the inclusion of the most severely ill in the calculations.
When tested for statistical significance, that number shows 6 standard deviations. Even if using a hypothetical death rate of only 90% among ICU patients treated without the drug combination, significance is still demonstrated at 2.3 standard deviations. Not surprisingly then, the study also shows that among hydroxychloroquine + azithromycin patients who did die, their length of stay in hospital was longer, congruent with lengthening of survival. One recalls, for example, the cause of interest in Remdesivir, even when that drug failed to produce any improvement in survival—Remdesivir shortened the hospital stays of patients.
These findings should be a source of hope.
It now appears that the disregard of hydroxychloroquine that followed the JAMA publication was gravely premature. Although the authors’ own conclusions about the drug were negative, their data actually contains potent indicators of the drug’s effectiveness, even when administered under far less than ideal circumstances.
Even more importantly, when that data is re-examined in scenarios that better approximate how proponents have asserted the drug should be administered, the numbers fall directly in line with the major studies performed in locales as diverse as Michigan, New York, France, Spain, and most recently Belgium.
These findings provide further important evidence that a highly effective, inexpensive, and widely available treatment for COVID-19 is already in hand. It should provide needed impetus to get these valuable drugs to those currently ill with the disease as rapidly as possible.
[List of JAMA authors: Eli S. Rosenberg, PhD; Elizabeth M. Dufort, MD; Tomoko Udo, PhD; Larissa A. Wilberschied, MS; Jessica Kumar, DO; James Tesoriero, PhD; PattiWeinberg, PA; James Kirkwood, MPH; Alison Muse, MPH; Jack DeHovitz, MD; Debra S. Blog, MD; Brad Hutton, MPH; David R. Holtgrave, PhD; Howard A. Zucker, MD.]
[Special thanks to Dr. Mike Brownnutt, Associate Director of the Faith and Science Collaborative Research Forum at the University of Hong Kong (PhD, Physics, Imperial College London), for his valuable help with statistical calculations and critique. Special thanks also to Dr. Chis Martenson (PhD, Pathology, Duke) whose scientific reporting on COVID-19 since the beginning of the outbreak motivated this reanalysis. Thanks also to two MDs who provided valuable consultation and insight.]
R. Clinton Ohlers, PhD is a historian of science and religion and a contributing editor for the FreePressMediaGroup. Previously, he held the position of Research Assistant Professor in the Humanities at the University of Hong Kong. His book, The Birth of the Conflict Between Science and Religion, is scheduled to appear in 2021. He received his PhD in history from the University of Pennsylvania.
FPI, Free Press International