Book Review: Ending Medical Reversal

Ending Medical Reversal by Vinay Prasad (MD MPH) and Adam Cifu (MD) is an essential book for medical students, physicians, and anyone even peripherally involved in medicine; for everybody else, it's merely highly recommended. The core of the book, paraphrasing, is that sometimes physicians decide to do things to patients without great evidence. These practices are, later on, 'reversed' by well-done trials, often to the great surprise of thought leaders and experts in a specialty. This is "Medical Reversal", which doesn't sound all too exciting, but is actually super important.

To understand why, we need to dig into a few foundational points. The most important of them is the following: medical theory, in the sense of scientific models that allow for accurate prediction of real-world outcomes, is very poor. Another way of saying this: biology is very complex, and predicting real-world outcomes in biological systems with any precision is mostly impossible.

The second most important point is that the vast majority of modern medical interventions, even when they do work, have modest effect sizes. A new cancer drug, if you're lucky, might increase survival by an average of a month, a new weight loss drug might let you lose another 5-10 lbs (Semaglutide, with its 14% weight loss, is a welcome recent exception!), and a surgery might save 79% of patients instead of 77%. These are made-up numbers, but the point stands: on average, when new drugs works, which is itself rare, they tend to be incremental improvements over the old standard of care.

There are exceptions, but they're rare in the modern era: for instance, you didn't need RCT's to see that antiseptic techniques improves surgery, because the effect sizes were enormous-- operating on the peritoneum, previously a death sentence, became possible. Amputation mortality rates fell by more than 50%. The effects were so powerful they could be seen with practically any study design, especially when compared to no treatment at all.

The third point is that most interventions, even the best funded and most plausible candidates, fail when tested in the real world. As the authors put in their book, this means that the prior probability of any given intervention working is very low, even in the late phases of drug development. This has implications for the types of evidence that physicians should require to shift their guess on whether a given drug works. Importantly, developing medical interventions is incredibly hard and expensive, and seems to be getting harder over time, giving rise to the reverse Moore's Law that has been termed "Eroom's Law" (incidentally, this may have reversed slightly in the last decade but that’s a preliminary finding).

As a result of these 3 factors, to really know if a medical intervention is effective, you need randomized controlled trials (RCTs) where the effects of a single intervention are tested on roughly similar groups of patients and outcomes are tracked in a rigorous manner. RCT's are onerous, slow, and require immense coordination– but they've overturned many pet theories and biologically plausible interventions. The fact that these theories and interventions turn out to not work is not the problem– after all, false starts and experimentation are how science progresses– the problem is that they're often implemented in clinical settings prematurely, before rigorous testing, and only later, as doubt builds, are they truly tested.

This is "Medical Reversal". There are other important arguments in the book, but they're mostly subsidiary to these 3 key arguments. Among them is a critique of surrogate endpoints, criticism of sloppily and hastily implemented 'systemic' interventions in hospital systems, the profusion of observational studies purporting to show harmful or beneficial effects of lifestyle factors, and the supplement industry.

There's a more radical part of the book, which I think is underappreciated: the call to action in the later half of the book. The authors argue for a new era in medicine, wherein a large fraction of patients will be enrolled in RCT's, constantly testing medical dogma, even for simple questions that are currently poorly understood. They want an easier and more seamless way to recruit patients in clinical trials, which would be done at lower cost than current methods, and think this can be done partly by viewing RCT's as the default.

Any practice in medicine that is currently understudied would be subject to this "RCT by default, opt out if you want" framework. Since RCT's would be testing treatments that are, theoretically, in equipoise, this would be more ethical than our current era, in which many medical interventions are only tested rigorously AFTER being used on patients in uncontrolled settings. As they correctly point out, the"safest way to receive a new drug is in a trial with a control arm. The randomized-controlled-trial design provides a built-in safeguard—trials are stopped if the treatment turns out to be harmful".

As they acknowledge, this radical change is a big ask, but "evidence-based medicine...is the only rational way to provide care".

Later in the book they critique medical education for it's elevation of purported drug mechanisms and biological minutiae over a better understanding of medical evidence, landmark clinical trials, and more contact with patients in clinical settings. They call for an overhaul of medical education.

The primacy of the basic sciences is the reason that cardiologists could not accept the finding that niacin did not save lives. It is why radiologists could not accept that vertebroplasty did not help back pain. It is the reason orthopedists could not accept that repairing torn menisci did not help knee pain.

The book isn't perfect. There are some relatively minor factual stumbles which don't affect the core arguments. For example, it uncritically cites "Nudge" by Sunstein and Thaler, which is based in large part on Behavioral Economics, but at least some of that work has not fared well in the replication crisis, as Kahneman himself (of Thinking Fast and Slow) has pointed out in a blog post. So that's pretty ironic.

I also think its deceptive to say that supplements don't have to admit to a lack of evidence. They do. Every supplement I've ever purchased has had, in clear writing, "This product is not approved....to treat or prevent any disease". Of course, these same supplement bottles claim benefits, so its a mixed message, but supplements certainly have labeling to indicate they're not approved.

The authors also repeatedly criticize pharma-funded trials and influence, but my understanding is that pharma-funded trials generally comply with reporting regulations better than academia funded trials.

Then there are the ideological disagreements I have with the book. I wonder what the authors would think of cosmetic surgery when they write “the list of medical practices that improve outcomes among healthy individuals is a very short one." They're right about medical outcomes, but individuals have goals besides health, and cosmetic surgery can occasionally help achieve those goals.

In regards to supplements, individuals often have goals besides health as well: performance in the gym, aesthetics, etc. There are a small handful of supplements with RCT evidence of efficacy: creatine is probably the best example. It works pretty reliably, though with a modest effect, on increasing muscular endurance and likely muscle growth. Caffeine reliably improves performance in some categories of exercise. Both are used by millions, with very good safety profiles. Would the authors propose that all non-medical usage of supplements be banned? I would love to hear their thoughts on that. More controversially, plenty of people would make trade-offs between quality of life and lifespan. How would a clinical trial be run on these questions? They could theoretically be done, but as it stands I think any trial that, for instance, allowed healthy middle-aged men access to exogenous testosterone in supraphysiological doses, which might shorten lifespan but improve quality of life, would struggle to get run.

Cosmetic surgery and supplement use for non-medical purposes both push on an interesting corner of their argument: freedom. For the clinical trial infrastructure to work, participating in a trial has to be the main way that drugs get approved and patients get access to experimental medications. If alternative pathways to drugs are available in large numbers, and enrollment in trials is threatened, this whole RCT structure falls apart. Medical freedom is not exactly compatible with "RCT's for all".

To their credit, the authors sort of touch on this: they focus on the pull side of RCT incentives— if clinical trials are much easier to run, if inclusion criteria for new drugs were not so strict, if many more patients were enrolled, etc. then patients would have fewer incentives to try to get drugs outside of approved protocols. And yet, with urine and blood testing of drugs, medical tourism, and patients sharing information about side effects with each other over social media, it is becoming increasingly possible for patients to figure out who is in a control/experimental arm and get drugs outside of approved channels. These are real threats to RCT's, and I'm very curious what the authors think of these developments.

My final disagreement is with optimism in medicine. I'm not even sure the authors would disagree with this, since they're not really aiming their critique at drug researchers per se, but just in case: while drugs with large effect sizes are rare in medicine, they're not unknown. In the modern era, perhaps only HIV and Hep-C drugs, along with immunotherapy in some cancers and Gleevec meet this benchmark of large effect sizes. While most medicines will be incremental advances of the state of the art, I hope we continue looking for moonshots. We should be properly skeptical of anybody who claims to have found one, but rejoice if a good RCT shows them to be right. Expect failure or incremental advance, rejoice when we're wrong!

Overall, I can't recommend this book enough and I wish I could make every medical school assign it as required reading. 5/5!