EBM's and RCT: Doubt, Scientism and unquestioned Ideologies

update 8-Apr-2012: Quotes from "Evidence-Based Medicine: Neither Good Evidence nor Good Medicine" by Steve Hickey, PhD and Hilary Roberts, PhD.

• The current approach to medicine is "evidence-based." This sounds obvious but, in practice, it means relying on a few large-scale studies and statistical techniques to choose the treatment for each patient. Practitioners of EBM incorrectly call this process using the "best evidence."
• Significant Does Not Mean Important...
• Large trials are powerful methods for detecting small differences.
• Now, tiny differences - even if they are "very highly significant" - are nothing to boast about, so EBM researchers need to make their findings sound more impressive.
• They do this by using relative rather than absolute values.
• Suppose a drug halves your risk of developing cancer (a relative value). Although this sounds great, the reported 50% reduction may lessen your risk by just one in ten thousand: from two in ten thousand (2/10,000) to one in ten thousand (1/10,000) (absolute values). 
• Such a small benefit is typically irrelevant, but when expressed as a relative value, it sounds important. (By analogy, buying two lottery tickets doubles your chance of winning compared to buying one; but either way, your chances are miniscule.)
• There is a further problem with the dangerous assertion implicit in EBM that large-scale studies are the best evidence for decisions concerning individual patients.
• This claim is an example of the ecological fallacy, which wrongly uses group statistics to make predictions about individuals.
• There is no way round this; even in the ideal practice of medicine, EBM should not be applied to individual patients.
• In other words, EBM is of little direct clinical use.
• As we have mentioned, EBM restricts variety to what it considers the "best evidence."
•  However, if doctors were to apply the same statistically-based treatment to all patients with a particular condition, they would break the laws of both cybernetics and statistics. 
• Consequently, in many cases, the treatment would be expected to fail, as the doctors would not have enough information to make an accurate prediction.
• Population statistics do not capture the information needed to provide a well-fitting pair of shoes, let alone to treat a complex and particular patient.
• As the ancient philosopher Epicurus explained, you need to consider all the data.
• A doctor who arrives at a correct diagnosis and treatment in an efficient manner is called, in cybernetic terms, a good regulator. 
• According to Roger Conant and Ross Ashby, every good regulator of a system must be a model of that system. Good regulators achieve their goal in the simplest way possible.
• In order to achieve this, the diagnostic processes must model the systems of the body, which is why doctors undergo years of training in all aspects of medical science.
• In addition, each patient must be treated as an individual.
• EBM's group statistics are irrelevant, since large-scale clinical trials do not model an individual patient and his or her condition, they model a population-albeit somewhat crudely.
• They are thus not good regulators.
• Once again, a rational patient would reject EBM as a poor method for finding an effective treatment for an illness.
• Diagnosing medical conditions is challenging, because we are each biochemically individual.
•  As explained by an originator of this concept, nutritional pioneer Dr. Roger Williams,
• "Nutrition is for real people. Statistical humans are of little interest."
• Doctors must encompass enough knowledge and therapeutic variety to match the biological diversity within their population of patients.
• The process of classifying a particular person's symptoms requires a different kind of statistics (Bayesian), as well as pattern recognition.
• These have the ability to deal with individual uniqueness.

---

The Friends of Doctors espouse an uncritical Ideological belief in a simplistic doctrine:

Evidence Based Medicine is the only source of Good Science and hence Good Medicine.
All else is, by definition, irrelevant, invalid and, at worst, quackery.

Which is a variation on Scientism, "the universal applicability of the scientific method and approach".

In 1898 you might've excused a Great Expert from declaring "We know everything and have invented everything" [paraphrased] - but in the 21st Century, for anyone to have the arrogance and hubris to make universal/absolute statements that are not dissimilar is unbelievable.
Doubly so, if like FoSiM, they hold themselves up as Great Experts (Professors with many awards and decades of experience).

I have a very specific objection to the FoSiM position, roughly, EBM/RCT's are OK as far as they go, but are far from being the only thing:

Evidence Based Medicine (EBM) and Randomised Double-blind, placebo Controlled Trials (RCT's) are NOT definitive in themselves, they are far from the only source of valid evidence to support clinical practices and treatment.
While the underlying Science is necessary, a lot more is needed to arrive at safe treatments consistently delivered in Practice.
RCT's are a necessary, but not sufficient, way to gather evidence, but can never provide proof. Popper's "falsification" notion says theories can never be proven, only disprove with 1 counter-example. The source of the economics/finance term "Black Swan" - something completely new and unexpected.

Why would a group of eminent persons go out of their way to make themselves look complete fools, espousing an entrenched and immovable position that is obviously flawed?

The only reasonable answer I can come up with is:

They are fighting a Turf War and using EBM/RCT's as an overwhelming strength with which to beat-up their opponents. But if the opponents start to provide RCT's, then they can either play "Change the Rules" or "Move the Goal Posts" to force the opponents to waste time and resources.

The unreasonable explanation is these folks are uncritically and intractably wedded to the Ideology, "EBM and RCT's are everything".

---

So, if RCT's are a good experimental methodology and EBM beats the pants off what Establishment Medicine was doing 50 years ago, why isn't that the end of it?

There are two headline problems and a lot of "fineprint" to uncover with RCT's.

The headline summary is: EBM based solely on RCT's are fatally flawed - they completely ignore real-world Service Delivery and Patient Safety issues.

• Good Practice does not automatically flow from "Good Science".
• Just delivering good enough service, treatment and medicine, economically to all people in a population is a major challenge to which the underlying "Science" is at best tangential, at worst, irrelevant.
• Practitioners like "Dr Death" of Bundaberg amply demonstrate the "Knowing, Doing Gap", let alone the "Potential vs Actual" service delivery problem.
• For the worst doctors, efficacy of treatments is completely irrelevant. Patient Safety is of prime importance and absent if Safety and Quality systems are inadequate or absent.

• Treatments and Medicines need to be Safe and Effective. RCTs only seek evidence for one half of the equation - efficacy.
• RCT's do not, nor seek to, establish the real-world safety of treatments/medicines. It takes years of treatment and many iatrogenic injuries and deaths to establish a good view of Patient Safety. Then a judgement needs to be made if Efficacy trumps Safety or not.
• Drugs like Vioxx, "linked to thousands of deaths", are the predictable and inevitable outcome of unquestioning acceptance of the Ideology of "RCT's and EBM is the only One True Base for Medicine".

RCT's, or more properly, Randomised Double-blind, placebo Controlled Trials sound pretty impressive. What could possibly be wrong with them?
Let's go back to the full name and pick it apart.

• "Randomised Double-blind". Why? Because people can bias experimental results without knowing.
• This sort of experiment is very sensitive to bias, accidental or systemic, throwing doubt onto the results of every Controlled Trial.
• Psychological effects on experimenters and subjects is exceedingly subtle and complex, to say "there can be no bias because we know everything about doing this" is supreme arrogance. That there have been many significant effects uncovered in the last 50 years suggests that many more will be uncovered in the next 50 years.
• The simplest of biases comes from the difference between the target population and the testing population.
• The most subtle is the genetic differences in populations between countries. What works in Africa, may not in Asia or European countries, and vice versa.
• The Elephant in the Room for RCT's is One Team, One Environment, One Test.
• There is an iron-clad law of Quality: You cannot test your own work.
• With experiments, this translates to, you cannot validate or check your own experimental data.
• As a minimum, RCT's need to be Triplicated in diverse contexts with diverse populations to address even this simple type of bias.

• "Placebo Controlled Trials". Why? Because "mere suggestion" has very powerful healing effects.
• The baseline controlled against is never "no treatment" but a "sham treatment".
• The reason "Controlled" trials are needed is because if people think they are getting treatment, their bodies start to heal themselves. It's a powerful enough effect that it cannot be ignored - it's enough to overwhelm normal statistical analyses.
• Here's the nasty secret RCT-base EBM doesn't discuss: Nobody knows the mechanism of the Placebo Effect. Because its named, people think its explained.
• So what if we don't know the exact mechanism, we've developed effective treatments for a century or more without knowing exact mechanisms?
• But that's the whole point of RCT-backed EBM, to be very exact about what works, and how it works. Who's to say the efficacy of a large fraction of treatments/medicines who's can't be doubled or tripled by "tweaking" the placebo effect.
• The secret is nastier: whatever the Placebo Effect mechanism is, it cuts both ways. It can promote or retard healing. Without knowing the mechanism(s), RCT results have all the validity of home-made magic potions. Is there something that normal research teams do that modifies the action of treatments but isn't replicated in normal Practice? Nobody knows - everyone pretends, without any certainty, the effect has been compensated for, which is wishful thinking, not Good Science.
• This is a central contradiction of RCT's used for EBM: it gives the appearance of Science because there are number with a lot of decimals and many fine, complex statistical analyses, but its all smoke and mirrors to disguise the truth, "We really don't know what's going on, but are pretending its All Good."

• In a Controlled Trial, drugs are tested alone. This is necessary to get "clean data" and verifiable, trustable results. It also isn't how drugs are used. Drugs are used in combination with many things, which are specifically untested in RCT's.
• Drug interactions, especially increased side-effects, are a major problem with new drugs.
• With the increasing numbers of drugs, treatments, supplements, exotic foods and traditional medicines in use, it isn't possible to test new drugs against all expected use-cases, let alone against all possible combinations.
• EBM quietly overlooks that Controlled Trails, while presenting valid single-drug data, provide no evidence about real-world usage. Each patient is a new experiment.

• What does a failed treatment experiment look like? When RCT's don't cut it?
• Iatrogenic Injuries and deaths...
• Proponents of EBM don't go near this, it just wouldn't look good in Journal. Of course, the papers are already published and the Trials shutdown, there is no way to go back in time and add those problems to the RCT when published. Each patient is a new experiment.

• The reasons our species thrives and survives are many, but part of it is the constant genetic mixing from parents. We each have very individual physiologies - making us tolerant and sensitive to 'chemicals' in very individual ways. 
• Controlled Trails make two exceeding dangerous and simplistic assumptions:
• The trial and target populations correspond well enough. Until the trial is validated in practice, how can this assumption be validated? Each patient is a new experiment.
• Individuals won't react outside the bounds seen in the trial. All the trial data needs to be reduced into simple dosage instructions "xx mg/kg body weight". How does your  treating Physician discover you are either exceptionally tolerant or sensitive to this new challenge to your system? Each patient is a new experiment.
• Of course, we just don't mention abreactions, anaphylaxis and allergic reactions in Trials, or devise methods to avoid foreseeable problems. Each patient is a new experiment.
• Individual sensitivity to drugs is not constant over time. For example, dosage of opiates needs to be reduced over time as liver function reduces with days spent in bed. Each patient is a new experiment.

But wait, there's more...

• The bane of all experiments are Errors, Mistakes (in Design and Execution) and incorrect/incomplete models leading resulting in "knowledge gaps" - things that should be done, that aren't.
• Systemic and systematic errors in methods, skills, execution can't be detected within the originating team. Like every individual, we cannot see ourselves from the outside. We need others to check our work, we are perceptually blind to our mistakes and problems/faults.
• Normal RCT experiments fail basic Quality Systems design:
• there have to be checks built into the process to validate "production".
• This normally requires a separation of Design, Execution and Data Analysis.
• Again, I think Triplicated Controlled Trials are the minimum requirement for this methodology. It is trying for Perfection when Service Delivery is so severely flawed and defective that it makes new Trials mostly irrelevant. For the next ten years, if we just did what is known to work, everywhere and all the time, that would provide orders of magnitude improvement in Patient outcomes.

• Human Nature drives fraud, intellectual and commercial, it can never be eliminated from Science. medical or not, it can only detected and controlled. When the commercial rewards are high, the drive is much stronger and methods employed more subtle and devious. It becomes intentional, organised activity, not "rogue individuals" and "opportunistic amateurs".
• Because there is so much at stake, both in human lives, pain and suffering and money, you'd think that extraordinary efforts would be made to prevent and detect all types of fraud and rigged results.
• But you' be wrong. Major drug RCT's aren't subject to strict independent fraud checking. Just the usual and quaint "peer-reviewed academic journals".
• What this says about the gullibility and naivety of Medical Science is debatable, but they are sorely lacking in insight and a sense of public duty.

In summary:

Medical Science uses RCT's because it's the best thing they've got, but belief in them "should be held lightly", they are not infallible nor free of serious deficiencies.

Evidence Based Medicine is a good servant and a poor Master. The emphasis must be on Medicine, not 'Evidence', on providing good patient care and outcomes. Chief of which is focussing on Patient Safety, not the glittering bauble of "efficacy". "First, do NO harm"...