Evidence
23 pieces tagged Evidence.
The Pyramid Schema: The Origins and Impact of Evidence Pyramids
[This paper has been updated in 2026 from the 2022 preprint] Evidence pyramids are amongst the most recognisable artefacts of the Evidence-Based Medicine movement. Yet no study has established the origins of evidence pyramids, or analysed whether they offer any information beyond simple lists or tables. In this paper, I establish the origins of the first evidence pyramid and argue that the pyramidal turn is a retrograde step in evidence appraisal.
New publication: Grading evidence from qualitative research
Announcing a new paper co-written with a EULAR working group, reporting the findings of a systematic review of qualitative evidence appraisal tools.
Machine Evidence II: The Abstract Setting
A recent study by Gao et al. (2022) validates the warning of 'Machine Evidence' (Blunt, 2019) that language models would soon become capable of beating detection attempts by human peer reviewers. This piece looks at the near-term steps that journal editors and conference organisers can take to prevent AI-generated abstracts bypassing their screening processes, along with a warning for the long-term viability of those strategies.
[Superseded] The Pyramid Schema: The Origins and Impact of Evidence Pyramids
Evidence pyramids are amongst the most recognisable artefacts of the Evidence-Based Medicine movement. Yet no study has established the origins of evidence pyramids, or analysed whether they offer any information beyond simple lists or tables. In this paper, I establish the origins of the first evidence pyramid and argue that the pyramidal turn is a retrograde step in evidence appraisal.
Bias and the Myth of the Objective Average
Would you choose a black box AI surgeon with a 90% success rate over a human surgeon with 80% success? The answer exposes a fundamental and harmful assumption within dominant models of medical evidence.
Visions of Evidence
How does a machine learning algorithm picture hierarchies of evidence and evidence-based medicine - and what do these visions of evidence remind us of the way we understand, order and assemble the information we use to guide clinical practice?
Lumping and splitting: brain tumours in white Britons
A much-publicised report suggests that white Britons' brain tumour survival rates are lower than other ethnicities. But analysing the ethnicities categories used, and considering the diversity of the "brain tumour" label, complicates the picture, as the 'Dismal Disease' of Glioblastoma continues to confound.
On the Global Summit: do critiques of evidence hierarchies favour chiropractic?
The Global Summit systematic review claims that spinal manipulation therapy is not effective in preventing any non-musculoskeletal disorders. But a breakaway group has challenged their findings, in part based on my arguments regarding evidence hierarchies. Are they correct? Does my critique undermine the Global Summit review? If so, does the evidence base favour chiropractic?
Minding the Gaps: Statistical Misrepresentation in Attainment Gap Research
Political interests configure the stories we tell with data. Closing the gap in attainment between disadvantaged students and their advantaged contemporaries is pivotal to an agenda to use education as a positive social force. But both the measurement and representation of this gap is politicised, skewed and open to manipulation. This paper shows how two organisations with inverse aims represent—and misrepresent—their measure of the attainment gap to portray diametric trajectories in the pursuit of equal attainment.
Conversion Therapy: Evidence is Irrelevant
Pressure mounts upon equalities minister Kemi Badenoch to resign over the UK government's failure to ban conversion therapies. Attention has focused on the government's failure to publish research commissioned in 2018. But evidence about whether conversion therapy works is irrelevant: conversion therapy is not a medical intervention.
195 Hierarchies: A Systematic Database Update
The database of evidence hierarchies has been updated based on a new systematic review of the medical literature, and now contains over 195 hierarchies.
A Ghost of Progress - How Hierarchies Become Fixtures
I have written extensively on hierarchies of evidence in evidence-based medicine. The origin story of hierarchies of evidence is a little contentious. Several sources in EBM cite Campbell and Stanley's 1963 classic "Experimental and Quasi-experimental Designs for Research" as...
The Dismal Disease: Temozolomide and the Interaction of Evidence
Evidence interacts. To understand the evidence base for an intervention we must look not only at individual studies, but the relationships between them.
All that Glitters is not... Evidence
Recently, I wrote about a new machine learning model called GPT-2 which was conspicuously not released by OpenAI. GPT-2 is a massive language model which can be used to generate often highly convincing text when given a prompt. Using the 'attention' framework, the language...
Machine Evidence: Trial by AI
Take a look at the following snippets from descriptions of clinical trials, thinking about how you'd rate the quality and strength of the evidence that comes from each:
Echoes of Evidence
Do you have an idea of what makes good medical evidence in your mind? Do you have ideas about the kind of evidence you’d pay attention to, and the kind of evidence you can safely disregard? Or ideas about the evidence that would strongly sway your beliefs and actions, and the...
The Authority of Evidence-Based Medicine
In the early part of the 20th century, the philosopher Ludwig Wittgenstein sought to demonstrate that metaphysical claims are meaningless. Statements which couldn’t be proven true in some way—through logic or evidence—weren’t even false, they had no meaning at all. But he ran...
The Positivity Machine: "Evidence-Based Alternative Medicine" and Grades of Recommendation
In his beat-poem Storm, the musician and comedian Tim Minchin lays out an apparent paradox. By his criteria, there can never be evidence-based alternative medicine. As soon as there’s evidence that a treatment works, it joins scientific medicine. Minchin goes on to list...
The Parachute Problem: Extracorporeal Life Support and the Demand for Trials
The United States Parachute Association recorded 120 deaths while skydiving in America in 2008-2015. Most were due to human error, while others resulted from collisions with other parachutes or aircraft. Only 5 were due to equipment failure. With around 3 million parachute...
Medicine Needs Diverse Evidence
Do not centralize evidence. If you believe that all the evidence you need to make your decisions is of a single kind, you will be so much easier to mislead. Those who say 'I’ll believe it when I see this kind of study’, once their singular preference is declared, are so much...
The Avoidable Scandal: Benoxaprofen and Theories of Medical Evidence
Benoxaprofen, marketed as Opren in the UK, created a scandal in 1982 when it was withdrawn from sale amidst reports of over 60 deaths and thousands of adverse reactions. The question of who was at fault for this disaster has been addressed many times in the academic...
Against "Effective Treatments"
I am agitating for philosophers of medicine and philosophically-minded clinicians to lead the charge against the term “effective treatment”. This phrase has become ubiquitous in the medical and philosophical literature. But it is a misguided choice which misleads the public...
An Honest Mess
Evidence-Based Medicine is an attempt to simplify and streamline a complex reality into a more manageable structure. Reflective EBM proponents know that matters are very complicated. But they understand that practitioners have very limited time (and abilities) to deal with...