Dr. Burton Lee returns to MarylandCCProject to discuss the second half of his deeper dive into numeracy of the medical literature (“deep dive” is a bit of a stretch for him, but for us – there’s fractions involved…. FRACTIONS)! All kidding aside, we are always grateful to spend time with Dr. Lee and lucky to have him teach us how sift through the morass of literature guiding our medical decision making.
Podcast: Play in new window | Download
Subscribe: Apple Podcasts | RSS
Pearls
Causes for medical literature reversal
- Poor methodology (RCTs, meta-analyses, quantitative reviews, etc.)
- Conflict of interest
- Innumeracy with the medical literature ***
Core concepts of numeracy continued…
- Law of small numbers: Relative statistics (such as percentages) can be misleading with small sample sizes.
- Regression to the mean: The most likely outcome for any test is the average result. Clinical relevance: The outcome of an intervention may not actually make an actual difference, only a perceived difference if the experimental group is different from the true mean. Also known as the gambler’s fallacy.
- Apparently effective vs. Actually effective: Remember, even if a perfect RCT shows a significantly positive finding, it is only reporting an apparently effective result. When taking into account both alpha & beta error, the chance a paper is reporting an actual truth is usually around 50%!!
Interesting concept – the Number Needed to Read (McKibbon et al., 2004) How many journal articles must you read before finding ONE quality study? The suggested numbers may surprise you!
So where do we go from here? In his paper, “Why Most Published Research Findings are False” John Ioannides suggests,
Instead of chasing statistical significance, we should improve our understanding of the range of R values—the pre-study odds—where research efforts operate. Before running an experiment, investigators should consider what they believe the chances are that they are testing a true rather than a non-true relationship. Speculated high R values may sometimes then be ascertained. As described above, whenever ethically acceptable, large studies with minimal bias should be performed on research findings that are considered relatively established, to see how often they are indeed confirmed. I suspect several established “classics” will fail the test.
