Spiegel: Science vs. truth, an approach to journal analysis

Today we are visited by Dr. Rory Spiegel. The man behind EM Nerd and the most recent winner of the EMRA Educator and FOAMer of the year! Dr. Spiegel is most recently completing his Resuscitation Fellowship at Stony Brook University Medical Center where every day activities focus on two things: the creation of an ED-ICU model AND reviewing/critiquing scientific papers. Today he will be doing his best to concentrate his brilliance into a 45 minute presentation. I have heard a ton of lectures on EBM, but never before has one made so much sense in such a short period of time!

Clinical Pearls (Thanks to Dr. Neil Christopher)

  • Truth is the underlying reality & Science is the means by which we approach it
  • Why use Science rather than empiric observations?
    • The use of bloodletting in medicine is an example of when our observations do not correlate to the underlying truth
    • Empiric observations do not take into account confounding factors (i.e. patients do well in spite of our interventions)
  • Elements of Journal Article Evaluation
    • Error – difference between the TRUE benefit and the experimentally determined benefit
      • Quantified by the P value – Reject the NULL hypothesis falsely 5% of the time
    • Hypothesis Testing – The importance of stating the hypothesis prior to obtaining data
      • As you increase the number of observations, the chance that you will randomly find a statistically significant result increases
      • Ex: ISIS-2 trial – ASA benefit in MI – subgroup analysis leads to sampling error which can make insignificant results appear statistically significant
    • Bias
      • Internal Validity – Applicability of the results to the population within the study
      • External Validity – Applicability to the wider population
      • Randomization allows for control of bias and confounding variables
        • Allocation concealment – treating physician does not know what patients got randomized to what treatment
          • Prevents selection bias
        • Permutated block randomization – Attempt to equalize randomizations so that the number of patients in each group is comparable
          • Can lead to unconcealment of blinding (physician can predict which patient will get which treatment)
    • Power
      • Probability that test rejects the null hypothesis when the alternative is true
        • Beta error = statistical power
      • Assumes a fixed percentage difference between patients – the minimum difference between the groups that you are going to call significant
    • Confidence interval – if study was repeated the results would fall within the CI 95% of the time
    • Statistics:
      • Frequentist Statistics
        • Reject or accept the null hypothesis based on a single test – ignores additional information that affects the likelihood of a true result
      • Baysean Statistics
        • We should be using baysean statistics – Probability is based on the pretest probability as well as the strength of the test that we are using
        • Bayes Factor = Probability of data given the null hypothesis / probability of data given the alternative hypothesis
        • The higher your pretest probability, the less strong of a test you need to confirm your hypothesis
  • Example – FLORALI trial – Examined the use of High Flow Nasal Cannula vs BiPAP vs Facemask in patients with hypoxemic respiratory failure
    • Blinding – not done in this trial
      • Contamination – Significant amount of crossover between groups
      • Makes the difference between the groups smaller
    • Excluded very sick patients – Affects the external validity
    • Results
      • No statistical difference between groups in primary outcome (intubation at 28 days)
      • Statistically significant change in P/F ratio, ventilator free days, mortality favoring high flow nasal cannula
      • Many studies show that BiPAP has an approximate failure rate of 50%, as shown in this study
      • When these patients fail BiPAP, they have higher mortality
    • Conclusions
      • Negative trial for HFNC – Not enough data to say whether HFNC is beneficial
      • Enough data to say that BiPAP can worsen outcomes in this population

Suggested Reading

  1. Frat J-P, Thille AW, Mercat A, et al. High-flow oxygen
    through nasal cannula in acute hypoxemic respiratory failure.
    N Engl J Med 2015;372:2185-96.[NEJM Link]
  2. Goodman SN, Berlin JA. The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Ann Intern Med. 1994 Aug 1;121(3):200-6.[PubMed Link]
  3. Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med. 1999 Jun 15;130(12):995-1004.[PubMed Link]
  4. Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999 Jun 15;130(12):1005-13.[PubMed Link]
About the Author

Jim Lantry

Twitter Google+

Just your average critical care doc: Wandering the ED and ICUs for the USAF down in the San Antonio Military Medical Center, traveling the globe to cannulate for ECLS wherever the need arises, and trying to keep up with great minds of today. E:

Share this Post


Leave a Comment