Spiegel: Science vs. truth, an approach to journal analysis

Clinical Pearls (Thanks to Dr. Neil Christopher)

• Truth is the underlying reality & Science is the means by which we approach it
• Why use Science rather than empiric observations?
  • The use of bloodletting in medicine is an example of when our observations do not correlate to the underlying truth
  • Empiric observations do not take into account confounding factors (i.e. patients do well in spite of our interventions)
• Elements of Journal Article Evaluation
  • Error – difference between the TRUE benefit and the experimentally determined benefit
    • Quantified by the P value – Reject the NULL hypothesis falsely 5% of the time
  • Hypothesis Testing – The importance of stating the hypothesis prior to obtaining data
    • As you increase the number of observations, the chance that you will randomly find a statistically significant result increases
    • Ex: ISIS-2 trial – ASA benefit in MI – subgroup analysis leads to sampling error which can make insignificant results appear statistically significant
  • Bias
    • Internal Validity – Applicability of the results to the population within the study
    • External Validity – Applicability to the wider population
    • Randomization allows for control of bias and confounding variables
      • Allocation concealment – treating physician does not know what patients got randomized to what treatment
        • Prevents selection bias
      • Permutated block randomization – Attempt to equalize randomizations so that the number of patients in each group is comparable
        • Can lead to unconcealment of blinding (physician can predict which patient will get which treatment)
  • Power
    • Probability that test rejects the null hypothesis when the alternative is true
      • Beta error = statistical power
    • Assumes a fixed percentage difference between patients – the minimum difference between the groups that you are going to call significant
  • Confidence interval – if study was repeated the results would fall within the CI 95% of the time
  • Statistics:
    • Frequentist Statistics
      • Reject or accept the null hypothesis based on a single test – ignores additional information that affects the likelihood of a true result
    • Baysean Statistics
      • We should be using baysean statistics – Probability is based on the pretest probability as well as the strength of the test that we are using
      • Bayes Factor = Probability of data given the null hypothesis / probability of data given the alternative hypothesis
      • The higher your pretest probability, the less strong of a test you need to confirm your hypothesis
• Example – FLORALI trial – Examined the use of High Flow Nasal Cannula vs BiPAP vs Facemask in patients with hypoxemic respiratory failure
  • Blinding – not done in this trial
    • Contamination – Significant amount of crossover between groups
    • Makes the difference between the groups smaller
  • Excluded very sick patients – Affects the external validity
  • Results
    • No statistical difference between groups in primary outcome (intubation at 28 days)
    • Statistically significant change in P/F ratio, ventilator free days, mortality favoring high flow nasal cannula
    • Many studies show that BiPAP has an approximate failure rate of 50%, as shown in this study
    • When these patients fail BiPAP, they have higher mortality
  • Conclusions
    • Negative trial for HFNC – Not enough data to say whether HFNC is beneficial
    • Enough data to say that BiPAP can worsen outcomes in this population