Friday, March 29, 2024

Evaluating Evidence is Subjective: In JAMA This Week

Header: JAMA Health Forum (open access) has a lead article, "The Subjective Interpretation of the Medical Evidence," by Bauchner and Ioannidis (BU and Stanford, respectively.)


Opening paragraph:

Experts often subjectively disagree on how they interpret the same evidence and what recommendations they derive from it. Meticulous processes to resolve diverging views in guideline development efforts, for example, may not remove subjectivity. Even the most prestigious organizations sometimes have different guideline recommendations. Subjective disagreements can be common, extreme, and unsettling when evidence is limited and rapidly evolving—as in many questions related to COVID-19. However, subjectivity exists, and differences ensue even for common diseases where evidence has accrued and been evaluated for decades. For example, the American College of Physicians, the American Cancer Society, and the US Preventive Services Task Force (USPSTF) vary on when to initiate screening for colorectal cancer and the preferred screening methods. Breast cancer and depression screening recommendations have been debated for decades.


Therefore, the public should be reminded that uncertainty—and even the extent of uncertainty—is common. This realization is not paralyzing; medical decisions can still be made with the best available evidence, once values, preferences, and biases are carefully vetted.


Cited by JAMA alongside this article, include:


AI CORNER with Chat GPT 4

AI summarizes 7 takeaways of Bauchner & Ioannidis.

  1. Subjective Disagreements in Medical Guidelines: Experts and guideline committees often subjectively interpret the same medical evidence differently, leading to divergent recommendations even for well-studied conditions such as colorectal cancer and breast cancer screening.
  2. Specialist vs. Generalist Perspectives: Specialists and generalists may value different aspects of evidence, affecting guidelines. For instance, the USPSTF's recommendations may differ from those of specialist societies due to different evaluation criteria and emphasis on certain types of evidence, like randomized clinical trials.
  3. Impact of Legal and Policy Considerations: Recommendations may be influenced by their potential impact on insurance coverage and policy, with entities like the USPSTF being particularly aware of the implications of their guidance under laws like the Affordable Care Act.
  4. Methodological Differences in Guideline Development: The approach to developing guidelines, including how evidence is synthesized and evaluated, varies significantly among organizations, contributing to inconsistencies. Techniques like the GRADE approach aim for reproducibility but still encounter variability.
  5. Financial and Ideological Influences: Financial support from the industry and allegiance bias among guideline committees and advocacy groups can shape recommendations, with contentious areas such as COVID-19 and cancer screening being particularly susceptible.
  6. Role of Subjectivity in Weak Evidence: In areas where evidence is limited, subjective opinions heavily influence recommendations, highlighting the challenge in balancing emerging evidence with existing treatment options, such as in the management of obesity in adolescents.
  7. Misinformation and Public Trust: The subjective interpretation of medical evidence extends beyond professional circles to media and public discourse, contributing to misinformation, polarized views, and a decline in trust in medical science. This underscores the need for clear communication about the uncertainty and quality of evidence behind medical news and guidelines.

GPT4 illustrates Millar 2023 on Hype (Cited in Bauchner 2023 JAMA Open Netw)