Can you use AI in real life? Here's an example where AI can act as a "second reader" and help you cross-check your opinions, or be a backstop for something you may have missed.
On September 29, 2023, FDA released a 26-page proposal to rapidly regulate all LDTs, as well as a 127-page "Preliminary Regulatory Analysis" with speculative financial projections. For me, one of the oddest things is you have to read up to about page 100 of the online appendix document to discover that costs to industry would be some $40B in the first several years (representing >100,000 man-years of regulatory time). Also, page 29-52 of the online appendix show the "benefits" - usually presented as $30B a year of savings - actually represent extremely jerry-rigged speculative life-value extrapolations, not cash costs.
Chat GPT 4 Gives an AI Viewpoint
In a Google doc, I offer about 12 pages of Q&A with Chat GPT 4 about the proposal and the PRA. Here, I clip some sections abou the Benefits Analysis, taken direct from GPT 4. Are these AI remarks, the end-all and be-all of the analysis? No, but the AI-generated viewpoint may well be helpful in cross-checking the human opinion you form by reading the original documents.
How would one present an argument that he has read the FDA benefits framework carefully, and he is concerned it is full of loose connections, guesswork, and quick abstractions, and the dollar values (such as $30B of health care new value per year) resulting from new FDA regulations, cannot be taken at face value as an accurate dollar (or dollarized) value.
AI:
To present an argument that the FDA's benefits framework is speculative and its projected dollar values for health care benefits are not reliable, one would need to critique the methodology, assumptions, and data sources used by the FDA in their analysis. Here’s a structured approach on how to articulate these concerns:
Methodological Concerns:
Uncertain Baseline Risks: Point out that the baseline risk of problematic IVDs is not well-defined due to the FDA's current enforcement discretion. The lack of rigorous data on the baseline creates an uncertain foundation for the entire analysis.
Predictive Difficulties: Emphasize the speculative nature of predicting future benefits based on past events, especially given the rapidly evolving landscape of medical diagnostics and the unique circumstances of each case.
Assumption Challenges:
Misdiagnosis Rates: Challenge the assumption that 50% of misdiagnoses occur after testing with an IVD and that 50% of these are associated with LDTs. These figures could significantly inflate the estimated benefits if not accurately reflective of real-world data.
Problematic IVDs Proportion: Critique the assumption that 47% of LDTs are problematic, which is based on a single study's findings. This could introduce bias if that study isn't representative of the broader situation.
Value of Statistical Life Year (VSLY): Question the application of VSLY in translating health outcomes into monetary values, as this can be inherently subjective and varies significantly based on the chosen discount rate.
Data Source Scrutiny:
Limited and Biased Data: Address the reliance on selected studies and reports, which may not represent the full spectrum of IVD performance or the diversity of conditions affected by diagnostic accuracy.
COVID-19 Case Study Limitations: Point out the limitations in generalizing findings from COVID-19 test evaluations to all IVDs. The pandemic presented unique challenges that may not be indicative of the usual regulatory landscape.
Abstract Calculations:
Life Expectancy Gains: Highlight the overly abstract calculations used to estimate life expectancy gains, which may not account for individual variability in disease progression and treatment response.
Quality Adjusted Life Years (QALYs): Discuss the problematic nature of QALYs, which attempt to quantify the quality and quantity of life, but are difficult to accurately measure and are often contested in health economics for their simplification of complex health states.
Economic Projections:
Dollar Value Estimates: Critique the broad range of estimated dollar values ($1.78 billion to $80.8 billion), which suggests high uncertainty in the projections. Such a span implies that the FDA is not confident in its estimates and that they may not be reliable.
Annual Benefit Projections: Address the speculative nature of projecting annual benefits from averted cases of a COVID-like pandemic, which relies on the uncertain likelihood of such an event occurring in the future.
Regulatory Precedent and Public Comment:
Absence of Regulatory Precedent: Mention the lack of strong regulatory precedent for such valuation of benefits, which could mean that the FDA is venturing into relatively uncharted territory without established methodologies.
Solicitation of Public Comment: Note that the FDA itself requests public comments to improve the accuracy of its benefit projections, indicating that the current analysis is a preliminary and not a definitive assessment.
In presenting this argument, it's important to acknowledge the inherent difficulties in quantifying the benefits of health regulations and to propose alternative methods or suggest the need for more robust data collection to improve the reliability of such projections. Finally, your critique should be framed constructively, aiming to enhance the FDA’s approach rather than merely dismissing it.
AI:
You're correct, and your point is well-taken. When regulatory agencies, like the FDA, present the outcomes of cost-benefit analyses, clarity is paramount, especially in communications intended for the general public, such as press releases and rule summaries. If the distinction between abstract valuation and direct cash costs is not made explicit, it can lead to misunderstandings about the rule's implications:
1. Clarity in Communication: It's crucial for the FDA to clearly differentiate between the abstract value of health benefits and the real monetary costs in all public-facing documents. Failure to do so could mislead stakeholders and the public about the actual financial impacts of the rule.
2. Expectation of Transparency: The public generally expects government communications to be transparent and easily interpretable. Misconceptions could arise if the presentation of costs and benefits is ambiguous or if there is a lack of context explaining the nature of these values.
3. Misinterpretation of "Dollars": If not properly explained, the term "dollars" could be misinterpreted as representing direct economic savings or expenditures, when in fact, part of it refers to a theoretical construct used to value improvements in health or life expectancy.
4. Importance for Decision Making: Policymakers and stakeholders rely on these figures to make informed decisions. If the figures are not contextualized properly, it may lead to inappropriate policy responses, misallocation of resources, or skewed public opinion.
5. Recommendation for Correction: The FDA should consider revising its public statements and documentation to ensure that the types of costs and benefits are clearly labeled and explained. This would include distinguishing between cash costs that will impact budgets now and the abstract value of benefits that represent potential future societal savings or gains.
6. Educating the Public: Part of the FDA's responsibility in these cases is educational. It should ensure that readers understand the complex nature of cost-benefit analysis in a regulatory context, especially when it comes to valuing human health and life.
In conclusion, it's essential for the FDA to maintain a high standard of clarity and transparency in its communications. The abstract value of health benefits, often represented through metrics like QALYs or the value of statistical life, should be distinctly presented against tangible, out-of-pocket costs to prevent any confusion or misrepresentation of the rule's financial implications.
Chat GPT wrote me a full-length formal FDA LDT comment letter, here.
