An Early View: Stakeholders Respond to FDA's Concept Paper on LLM-SAMD

One brick at a time, FDA has been advancing LLM-based SAMD.  See the approval of draft-report software for ultraound and see the breakthrough status for radiology CXR draft-report software and see FDA's $1.3M contract for accelerating the regulatory science for radiology diagnostic reporting with AI.

There's also the "big story"- in August FDA released a major concept paper for SAMD in an increasingly LLM world.  

By October 19, some 540 comments were submitted.  Up to yesterday, FDA had released 129 of the comments.  I had Chat GPT filter them for "most important ten comments" on which this preliminary article is based.

See e.g. the AMA's press release on its own comment to the FDA - here.

###

AI Corner Chat GPT

###

FDA Generative AI Medical Device Consultation

What the early comments reveal

ChatGPT 6 • October 20, 2026

FDA’s discussion paper on generative AI medical devices asks how to evaluate software that can produce open-ended clinical responses, change over time, and increasingly take actions. The early comments suggest substantial agreement on the need for a flexible framework, but a significant disagreement about what flexibility should mean. The central question is whether stronger supervision and monitoring can justify less evidence before deployment—and what would demonstrate that those safeguards actually protect patients.

What FDA proposed

Released in August, FDA’s paper outlines a possible approach to generative AI-enabled medical devices and poses 26 questions to stakeholders. It is an exploratory discussion paper, not a new rule or guidance. Its scope extends beyond text-based chatbots to multimodal systems, including applications that interpret images and agents that use tools or execute clinical tasks. [1]

The paper begins with two dimensions of risk: what a device does, including how directly it influences or initiates action, and the consequences of an erroneous output. A general explanation of a disease differs from a patient-specific recommendation, which in turn differs from a system that places an order. But the distinctions are not always clean. An apparently informational response can influence treatment, and a conversation can move from education into urgent medical advice.

For evaluation, FDA proposes competency-oriented benchmarking followed by clinical confirmation. Benchmarks would test capabilities such as clinical knowledge, information gathering, communication, recognition of uncertainty, escalation, and staying within scope. Clinical confirmation would then examine whether those capabilities translate into acceptable performance in the intended setting. The paper contemplates evidence proportionate to risk, rather than requiring the same clinical study for every application.

After deployment, monitoring and change control would address evolving models, shifting populations, and failures that initial testing could not fully anticipate. FDA also asks about independent evaluators, synthetic data, foundation-model information sharing, and the additional risks of autonomous agents.

Two of the 26 questions expose the most consequential choices. Question 18 asks whether stronger postmarket monitoring could justify accepting greater premarket uncertainty. Question 15 asks whether performance should sometimes be compared with the care actually available without the device, including delayed care or no intervention. These questions connect regulatory flexibility to patient access, while raising concerns about who bears the risk of getting that balance wrong.

An early but substantial response

This assessment draws on a selection of substantive early letters, largely from September, together with major October responses from the American Medical Association, American Hospital Association, neurosurgical societies, Personalized Medicine Coalition, and U.S. Chamber of Commerce. It also considers University of California Health’s publicly summarized recommendations. The public docket snapshot showed 541 submissions received but only 129 published, with the available docket export extending through September 30. The findings are therefore an interim reading of significant arguments, not a measure of opinion across the completed consultation. [2]

The device and its workflow matter more than the model label

A recurring response is that risk depends on a particular clinical function and its surroundings. How quickly could harm occur? Can an action be reversed? Can a clinician check the original evidence? Does the user have the necessary expertise? FDA already raises many of these considerations; commenters are pressing for clearer ways to translate them into evidence requirements.

The American College of Allergy, Asthma and Immunology offers a concrete example. Advice about anaphylaxis operates under very different time constraints from advice about long-term asthma management. A similar-sounding recommendation can carry a very different risk when there is little time to recognize and correct an error. Newton’s Tree argues that FDA needs an explicit analysis of hazards, controls, and remaining risk more than it needs additional axes on a diagram. [3, 4]

UltraAI contributes a useful distinction between presenting an existing clinical claim and generating a new one. A system that faithfully summarizes a physician’s report is doing something different from one that infers a diagnosis from an image. Even a summary becomes clinically generative when it changes “cannot exclude” into “is present,” or combines findings into a diagnosis that the source never made. Attaching a citation does not establish that the cited material supports the claim. [5]

These arguments favor evaluation at the level of the clinical function. They also challenge the idea that cautious wording or a disclaimer can determine risk. The Chamber adds a separate qualification: a useful risk framework should not silently become a new legal classification system or automatically determine the regulatory pathway. [6]

Supervision is a clinical capability that must be tested

The AMA’s October 16 letter makes one of the strongest arguments in the consultation: meaningful physician supervision requires more than placing a human at the end of an automated process. The reviewer needs relevant information, expertise, time, cognitive capacity, and authority to reject or stop the system. Supervision also consumes resources. A physician working under ordinary clinical pressure cannot be assumed to perform like a specially trained research reviewer with unlimited time. [7]

University of California Health’s summarized recommendations similarly emphasize the deployed combination of model, prompts, information retrieval, safeguards, and workflow. S. Joseph Sirintrapun, writing in a personal capacity from his experience in pathology and informatics, stresses the practical ability to identify consequential errors. The implication is that supervision itself belongs in the validation study. [8, 9]

RADIN and Total Medical Imaging propose a particularly specific radiology application. They ask FDA to exercise narrowly bounded premarket enforcement discretion for AI-generated drafts that remain private to the interpreting radiologist until affirmative specialist sign-off. The radiologist would have the source images, could edit or reject the text, and would control whether anything enters the clinical record. The proposal excludes automatic finalization and independent ordering or triage. This is a requested policy for a defined workflow, not permission already granted by the discussion paper. [10]

The tension with the AMA is productive. RADIN asks how much regulatory credit a tightly controlled specialist workflow should receive; the AMA asks what evidence establishes that the reviewer can provide the assumed protection. A persuasive answer would measure the performance of the human–AI team: errors introduced, errors caught, omissions corrected, review time, and effects on downstream care.

For radiology and pathology, that is a pivotal research question. A draft can improve performance by directing attention to an overlooked finding, or impair it by anchoring the specialist to a plausible error. Interface design and review burden help determine which effect predominates. Measuring the model alone will miss much of the clinical value—and risk.

Industry does not speak with one voice on premarket flexibility

The most important disagreement concerns the trade between evidence before deployment and surveillance afterward. It does not divide neatly into developers seeking flexibility and physicians demanding caution.

Commenter	Position on the trade
AMA	Necessary premarket assurance should not be reduced on the assumption that later monitoring will compensate.
UltraAI	Monitoring generally complements premarket evidence; substitution would require reliable detection and mitigation before harm.
Clearstep	Supports greater premarket uncertainty in exchange for robust, prespecified monitoring and architectural safeguards.
Personalized Medicine Coalition	Supports conditional flexibility when errors are detectable, reversible, and subject to prompt intervention.
American Hospital Association	Supports streamlined pathways and lifecycle monitoring, with developer responsibility and adequate implementation resources.

 

These positions, set out in the respective submissions, identify the real decision facing FDA. [5, 7, 11–13] It is not simply how much monitoring a sponsor promises. It is whether that monitoring can detect the relevant failure early enough for intervention to matter. A readily reversible error differs from a missed diagnosis discovered months later or an autonomous action that cannot be undone.

UltraAI’s caution is especially noteworthy because it comes from a developer. The company argues that reducing premarket evidence can transfer the discovery of uncertainty to patients while giving the sponsor the benefit of earlier entry. Clearstep, by contrast, argues from its experience with hybrid clinical logic and ongoing clinician review that a stronger postmarket program can support greater flexibility. Their differences reflect competing theories of how safety can be demonstrated, not merely different attitudes toward innovation.

The economic question is just as important. The AMA objects to shifting surveillance work and unresolved product risks onto physicians. AHA emphasizes the burden on hospitals, particularly those with limited resources. A framework that depends on continuous clinical review needs a credible answer about who supplies the labor, data access, and funding.

Better benchmarks require better clinical measurement

The comments generally support structured evaluation, but warn against equating examination performance with clinical reliability. Newton’s Tree offers a useful three-part formulation: establish task competency, confirm clinical performance in the intended environment, and maintain assurance during operation. Each answers a different question. [4]

UltraAI proposes scoring the clinical claims within a report against a reference standard. Different wording can express the same correct finding; elegant wording can conceal a false one. For higher-risk uses, the company also urges developers to constrain the range of inputs and outputs where feasible, rather than assume every application needs unrestricted generation. [5]

The ARPA-H ADVOCATE and Johns Hopkins Applied Physics Laboratory team response adds statistical and operational discipline. Repeated answers to the same case are not independent patients. A large synthetic dataset should not overwhelm evidence of weak performance on real cases. And deployment scale matters: a seemingly modest error rate can become a substantial burden when repeated across a large population. [14]

Several submissions also envision shared evaluation infrastructure. UC Health’s summary proposes refreshed, sequestered benchmarks distributed across health systems; AHA favors multiple qualified evaluators rather than a costly bottleneck. Such arrangements could reduce duplication and improve comparability, but would require sustained governance and funding. The consultation is beginning to identify a market for trusted evaluation services as well as a regulatory need for them.

The less obvious failures deserve attention

Some of the most useful comments broaden the discussion beyond hallucination. Nina Vasan’s Stanford Brainstorm letter draws on evaluations of consumer and youth-facing AI to emphasize omissions, harmful affirmation, and risks that build over repeated interactions. An answer can be factually defensible while reinforcing unhealthy behavior or failing to ask the question that would reveal an urgent problem. These examples inform potential medical-device failure modes rather than establish performance rates for regulated devices. [15]

Escalation provides another example. A system can recognize the need for help and still fail if its alert arrives late, reaches nobody, or sends the patient to the wrong service. Newton’s Tree and Stanford therefore emphasize the completed handoff, not merely the generated warning. This turns an apparently linguistic capability into an operational outcome that can be measured. [4, 15]

Luma’s focused response to FDA’s no-device comparator question raises a different concern: in assistive communication, a system may generate apparent patient speech when no meaningful input was supplied. In that situation, silence can be safer than a fluent fabrication. The AMA approaches the comparator issue from an equity perspective, warning against using limited access to care to justify a lower safety standard for underserved populations. Together, these arguments show why access benefits and minimum safety requirements need separate consideration. [7, 16]

Stanford also raises a rarely discussed lifecycle problem: what happens when a patient has come to depend on a service and the product disappears? For longitudinal applications, advance notice, access to records, referral, and transition support can be clinically important. The relationship between patient and product may outlast the business offering it.

Updates and agents make accountability harder

A medical AI application can change without an obvious change to its interface. The underlying model may be replaced, a prompt revised, a retrieval source updated, or a tool given new permissions. Any of these can alter clinical behavior. UC Health’s summarized call for users to know when the AI changes is particularly important: a model that improves on average can still degrade a previously validated use. [8]

This makes upstream information and control essential. Stanford seeks enforceable rights to version information, change notice, incident disclosure, and support for investigations. Newton’s Tree sees voluntary foundation-model master files as potentially useful, but insufficient to demonstrate the safety of a finished device. Accountability is difficult to exercise when a manufacturer cannot identify or control a dependency on which its product relies. [4, 15]

Agents add the ability to act. A system that schedules, orders, or writes to records must be evaluated across the sequence of decisions and tool uses, including its permissions and ability to stop or recover. The ADVOCATE/JHU APL response and Newton’s Tree emphasize these system-level tests. Individual commenters, including C. D. Rowsell Jr. and Tanisha Tejada Thoreau, add proposals for revalidating authority during execution, audit trails, and runtime stopping controls. These are candidates for development and validation, not established FDA requirements. [4, 14, 17]

An AI supervisor introduces a further dependency. A second model may share the first model’s blind spots rather than independently detect them. The monitoring system therefore needs its own evidence of reliability—a point that connects directly with FDA’s radiology research activity.

The overlapping radiology research contract

FDA’s approximately $1.3 million, 18-month research contract with Cognita Imaging illustrates the related evaluation problem. Announced by Cognita Imaging/Radiology Partners in September, the project uses multiple LLM evaluators and radiologist adjudication of important disagreements to study radiology-report assessment across approximately one million examinations. Its aims include scalable evaluation methods, cohort-size requirements, and practical guidance relevant to premarket and postmarket assessment. [18]

The overlap with the consultation is clear: can automated evaluation make clinical assessment more scalable while retaining trustworthy clinical judgment? There is no established formal linkage between the contract and this docket. They nevertheless occupy overlapping parts of FDA’s research agenda—benchmark validity, AI-assisted supervision, and reliable assessment of generated clinical content. The contract is an investment in evaluation methods, not a regulatory authorization or an answer to the broader debate.

What FDA is getting for its effort

The consultation is producing something more useful than a tally of support and opposition. It is identifying the conditions under which a flexible approach might be credible, and the evidence needed to distinguish effective safeguards from reassuring descriptions.

Its most valuable contribution may be to make broad terms testable. “Human oversight” becomes performance under realistic workload. “Escalation” becomes a completed, timely handoff. “Continuous monitoring” becomes a specified ability to detect a failure before harm accumulates. “Change control” becomes the ability to identify, evaluate, and, when necessary, reverse a clinically consequential modification.

FDA is also learning where implementation could fail for reasons beyond model accuracy. Responsibility, information, operational control, and funding may sit with different organizations. A manufacturer may remain accountable while depending on a foundation-model supplier for change information and a hospital for the data needed to detect problems. Resolving those relationships is part of making lifecycle regulation work.

The comments do not yet prove that the proposed methods are sufficiently reliable or affordable. Nor does conditional support establish consensus about how much premarket uncertainty is acceptable. But the submissions are sharpening that disagreement into questions FDA can investigate: which failures are detectable, how quickly, by whom, and with what effective response?

As more October submissions become available, the most valuable additions will be evidence about those questions—comparative workflow studies, measured supervision performance, monitoring costs, and experience from patients and health systems. The initial record has already moved the debate toward the practical terms on which generative AI could earn a place in clinical care. FDA’s next challenge is to turn those terms into proportionate, reproducible evidence requirements.

Selected sources

1. FDA, Considerations for the Regulation of Generative AI-Enabled Medical Devices, August 2026, especially Appendix B’s 26 questions. https://www.fda.gov/media/194242/download

2. Regulations.gov, docket FDA-2026-N-7874, public-comment snapshot reviewed for this assessment. https://www.regulations.gov/docket/FDA-2026-N-7874

3. American College of Allergy, Asthma and Immunology, September 25 comments.

4. Newton’s Tree, response to FDA’s generative AI discussion paper.

5. UltraAI, September 28 comments, especially the summary on pages 3–6.

6. U.S. Chamber of Commerce, October 19 comments.

7. American Medical Association, October 16 comments.

8. Karandeep Singh, public summary of University of California Health’s October 19 response.

9. S. Joseph Sirintrapun, personal-capacity comments, September 8.

10. Total Medical Imaging and RADIN, joint comments on radiologist-supervised report predrafting, September 25.

11. Clearstep, response to FDA’s generative AI discussion paper.

12. Personalized Medicine Coalition, October 19 comments.

13. American Hospital Association, October 19 comments.

14. ARPA-H ADVOCATE and Johns Hopkins Applied Physics Laboratory team response.

15. Nina Vasan, Brainstorm at Stanford, August 19 comments.

16. Christman/Luma, response to Question 15 on counterfactual comparators.

17. C. D. Rowsell Jr. and Tanisha Tejada Thoreau, separate comments on runtime controls and agent governance.

18. Radiology Partners/Cognita Imaging, announcement of FDA report-evaluation research contract, September 16. https://www.radpartners.com/2026/09/cognita-imaging-receives-fda-contract-to-test-new-approach-to-evaluating-generative-ai-in-radiology/