ChatGPT Poses New Regulatory Questions for FDA, Medical Industry

Large language models popularized by ChatGPT show up in nearly all of FDA Commissioner Robert Califf’s speeches lately, as he makes clear he wants to get ahead of regulating the disruptive technology.

But the “how to” part has perplexed both the US drug and device regulator as well as the companies producing the LLM artificial platform that’s known in pop culture for assisting with tasks from rewriting resumes to offering relationship advice.

Large language models are “going to be transformative and it’s got to be regulated,” Califf said recently at the Biotechnology Innovation Organization convention in Boston. “We’ve met with the officials of the companies that are in the lead, and they want to be regulated. But even they don’t really have good suggestions now on how to do it.”

The Food and Drug Administration has the power to regulate the use of large language models in the products it regulates, FDA attorneys, regulatory experts, and legal scholars told Bloomberg Law. But that authority will largely depend on whether the intended use of these models is to treat or diagnose a condition or disease.

Supervised vs. Generative AI

The FDA isn’t new to regulating artificial intelligence. The agency has approved hundreds of AI or machine learning-enabled medical devices, going back almost three decades. The FDA also regulates software intended for medical purposes under its device regulations and has released an action plan on AI and machine learning.

What is new with LLMs is their ability to translate, predict, and summarize texts. “If you take, for example, the 100 or 150 or so software as a medical device approvals that have gone through the FDA, what’s going on with many of them is a supervised learning algorithm,” Alan Karthikesalingam, research lead for Google Health, said.

“When it comes to large language models, they’re quite different,” Karthikesalingam said. “These models are more flexible, maybe than some of those supervised learning classifiers, in that they can do a number of different things,” such as answering questions about how well certain therapies worked to summarizing papers in more conversational language.

Key Question: What’s the Intended Use?

David Peloquin, a health-care attorney at Ropes & Gray, said, “It seems like some of the principles that are already around in the software, the medical device framework could be applied here to ChatGPT. As we become more familiar with the technology, there might be changes that are made specifically for this. But it strikes me that a lot of it is going to be following on the principles that already exist.”

“It’s the same as if we’re a lawyer using it. What is our intended use, and can we trust the output of it?” Peloquin said.

Jesse Atkins, a senior attorney with Gardner Law who has written about the evolving FDA regulatory landscape of AI, said some of his clients have started to use ChatGPT in a limited capacity, such as rewriting highly technical language on how to use a medical device so it’s at a sixth-grade reading level, for example.

“It’s kind of used at the ideation level, which is to say it’s occurring at a place that lives in time prior to where FDA oversight attaches,” Atkins said.

He doesn’t see ChatGPT in and of itself ever being subject to FDA regulation because that’s not the intended use of the technology.

“ChatGPT’s got its own path, and I just can’t imagine them ever going that way,” Atkins said. “I certainly could imagine a ChatGPT-like product that was specific to clinicians” that would be subject to FDA oversight if a company was selling it for the purpose of providing diagnostic, treatment, or clinical decision support.

Next Stage of FDA and AI

Given that the FDA already approves medical devices using locked artificial intelligence, the next stage will be approval of devices that incorporate learning AI, said Catherine M. Sharkey, a professor of regulatory law and policy at New York University.

Sharkey, who co-authored an article in Duke University’s law journal on AI and the regulatory paradigm shift at the FDA, drew parallels to what’s happening now with ChatGPT and large language models, with the initial rollout in the early 2000s of direct-to-consumer genetic tests like 23andMe.

There were arguments that these genetic tests were a technology or information service that didn’t fall under the FDA’s regulatory thumb. But the FDA decided it did have authority to regulate those tests, which Sharkey said was the right decision.

Likewise the FDA “clearly has authority to regulate” the use of large language models that are incorporated into medical devices or drugs, she said.

“There can be questions like those raised with direct-to-consumer genetic testing, like the boundary line between what’s a medical device and what’s just information that they’re regulating,” Sharkey said. “My own view is that the FDA has rather broad authority.”

Engagement With Industry, Regulatory Science

Before the FDA can determine whether it needs new authorities to regulate large language models, there needs to be a lot of initial engagement between the FDA and those that are doing the work, Susan C. Winckler, chief executive of the Reagan-Udall Foundation for the FDA, said. That interaction needs to include those in the AI space as well as experts in regulatory science, or the science of developing tools, standards, and approaches for how the FDA does its regulatory work, she said.

“Then you get to the question of whether it’s different authority that’s needed,” Winckler said. But it’s really important for any regulating body to understand first what’s happening with the science, and how the application of that science is developing, she said.

Amy Abernethy, president of product development and chief medical officer at Verily, said the regulators, industry, academia, and everyone else involved in this space will need to work together for common definitions and standards. Those definitions will need to cover everything from the documentation around the data sets used to generate the models to how to do that consistently and efficiently, she said.

“Models that update and learn, potentially could veer off from their original tasks and learn in the wrong direction if we’re not keeping track. Are they continuing to improve across time?” Abernethy said. “Models need to be monitored, software needs to be monitored to make sure it’s continuing to perform like expected.”

Until there’s a clear path forward as to what LLM regulation will look like, industry can look to the FDA action plan on AI, take the understanding of device regulations and “we can, as an industry, start showing up with the responsible set of documentation and capabilities that at least start to line up with what’s been asked for, and do our best to set the stage of here’s what’s possible—and here’s what good might look like,” Abernethy said.