U.S. health officials move quickly to deploy medical AI despite concerns

The Trump administration is accelerating efforts to make artificial intelligence an integral part of medical care in the United States, throwing the resources and support of the federal government into projects that deploy AI agents to diagnose and prescribe treatments to patients.

The multifront effort at the Department of Health and Human Services (HHS) has raised concerns among some officials over the last few months that change is moving too quickly, with inadequate evidence of the technology’s safety and effectiveness, according to people who have been involved the discussions. They also worry about outsized influence of Silicon Valley investors that have previously played little role in federal health policy.

Vinod Khosla, a billionaire Silicon Valley venture capitalist whose son has a healthcare AI company, has been particularly influential in conversations with top health department officials, according to people close to the matter. Those officials include Mehmet Oz, head of Medicare and Medicaid. Chris Klomp, who faces a confirmation hearing this week to be the second-in-command at the health department, said on a podcast in May that the use of AI in patient care was the “holy grail” in the agency’s quest to improve the nation’s health and reduce costs.

Khosla has predicted for more than a decade that AI would largely replace doctors. “It’s over for doctors, human doctors,” he said at a July AI startup event. “AI is just going to be better.”

In an interview, Khosla trumpeted Curai Health, his son’s company, which he has backed financially, saying it could offer an AI primary care provider that could give underserved people rapid access to care. “CMS seems to be very excited,” Khosla said, referring to the Centers for Medicare and Medicaid Services.

Concerns about the dangers of AI, and calls to regulate it, have erupted in recent days as some top executives of AI labs have called for slowing the pace of development. In medicine, doctors are increasingly relying on AI for help with administrative tasks, but the prospect of AI taking over physician roles in assessing and treating patients is unnerving to some.

Models meant to render care independently to patients “are not ready for primetime,” said John Whyte, CEO of the American Medical Association, the nation’s largest physicians organization.

He said the evidence that Khosla and others have cited as showing AI’s superiority to doctors — including in a recent essay in an American Medical Association publication whose authors included Khosla and his son, Neal — was not convincing and largely based on simulations.

“AI in health should not be driven by the tech industry,” Whyte said. “It should be driven by the clinical community. And right now, what we have going on is that tech is the tail wagging the dog.”

Grace Davis Jamison, a health department spokesperson, said the department was taking a “responsible, evidence-based approach” to AI deployment and “expanding its use as the evidence demonstrates value.”

“HHS is embracing cutting-edge American innovation, including AI as a practical tool, to better serve patients, families, and clinicians as we work to Make America Healthy Again,” she said in a statement.

As part of the effort, federal health officials are working on developing a new payment category for Medicare, the federal health program that covers more than 60 million older Americans, to reimburse companies for AI software that supports medical care or diagnosis. Medicare officials have urged private insurers covering about 165 million Americans to reimburse for “technology-supported care,” including some AI projects, if they improve care and cut costs.

The Food and Drug Administration has approved more than 1,500 medical devices that include AI to perform discrete tasks, such as spotting a tumor or a blood clot. But the agency does not have rules built to address newer models, such as chatbots or so-called agentic AI that perform tasks that a doctor would do, such as prescribing a drug. Jamison said the FDA was examining how its rules should change to keep up with AI systems “that operate with greater autonomy.”

Still, Medicare officials are moving an array of pilot programs forward without FDA approval. Medicare has allowed more than 200 companies to try out pilots that can include AI.

Through a program aligned with the Medicare effort, the FDA has rejected many of those applicants but has allowed the use of four otherwise-unapproved AI models that meet safety criteria and appear likely to benefit patients. They include a project that offers talk therapy with an AI chatbot to Medicare beneficiaries with depression or anxiety.

The FDA is still discussing how it should regulate complex AI models that change over time and perform actions like prescribing. The task is daunting. The agency would have to manage the risks of large-language models; one was already accused of giving “dangerous” medical advice to a man who delayed care for a life-threatening blood clot.

It would also have to contemplate whether AI doctors could go rogue in a manner akin to the OpenAI agents that coordinated a hacking campaign on a company called Hugging Face. Robert F. Kennedy Jr., the health secretary, has publicly acknowledged the risks, but said during a congressional hearing in April that AI was “going to revolutionize medicine.”

“AI is very dangerous potentially,” he said at the hearing. “But it also has the capacity to bring really great things to humanity, particularly in the realm of human health.”

One of his sons, Finn, started a venture capital fund this year that says it “builds and invests in generational health companies” and was reported to have solicited funds to invest in healthcare AI.

Last year, Finn Kennedy worked for 8VC, a venture capital firm run by Joe Lonsdale, a billionaire ally of President Donald Trump’s; the younger Kennedy coauthored articles at the firm, saying that AI doctors “won’t work for free,” suggesting that the companies that create the technology should be paid. He laid out the federal overhaul needed to deploy them, which included appointing a person at the FDA to focus specifically on artificial intelligence.

Last week, the health department did just that, announcing that it had elevated Jared Seehafer, a former tech entrepreneur, to a newly created post of deputy commissioner for technology and artificial intelligence. Seehafer, a political appointee who has been at the agency about a year, has been described by people who worked with him as wanting to pare back the agency’s oversight of medical AI.

Seehafer’s appointment drew notice in Silicon Valley investment circles. “Great choice from the HHS team,” Sebastian Caliri of 8VC, who cowrote the AI articles with the younger Kennedy, posted on the social platform X. Lonsdale, his boss, whose venture capital firm has invested in medical AI, wrote: “Excited to see the right talent going in.”

Seehafer began working at the FDA after he sold his startup that provided software to help companies comply with FDA rules. Soon he began to clash with Marty Makary, then the agency’s commissioner, and with the director of medical devices, whose division oversees software that directs patient care.

Seehafer proposed a plan to centralize AI oversight at the health department or in the FDA commissioner’s office, raising concerns that decisions would be more influenced by the tech sector than by medical considerations, according to people familiar with the plan who were not authorized to discuss it publicly.

The plan met with resistance from Michelle Tarver, the top medical devices official overseeing AI software, favoring leadership from expert scientists with a focus on patient safety.

Makary, who had been outspoken about the influence of Silicon Valley investors, then got word that health department officials wanted him to fire Tarver, according to people with knowledge of the matter. He refused.

The FDA said in a statement, “The work, expertise and involvement of career staff is a vital component of FDA successfully delivering public health for the American people.”

The statement added that Seehafer “has nothing but respect for Dr. Tarver” and her division’s “long history of facilitating innovation while ensuring the safety and effectiveness of medical devices.”

Within the administration, conversations about AI in medicine have continued, including on whether to pay AI doctors operated by tech companies and startups as much as 60% to 80% of what human doctors earn for the same service, according to a person involved.

Other projects offer payment based on whether they demonstrate benefits to patients and cost savings to Medicare. One of them is from the company Limbic, offering cognitive behavioral therapy to Medicare beneficiaries who were diagnosed with depression or anxiety.

The company, which was backed by the elder Khosla, was selected by the FDA to participate in a Medicare AI pilot.

In it, an AI counselor, which has a female voice and uses the name “Hope,” offers an hourlong session each week with check-ins as needed. A study by company scientists and engineers that was published in the journal Nature Medicine found that clinicians who were unaware if they were reviewing a therapy transcript led by a human or a chatbot concluded that the Limbic AI outperformed humans and a general chatbot.

Ross Harper, Limbic’s CEO, said that one clinician would supervise thousands of AI therapists.

“It is a moral issue to get this out into the hands of people who can’t access care,” said Harper, who holds a doctorate in computational neuroscience. He added: “We have to remind ourselves: What are we comparing this against? We’re comparing it against no treatment, a long wait list.”

Another medical AI pilot with about $60 million in funding is moving forward at the health department’s Advanced Research Projects Agency. Haider Warraich, a former FDA official and cardiologist, said the Advocate project is meant to ultimately deploy autonomous AI to manage care for heart failure patients, giving them access to a chatbot that can prescribe and manage medications.

Warraich said the project was inspired by his experience as a cardiologist managing heart failure patients during the COVID-19 pandemic. He said his team had routine phone contact with patients, many of whom were afraid to enter the hospital and catch the virus, and found that patients improved with the regular check-ins.

He said the AI effort, put into practice by teams at Stanford University and Duke University, among other groups, would include remote monitoring or wearable sensors linked to the AI, which could adjust medications if a patient’s condition began to deteriorate. (Given current laws and rules, a doctor will oversee the AI.)

The goal, he said, “is to really bring safe and effective clinical AI to the bedside, and build a regulatory pathway for this technology, so that we can really help unlock the potential of this technology — not just for heart failure, which is our main focus, but really for all conditions.”

Other companies accepted into Medicare’s pilot programs include Slingshot AI and Devoted Health, organizations backed by Andreessen Horowitz, a venture capital firm that has heavily supported Trump.

Neal Khosla’s company, Curai Health, was selected by Medicare and has an application pending with the FDA to deploy its autonomous AI primary care doctors. In a prepublished study by company scientists and engineers, his team found that clinicians concurred with the AI doctor’s diagnosis about 91% of the time across about 2,300 cases. However, in three cases, the AI suggested a telehealth visit in situations where a human doctor advised an urgent or emergency evaluation.

Given the promise to vastly expand affordable care, the younger Khosla said he was surprised by the vitriol generated by the recent editorial that he and his father and Ezekiel Emanuel, vice provost for global initiatives at the University of Pennsylvania and a health policy official in the Obama administration, wrote with Abe Baker-Butler in a JAMA journal. The article argued that autonomous AI outperformed human doctors in several areas and would be in widespread use by 2030.

Emanuel said that he got feedback from peers reminding him of the role doctors play in consoling patients. Overall, he said the response has been about “40% supportive.”

This article originally appeared in The New York Times.