Why Your Next Diagnosis May Be Guided by an A.I. Helper

Dr. Nicholas Gavin, an emergency medicine doctor at Mount Sinai in New York City, was working an overnight shift last summer when a patient came in with a puzzling set of symptoms. Within seconds, his three younger colleagues — two medical students and a resident — were consulting a free artificial-intelligence-powered app for physicians, OpenEvidence.

Dr. Gavin soon learned that they were far from outliers. A third of Mount Sinai’s 9,000 doctors were already regular OpenEvidence users, the health system’s executives found out in a meeting last year with the start-up’s leaders.

“That was an ‘aha’ moment for our leadership,” said Dr. Gavin, who is also the system’s chief clinical innovation officer.

OpenEvidence’s A.I. app, essentially a chatbot for medicine, has become a viral hit with physicians. Talk to a doctor and chances are he or she uses the app to ask specific medical questions or bounce ideas off it in a diagnostic dialogue.

More than half of the nation’s physicians are regular users. Last month, they used it for 30 million questions and consultations, nearly twice the volume from six months earlier, according to the start-up. A separate survey last year of 1,000 physicians found that 45 percent of them used the app, nearly triple the percentage who used ChatGPT, according to Offcall, a career information service for doctors.

That growth propelled the start-up to a $12 billion valuation in January, up from $3.5 billion last July.

But doctors’ quick adoption of the app since its introduction in 2024 — one of a handful of A.I.-enhanced programs on the market seeking to win over physicians — has heightened concerns about how and when the technology should be used in life-or-death situations. In a high-stakes field like medicine, health care systems are navigating thorny matters of patient privacy, safety and trust, as well as the limitations of the technology itself.

“It’s not an oracle, it’s a tool,” said Daniel Nadler, founder and chief executive of OpenEvidence. “Knowledge and knowledge workers still matter.”

The doctor’s office has been a target for computer-assisted decision making for decades, with very limited success until the recent advances of A.I.

The first wave of A.I. in medicine focused on easing the heavy burden of documentation that contributes to physician burnout with transcriptions and summaries of patient visits, called A.I. scribe software. The second wave, which is just getting underway, aims to use A.I. to assist doctors with reliable information and advice to guide diagnosis and treatment while at a patient’s bedside.

The competition has intensified in recent months. UpToDate, a popular legacy electronic reference for doctors, has given its service an A.I. makeover with a chatbot interface. Doximity, an online professional network for physicians, bought an A.I. start-up that mines medical literature and generates summaries. Abridge, a fast-growing A.I. scribe maker, is adding decision-support tools. And last month, OpenAI introduced ChatGPT for Clinicians.

OpenEvidence became a front-runner in part because it exclusively used medical journals and other high-quality research as data to train its A.I. models. Physicians can ask the app specific questions or enter the characteristics and symptoms of a patient and ask for potential explanations. The app is compliant with the federal law that protects patient health information, and physicians are told not to enter any personally identifying information.

OpenEvidence responds with a summary of most likely diagnoses, and then offers other “most important not to miss diagnoses.” Each has links to the research articles that inform the summaries.

“A.I. is solving some of the problems that have long plagued the practice of medicine,” said Dr. Raja-Elie Abdulnour, chief clinical innovation officer at NEJM Group, which publishes The New England Journal of Medicine. “These tools just didn’t exist before, and that’s why people are so excited about them now.”

Yet the early enthusiasm should be tempered with a large dose of caution, medical experts agree. The research so far into the benefits and shortcomings of A.I. in medicine is decidedly mixed.

A.I. has aced standard licensing exams and outperformed human doctors in diagnosing certain cases. But A.I. has also stumbled, failing to accurately summarize research papers or giving wrong answers to diagnostic questions. And it isn’t going to replace humans anytime soon.

“The potential for A.I. is great, but we’re not there yet,” said Dr. Eric Topol, a cardiologist and an executive vice president at Scripps Research in San Diego. “It hasn’t really been tested and demonstrated in the messy, real world of medicine.”

Dr. Topol is a co-author of a recent paper, “The Illusion of Readiness in Health A.I.,” which found “significant competency gaps” in the capability of big A.I. systems when applied to health care.

The evaluations so far have largely focused on the performance of the so-called large language models of big tech companies like OpenAI and Google, which are trained on data across the open internet.

OpenEvidence, founded in 2022, took a more focused approach. It bet that smaller A.I. software models trained on highly specialized data could outperform the giant models in a specific, information-rich field like medicine. The start-up trained its software initially on the publicly available medical data from sources like the government’s National Library of Medicine.

Then the company struck content licensing deals with The New England Journal of Medicine, The Journal of the American Medical Association and other publishers of peer-reviewed medical literature.

OpenEvidence is available to any government-verified physician in America as a free, downloadable app.

“We treated physicians like consumers,” Mr. Nadler said. Users are presented ads, many of them from drug companies, during the five seconds or so they wait for the A.I. to reply. Physicians are served ads on only 5 percent of their questions, the company said.

Sidestepping the traditional gatekeepers of hospital technology departments has raised some issues. OpenEvidence has relied on the workplace behavior known as “shadow A.I.,” workers using such tools without the knowledge or oversight of their employers.

Some health systems are now focusing on bringing OpenEvidence into the institutional fold. Mount Sinai announced in March that it would provide a link to OpenEvidence directly from a patient’s electronic health record.

But the agreement does not give the start-up access to the medical center’s patient data. That integration could come later, Dr. Gavin said, but only after rigorous testing and controls.

Protecting patient privacy and safety will be “paramount,” he said, adding that “we’re not going to just throw a patient’s data over the wall to a private company.”

Doctors in smaller practices across the country, especially in rural areas, say the technology has won them over.

In Corinth, Miss., Dr. Ben Long counts himself as an A.I. skeptic. But he was reassured that OpenEvidence generates answers based on only high-quality, peer-reviewed information.

At first, Dr. Long used it mainly as a reference tool, asking factual questions. But now, he regards the app more as “a consultant, a thought partner” with which he has a dialogue, he said.

“A.I. forces you to think more deeply about your own thinking, challenging your assumptions and why you might be wrong,” Dr. Long said.

A.I. can also let doctors tap expertise that would normally be the realm of specialists.

Dr. Barbara Creighton often diagnoses and treats complex cases at a community hospital in Fairbanks, Alaska. They can involve multiple conditions and failing organs. At a large medical center, a team of specialists might be consulted — an infectious disease expert, a pulmonologist and a gastroenterologist, for example.

Dr. Creighton’s small hospital is not so richly staffed. It does have an arrangement with a big medical center to pay for specialist consulting sessions. She now relies increasingly on OpenEvidence to answer many questions, saving her time and her hospital money.

“It’s like having a bunch of specialists in your pocket,” Dr. Creighton said.

At Mount Sinai, Dr. Gavin said he viewed A.I. technology as a powerful tool to help realize the promise of precision medicine with treatments tailored to individuals.

Progress will require a “patchwork of solutions” from hospitals, medical schools and private companies, he said. Whether OpenEvidence thrives and plays a role in that long-term future remains to be seen.

“But it represents a step in that direction,” Dr. Gavin said.