How to read AI medicine news: from preprint to patient benefit

Headlines about AI in medicine often blur four different things. A model that "beats doctors" in a preprint, a tool "approved by the FDA," and a system that "saves lives" may be described with the same excitement, but they are very different claims. Here is how we separate them, and what to look for in any story, including ours.

1. Preprint

A preprint is a paper posted publicly before peer review, often on arXiv or medRxiv. Preprints move science quickly, and many good papers start this way. But nobody outside the authors has formally checked the methods. When a story is based on a preprint, it should say so in the first few lines.

Ask: Has this been peer reviewed? Who wrote it, and do they sell the product?

2. Peer-reviewed study

Peer review means other experts examined the paper before a journal published it. That is a meaningful filter, but it does not settle everything. Many studies of medical AI are retrospective: the model is tested on historical scans or records where the answer is already known. Results on past data can look much better than results in a busy clinic with different equipment, patients, and workflows.

Ask: Was the tool tested on patients from hospitals that were not used to build it? Was it compared with clinicians working normally, or under artificial conditions?

3. Regulatory authorization

In the US, the FDA reviews AI-enabled medical devices before they can be marketed, and it publishes a list of AI-enabled devices it has authorized. Many devices are cleared by showing they are substantially equivalent to a device already on the market. Authorization tells you a device can be legally sold for a specific intended use. It does not, on its own, show that patients do better because of it.

Ask: What exactly is the authorized use? Which pathway was used? Is the company describing the tool more broadly than its authorization?

4. Shown to help patients

The question that matters most is whether people are healthier, diagnosed sooner, or treated better when the tool is used. Answering it usually needs a prospective study, ideally randomized, that measures patient outcomes rather than accuracy on a test set. These studies are slower and rarer than the other three kinds of evidence.

Ask: Did the study measure outcomes for patients, or only how often the model agreed with experts?

How we label medical AI stories

Every medicine story on this site names the type of evidence in its sources list: preprint, peer-reviewed study, or regulatory record. When a claim rests only on a company announcement, we say so.