Skip to content

Note ·

Noted on 19 Nov 2026

A consultant dermatologist at Guy's has tested two of the most capable AI models available, GPT-6 Astra and Claude Fable 5.1, against a skin cancer model from 2015. On 200 biopsy-confirmed lesions, the old one picked the right diagnosis 62% of the time. The two new ones managed 42%.

Magnus Lynch used Inception v3, an image network Google published over 10 years ago, fine-tuned for about twenty minutes on public dermatology images. It's small enough to run in a phone browser. He's upfront about the limits: only 200 lesions, 30 of them melanomas, every one hard enough that someone decided to cut it out, and the old network is a baseline, well behind the regulated products built for this job.

What caught my eye was what happened when he asked the big models the way a patient would. Given a single photo and "do I need to see a doctor?", both said yes in almost every case, harmless moles included. As a diagnosis that's useless. As the answer a general model gives a worried member of the public, I think it's about right.

That’s how we're building Julie AI. Its answers come from a curated knowledge base grounded in NHS and NICE guidance rather than whatever the model happens to know, and when something sounds like it needs a professional, it says so and points to a GP, 111 or 999. A lot of the work goes into that decision process, particularly what it shouldn't answer.

So when I read that AI is better or worse than doctors at something, my first question is which kind of AI, doing which job. A narrow model trained to classify one kind of image and a general model built to hold a conversation are different tools, and this test mostly shows what happens when you hand the second one the first one's job.

A question for anyone buying AI tools for a health or care service: when a supplier says "AI", do you know whether you're getting a model built for the task, or a general one with a prompt on top?

Reply onLinkedIn

More notes

See all →

This week police forces and the Motor Insurers' Bureau are running Operation Drive Insured, their annual week of checks for uninsured drivers. MIB reckons about 335,000 people are driving uninsured on UK roads on any given day, and last year's week ended with 4,654 vehicles seized.

Sadiq Khan wants the power to cap how many robotaxis can run in London. He told the Guardian earlier this month that "London isn't San Francisco", and pointed out that there's no cap on private hire vehicles or taxis either, which City Hall has also lobbied for.