Note ·
Noted on 19 Nov 2026
A consultant dermatologist at Guy's has tested two of the most capable AI models available, GPT-6 Astra and Claude Fable 5.1, against a skin cancer model from 2015. On 200 biopsy-confirmed lesions, the old one picked the right diagnosis 62% of the time. The two new ones managed 42%.
Magnus Lynch used Inception v3, an image network Google published over 10 years ago, fine-tuned for about twenty minutes on public dermatology images. It's small enough to run in a phone browser. He's upfront about the limits: only 200 lesions, 30 of them melanomas, every one hard enough that someone decided to cut it out, and the old network is a baseline, well behind the regulated products built for this job.
What caught my eye was what happened when he asked the big models the way a patient would. Given a single photo and "do I need to see a doctor?", both said yes in almost every case, harmless moles included. As a diagnosis that's useless. As the answer a general model gives a worried member of the public, I think it's about right.
That’s how we're building Julie AI. Its answers come from a curated knowledge base grounded in NHS and NICE guidance rather than whatever the model happens to know, and when something sounds like it needs a professional, it says so and points to a GP, 111 or 999. A lot of the work goes into that decision process, particularly what it shouldn't answer.
So when I read that AI is better or worse than doctors at something, my first question is which kind of AI, doing which job. A narrow model trained to classify one kind of image and a general model built to hold a conversation are different tools, and this test mostly shows what happens when you hand the second one the first one's job.
A question for anyone buying AI tools for a health or care service: when a supplier says "AI", do you know whether you're getting a model built for the task, or a general one with a prompt on top?