There’s been a lot of press about OpenAI’s ChatGPT but I’ve been particularly interested in the chatter about its use in the medical field. I’ve been having some fun with it (see the last post about donuts) but haven’t asked it any medical questions. Notably, the AI passed all 3 steps of the USMLE test, implying it’s absorbed all the medical knowledge needed from med school and internship. I’m not really surprised by that. By the time you take those tests, you should know what you need to know. The scientific knowledge needed to be a doctor is not a secret and I don’t think the test would be too hard if you had the entire Internet in your brain. There’s also a lot of talk of using ChatGPT or a similar AI to do all the tedious staging and scoring worksheets used in medical practice.
But there was one interaction that gave me pause. I have been listening to Nickel Creek a lot because they just released a new album. I knew the band started when they were kids, so I asked the AI for a biography of the band before 2000. Part of the answer was:
"They released their first album, "Little Cowpoke," in 1993, when Sean Watkins and Thile were just 12 years old and Sara Watkins was 11."
I knew this was wrong because Chris Thile and Sara are around the same age and Sean is a few years older. So I told it that I didn’t think their ages in 1993 were correct. It responded:
You are correct, I apologize for the error. Nickel Creek's first album, "Little Cowpoke," was actually released in 1993 when Sean Watkins was 15, Sara Watkins was 12, and Chris Thile was 12. Thank you for bringing this to my attention.
Obviously there’s a lot I don’t understand about artificial intelligence, but the ages of these people are on Wikipedia and calculating how old they were in 1993 should be fairly trivial. Then when I point it out, the AI tries a little harder and gives me the correct information. Why wasn’t it correct the first time?
I’m certain that AI is going to become indispensable to medical practice in the coming years but I would be wary about using the free ChatGPT for real-world decisions right now. From my understanding, I’m just playing with the 2021 version. GPT-4 is supposed to be better and is available for paying customers. I’m excited to see what the future holds.
