ChatGPT is unable to accurately diagnose pediatric medical cases, according to a study published in JAMA Pediatrics. Conducted by researchers at Cohen Children’s Medical Center in New York, it evaluated the performance of ChatGPT in diagnosing 100 pediatric medical cases published between 2013 and 2023.
The research found that ChatGPT had an accuracy rate of only 17% in diagnosing pediatric medical cases, compared to a 39% accuracy rate in diagnosing general medical cases. The rather low success rate suggests that pediatricians won’t be out of work anytime soon.
Researchers believe that ChatGPT’s poor performance is due to two main factors. The first is the model’s difficulty in recognizing relationships between medical conditions: for example, in one case ChatGPT diagnosed a branchial sulcus cyst, a benign condition, when the correct diagnosis was branchio-oto-renal syndrome, a rare genetic condition that can cause malformations. In another case, the connection between autism and scurvy (vitamin C deficiency) has not been established, a risk factor that should be taken into account at the diagnosis stage.
The second factor is the lack of access to medical data: ChatGPT was trained on a dataset of text and code, which included only a small amount of medical data.
The researchers suggest that ChatGPT could be improved if it were specifically and selectively trained on accurate and reliable medical literature, and it should have complete real-time access to patient data.
Despite these limitations, the researchers believe that AI chatbots could still be a valuable tool in clinical care, particularly for tasks such as booking appointments, answering patient questions, and providing educational information. “These findings underscore the importance of further research to improve the accuracy and effectiveness of AI chatbots for clinical use,” the researchers said.
In the meantime, the world of chatbots and AI tools is now an integral part of our lives and there is a real race to develop increasingly advanced solutions. Google is introducing its new LLM, Gemini AI, underpinning Bard and other proprietary AI services. Speaking of Google Bard, the company has decided to use Reddit to probe what would be the most requested features by users in this 2024 and we can tell you that many of these will arrive with the debut of Gemini.
Meanwhile, OpenAI is firmly in the lead with its GPT-4: this LLM is the basis of paid ChatGPT, but it can also be used within Microsoft’s Copilot, for free.