Wilms tumor is the third most common solid malignancy in childhood, typically occurring in children younger than five years old. Families increasingly conduct their own independent internet search for information to better understanding a diagnosis. The reliability and quality of this information for patients and families has not previously been formally assessed. Artificial Intelligence (AI) is not new to medicine and health care, but public-facing AI driven search engines and chatbots have become extremely popular since the launch of ChatGPT in November 2022. We aimed to assess the ability of large language model AI chatbots to deliver quality and understandable information on Wilms tumours to patients and their families.

Google trends were used to evaluate the most asked questions related to Wilms tumour. Four AI chatbots (ChatGPT version 3.5, Perplexity, Chat Sonic, and Bing AI) were then used to assess these questions and their responses reviewed. Validated instruments were used to assess the quality (DISCERN instrument from 1 low to 5 high), understandability and actionability (PEMAT, from 0 to 100%), the reading level of the information and whether there was misinformation compared to guidelines (5-point Likert scale).

All AI chat bots provided a high level of patient health information with a median DISCERN score of 4 (IQR 3-5). Additionally, there was little to no misinformation in outputs with a median of 1 (IQR 1-1). The median word count per output from the AIs was 275 (IQR 156 – 322), with an advanced ease of reading level comparable to a high school or college student, median Flesch-Kincaid Readability level of 46.7 (IQR 41.1 – 52.2). The overall PEMAT actionability was poor with a median of 40% (40-65), while the PEMAT understandability of the AI chatbot outputs was high, 83% (IQR 75 – 91.2).

AI chatbots provide generalised, understandable and accurate information regarding Wilms tumour and can be used reliably to help inform patients and families seeking further information. However, much of the information is reliant on medical professionals and not easily actionable by consumers but may act as a guide to help with discussions and understanding treatments.