Large language model (LLM) chatbots have rapidly gained popularity in the past 2 years. Significant efforts have been undertaken the evaluate the ability of these LLM’s to provide accurate information to patient about a range of medical conditions. We aim to determine the accuracy and readability of information generated for patients by freely available LLMs regarding kidney cancer

The top 10 queries related to kidney cancer worldwide were identified via Google Trends. These queries were posed to 5 freely available LLMs – ChatGPT, Google Gemini, xAI, Perplexity AI and Microsoft Copilot. 2 independent reviewers assessed the LLM replies for accuracy in five categories, employing a 5-point Likert scale. A readability evaluation was conducted utilizing established formulas, including the Flesch reading ease score and Flesch-Kincaid Grade Formula. LLM replies that were graded at college level or above were requested to be simplified and the revised response was then recorded. Responses were compared to patient pamphlets provided by the EAU and AUA.

Responses on average required a higher reading level than the average literacy level of the general public. The average Flesch reading ease score was 40.8, 29.4, 32.1, 28.3 and 29.3 for ChatGPT, Gemini, xAI, Perplexity AI and Copilot respectively. These scores indicate a college level of reading required. When prompted to simplify their responses, reading ease score increased on average by 25.3, 21.3, 14.7, 25.2 and 15.4 points respectively. The LLMs outputs were overall rated as highly accurate. No grossly inaccurate or dangerous statements were identified. Some responses contained partial inaccuracies and minor detail omission.

LLM show promise as an adjunctive tool for patient education, however clinicians need to be wary of the possibility for partial inaccuracies and incomplete responses, and should warn their patients of the same. The readability of responses was above the average literacy level however all LLM show the capacity to provide more readable responses when prompted.