ChatGPT is a large language model (LLM), which is an artificial intelligence tool. This tool is yet to be validated for routine clinical use. In this study, we evaluate the capabilities of ChatGPT for routine urology practice in our institute.

We assessed the capability of ChatGPT in answering the common queries of our patients. The urologists from our department prepared 50 common clinical questions from different subspecialties of urology. They then graded ChatGPT-generated answers to these questions for accuracy on a 6-point Likert scale (range 1 – completely incorrect to 6 – completely correct). Scores were summarized with descriptive statistics and then compared.

ChatGPT was able to do many of the text-based works like writing consents for routine urological procedures, which could be translated later using Google Translate to Malayalam. These consents were assessed and ChatGPT scored a 4/5 on the Likert scale. This is of value in a language-diverse country like India and aids residents. We also tried LLM in academic activities like preparation of presentations, covering letters, book summaries etc. Across all questions (n=50), median accuracy score was 5 (between almost completely and completely correct) with mean score of 5.2. Median completeness score was 3 (complete and comprehensive) with mean score of 2.8. For questions rated easy, medium, and hard, median accuracy scores were 6, 5.5, and 5 (mean 5.0, 4.7, and 4.6; p=0.05). Accuracy scores for binary and descriptive questions were similar (median 6 vs. 5; mean 5 vs. 4.9; p=0.07). The quality of information was graded based on the section 2 of the DISCERN tool with a median score of 16, corresponding to poor.

ChatGPT generated largely accurate information to diverse medical queries as judged by urologists although with important limitations. It is also helpful in a multi-language country like India for clinical documentation purposes. Further research and model development are needed to correct inaccuracies and for validation.