This study evaluates the performance of real-time, citation-backed Perplexity artificial intelligence (AI) on the Taiwan Urology Certification Examination, assessing its accuracy across different years, urological domains, and difficulty levels, as well as the consistency and comprehensiveness of its responses.

A total of 742 single-choice questions from the 2019–2023 exams were administered to Perplexity AI using Pro Search. The AI rated question difficulty on a 1–6 scale. Two urology residents evaluated correctness, consistency, and comprehensiveness based on the provided explanations. Differences in proportions were analyzed using the chi-square test.

Perplexity AI achieved an average correct response rate of 54.4%, with yearly accuracies of 52.3%, 58.7%, 48.3%, 52.0%, and 60.8%, showing an upward trend. The highest accuracy (68.3%) was observed for basic-level questions. Consistency across all five years was high (88.4%), reflecting the AI’s ability to provide coherent and logical explanations. Accuracy was also higher (61.9%) for questions with comprehensive explanations. Among urological domains, Perplexity AI performed best in Infection, followed by Voiding Dysfunction/Female Urology and Transplantation.

Perplexity AI demonstrated strong consistency and improving accuracy over time. With further training and refinements, it has the potential to serve as a valuable, time-saving learning tool for urology residents.