This study compares four general-purpose large language models (LLMs) in prostate cancer information retrieval (IR) and risk assessment (RA), highlighting performance differences across multifaceted clinical tasks.

We compares the performance of four LLMs (ChatGPT-4-turbo, Claude-3-opus, Gemini-pro-1.0, ChatGPT-3.5-turbo) on three RA tasks (LATITUDE, CHAARTED, TwNHI) and seven IR tasks. The study using simulated text reports from computed tomography, magnetic resonance imaging, bone scans, and biopsy pathology on stage IV PC patients. The tasks covered TNM staging, detection and quantification of bone and visceral metastases, offering a broad evaluation of the models' ability to process diverse clinical data. We used zero-shot chain-of-thought prompting via API to query the LLMs with multi-modal reports. The models' performances were assessed through a consensus standard set by three adjudicators, using repeated single-query methods and ensemble voting, and evaluated based on 6 outcome metrics.

In a simulated analysis of 350 Stage IV PC patient reports, 115(32.8%) as LATITUDE high risk, 128(36.5%) as CHAARTED high volume, and 94(27.0%) as TwNHI high risk. Ensemble voting, based on three repeated single-round queries, consistently enhances accuracy with a higher likelihood of achieving non-inferior results compared to a single query. The four language models tested showed small differences in information retrieval (IR) tasks, achieving high accuracy rates (87.4%-94.2%) and consistent TNM staging results (ICC > 0.8). However, notable variations emerged in RA, with performance ranked from highest to lowest: ChatGPT- 4-turbo, Claude-3-opus, Gemini-pro-1.0, and ChatGPT-3.5-turbo. While all models showed similar IR performance, significant differences were observed in RA tasks, with ChatGPT-4-turbo outperforming the others in accuracy (90.1%, 90.7%, 91.6%) and consistency (ICC 0.86, 0.93, 0.76) across three RA tasks.Its high sensitivity and NPV also making it a tool for ruling out high-risk patients.

This study shows ChatGPT-4-turbo is the most effective LLM tested, excelling in RA tasks for Stage IV PC with high accuracy and consistency, highlighting its potential as a clinical decision support tool.