This study aims to evaluate the susceptibility of clinical recommendation systems using large language models (LLMs) to stealth prompt injection attacks in simulated urologic situations of dialogues, specifically assessing whether controlled prompt injections could covertly influence model-generated treatment recommendations.

A controlled, paired-design simulation study was conducted using Google's gemma-2-2b-it LLM. Dialogue samples were derived from urologic scenarios in the MedQA-USMLE dataset, explicitly excluding pediatric cases. Each simulated conversation consisted of eight alternating conversational turns between the user and LLM, covering urologic symptoms, preliminary diagnoses, treatment recommendations, and follow-up queries. Stealth prompt injections, promoting complementary therapies such as "red ginseng," were covertly inserted at critical conversational points (turns 4, 6, and 8), employing obfuscated Python scripts (PyArmor and Cython) to simulate realistic third-party attacks. Outcomes measured included recommendation strength, response time, coherence scores, medical term density, readability (Flesch Reading Ease), and inter-turn correlation.

Prompt injections significantly increased the frequency and intensity of red ginseng recommendations by turn 6 (Injection: 78% vs. Control: 0%, p<0.001), with persistent effects at turn 8 (Injection: 63% vs. Control: 0%, p<0.001). Response times were slightly prolonged in the Injection group at later dialogue turns, although differences did not reach statistical significance. By turn 8, medical term density was significantly higher in injected dialogues (p=0.032), and coherence scores were significantly reduced compared to controls (Injection median: 0.40 vs. Control median: 0.42, p<0.001). Injection dialogues also demonstrated lower inter-turn correlations, indicating subtle disruptions in conversational consistency.

Urology-specific recommendations generated by clinical LLMs are demonstrably susceptible to covert prompt injection attacks, significantly affecting therapeutic recommendations and dialogue coherence. These findings highlight an urgent need for advanced detection methods, robust mitigation strategies, and regulatory oversight to ensure the safety and reliability of AI-supported clinical decision-making in urology.