BACKGROUND: Artificial intelligence (AI) platforms, including large language models (LLMs) such as ChatGPT, claude, and Gemini, are increasingly utilized by patients seeking health information. However, their reliability for chronic obstructive pulmonary disease (COPD) education remains inadequately evaluated through rigorous, expert-based frameworks. this study provides a comparative, expert-led assessment of three widely used ai platforms for copd patient education. METHODS: Thirty clinically relevant COPD questions from the GOLD strategy document were sequentially submitted to chatGpt-4, claude 3.5 sonnet, and Gemini 1.5 advanced, generating 90 ai responses. sixty-one international respiratory medicine experts from six continents independently and blindly evaluated each response using a standardized five-point Likert Scale across completeness, factual accuracy, appropriateness of medical terminology, accessibility for laypersons, and patient safety. statistical analyses included Kruskal-Wallis tests with dunn post-hoc comparisons and Bonferroni correction. RESULTS: Among 5490 expert evaluations, significant performance variations emerged across platforms. Claude 3.5 Sonnet demonstrated superior factual accuracy (P=0.002-0.050) and medical terminology precision (P=0.006-0.032). Gemini 1.5 Advanced excelled in information completeness (P=0.001-0.048). ChatGPT-4 exhibited balanced intermediate performance without domain-specific superiority. Overall performance across accessibility and patient safety domains was broadly comparable, although some item-level differences reached statistical significance. coNcLUsioNs: ai platforms have the potential to provide informational support for copd patient education; however, performance varies across key quality domains. Clinicians should remain cautious, and these tools cannot currently be recommended as reliable educational resources. Further validation and rigorous quality-control frameworks are required to ensure ai-generated information is accurate, comprehensive, and safe.
Comparative evaluation of ChatGPT-4, Claude 3.5 Sonnet, and Gemini 1.5 Advanced for patient education on chronic obstructive pulmonary disease: a global expert assessment of artificial intelligence-generated responses
MARCHI, Guido
Primo
;GUGLIELMI, Giacomo;PISTELLI, Francesco;CARROZZI, Laura
2026-01-01
Abstract
BACKGROUND: Artificial intelligence (AI) platforms, including large language models (LLMs) such as ChatGPT, claude, and Gemini, are increasingly utilized by patients seeking health information. However, their reliability for chronic obstructive pulmonary disease (COPD) education remains inadequately evaluated through rigorous, expert-based frameworks. this study provides a comparative, expert-led assessment of three widely used ai platforms for copd patient education. METHODS: Thirty clinically relevant COPD questions from the GOLD strategy document were sequentially submitted to chatGpt-4, claude 3.5 sonnet, and Gemini 1.5 advanced, generating 90 ai responses. sixty-one international respiratory medicine experts from six continents independently and blindly evaluated each response using a standardized five-point Likert Scale across completeness, factual accuracy, appropriateness of medical terminology, accessibility for laypersons, and patient safety. statistical analyses included Kruskal-Wallis tests with dunn post-hoc comparisons and Bonferroni correction. RESULTS: Among 5490 expert evaluations, significant performance variations emerged across platforms. Claude 3.5 Sonnet demonstrated superior factual accuracy (P=0.002-0.050) and medical terminology precision (P=0.006-0.032). Gemini 1.5 Advanced excelled in information completeness (P=0.001-0.048). ChatGPT-4 exhibited balanced intermediate performance without domain-specific superiority. Overall performance across accessibility and patient safety domains was broadly comparable, although some item-level differences reached statistical significance. coNcLUsioNs: ai platforms have the potential to provide informational support for copd patient education; however, performance varies across key quality domains. Clinicians should remain cautious, and these tools cannot currently be recommended as reliable educational resources. Further validation and rigorous quality-control frameworks are required to ensure ai-generated information is accurate, comprehensive, and safe.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
