Utility of Artificial Intelligence in Patient Education: An Appraisal of Mohs Surgery Information Generated by ChatGPT-3.5 and ChatGPT-4

May 2024 | Volume 23 | Issue 5 | e139 | Copyright © May 2024


Published online April 30, 2024

Kennedy H. Sun BSa, William Liakos MDb, Vishal A. Patel MDa

aDepartment of Dermatology, George Washington University School of Medicine and Health Sciences, Washington, DC
bDepartment of Dermatology, Kansas University Medical Center, Kansas City, KS

Abstract

INTRODUCTION

Patients utilize the Internet to pursue health knowledge and better serve as advocates for their care. This frequently involves online searches, where information can often differ from guidelines,1 and increasingly, artificial intelligence (AI) systems such as ChatGPT. Recently, ChatGPT has demonstrated the ability to pass the USMLE,2 potentially reduce dermatologist workload,3 and communicate well with patients regarding common medical questions.4 However, ChatGPT and other AI systems have not been evaluated for the accuracy of their dermatology-related information. We investigated the utility of two of these AI systems, ChatGPT-3.5 (freely accessible, https://chat.openai.com) and ChatGPT-4 (paid premium version of ChatGPT), in providing information on Mohs surgery.

We input 12 common patient questions about Mohs surgery into GPT-3.5 and GPT-4 (Table 1). Questions were based on Mohs surgery pamphlets for patients at multiple dermatologic clinics and outcome measures align with recent literature assessing ChatGPT responses; both were reviewed by a board-certified Mohs surgeon for relevancy prior to survey administration. Answers were evaluated by five dermatologists, including four board-certified Mohs