Comparing Provider and Artificial Intelligence Chatbot Responses to Breast Reconstruction Patient Questions

Daniel Soroudi, B.S. 1, Aileen Gozali, M.M. 1, Nisha Parmeshwar, M.D. 2, Jacquelyn Knox, M.D. 2, Ryan Sadjadi, M.P.H. 1, Scott L. Hansen, M.D. 2, Merisa Piper, M.D. 2

1University of California San Francisco, School of Medicine,

2University of California San Francisco, Department of Surgery, Division of Plastic and Reconstructive Surgery

• Patients express dissatisfaction with information

received pre-operatively

• Patient-directed Electronic Health Record (EHR)

messaging burdens the physician.

• Tools like ChatGPT can revolutionize patient

communication.

• This study assesses ChatGPT's effectiveness in

answering breast reconstruction queries.

• Questions presented to ChatGPT-3, ChatGPT-4,

plastic surgeons, and advanced practice

providers (APPs)

• Readability via the Flesch Reading Ease

• Accuracy and Empathy via 1-5 Likert scale

• Flesch Reading Ease scores for providers w ere better than AI responses (53.33±13.27 vs. 35.97±11.62, p<.001).

• Empathy scores for providers were lower than AI responses (2.013±0.74 vs. 2.875±0.75, p<.001).

• Accuracy scores were not different between providers and AI responses (4.3 ± 0.9 vs. 4.5 ± 0.6, p=0.170).

• Word Count for providers was lower than AI responses (83.1 ± 71.9 vs. 225.5 ± 175.6, p<0.001)

• ChatGPT has potential to enhance

patient-provider communication

• Strong in empathy, accuracy, and

efficiency of response generation

• Weak in adhering to readability

guidelines

• Equitable access to AI technology

remains to be addressed

• Future studies should focus on:

- Optimization of AI tools

- Exploring the effectiveness of

prompting techniques

- Exploring synergistic effects of

AI-provider responses