Anesthesiologists’ rating of correctness, completeness and coherence of ChatGPT-4 answers to common patient preoperative questions

Daniel Werry FRCPC MSc*1, 2, Garrett Barry FRCPC1, Vishal Uppal FRCA MSc2, Jonathan G. Bailey FRCPC, MSc2

Introduction

o Patients are increasingly using artificial intelligence systems like ChatGPT to answer their questions before surgery.

o The purpose of this study was to evaluate anesthesiologists’ impression of the consistency, correctness, completeness, and coherence of GPT-4 responses to patient questions.

Methods

o Ten frequently asked patient questions were collated from major societal webpages and submitted to ChatGPT-4. Questions and answers were bult into a REDCap survey.

o GPT-4 answers were rated according to correctness, completeness and coherence using a 5-point Likert scale (1 – poor, 2- somewhat good, 3-good, 4-very good, 5-excellent) by anesthesiologists practicing at tertiary hospitals in Nova Scotia, Canada.

o The agreement among anesthesiologist ratings was judged using intraclass correlation averaged across raters (ICC3k).

o Prior to survey creation, consistency of GPT-4 responses was tested by rating 20 submissions each of a subset of questions (4) according to a scoring rubric (inadequate = 0, partial = 1, or complete = 2 for each aspect) and measuring intraclass correlation (ICC) between repeat submissions.

Results

o Surveys rating the correctness, completeness and coherence of GPT-4 responses to 10 patient pre-anesthetic questions were sent to 147 anesthesia faculty and residents. Fifty participants opened the survey and 26 completed it (Table 1).

o Phase 1 (Consistency) results: For all questions, ICC3 values were below the prespecified threshold of 0.75 (Table 2), suggesting inconsistencies among GPT-4 answers to different prompts.

o Phase 2 (Survey) results: Overall median ratings of the correctness, completeness and coherence of each question ranged from 3.4-3.9 (Table 3 and Figure 1), corresponding to average ratings between “good” and “very good”. Overall, ICC3k values were between 0.5- 0.7 for each of correctness, completeness and coherence, suggesting moderate agreement between anesthesiologists.

Discussion

We found that across all domains, on average, anesthesiologists rated the GPT-4 responses to patient questions as good to very good. However, consistency analysis showed that answers may differ with prompts and that anesthesiologists ratings may differ independent of prompts. In conclusion, ChatGPT offers appropriate and moderately consistent answers to anesthesia-related patient questions. Anesthesiologists should be aware that while ChatGPT responses are generally correct, complete and coherent, this does not translate to patient comprehension.