Introduction: 

As artificial intelligence continues to advance, ChatGPT could emerge as a valuable asset in the educational landscape for medical residents. The aim of this study is to evaluate the effectiveness of ChatGPT as a supplementary tool for pain management fellows preparing for their board exams. 

Methods:

 

Sample online questions were gathered from four widely utilized educational databases Huntoon Pain Medicine Review Book (HPM), Pain Medicine ABA and ABPM sample questions, BoardVitals (BV), and TrueLearn (TL). Chat GPT was given the complete question prompt along with the multiple choice answers. Exclusion criteria included questions containing images. All responses by ChatGPT were qualified using the properties of natural coherence, focusing on logical reasoning and stepwise thinking, internal information from the question stem, and support by appropriate external information. Incorrect responses were further analyzed and categorized based on type of error: logical, informational, or explicit fallacy. The same set of questions were given to five pain management fellows preparing for their board exams to compare to ChatGPT. As the study is devoid of patient identifiable information, it is exempt from IRB review requirements as per Rutgers Health policy.

Results:

A total of 101 questions, 18 HPM, 18 ABA, 5 ABP, 23 BV, and 37 TL were used. ChatGPT performed with an accuracy of 84% and incorporated step-wise logical reasoning, internal information, and external information in 49 (58%), 56 (67%), 83 (98%) of its answers respectively. The average human score for this set of questions was 72%. 

Discussion:

ChatGPT may be an effective resource for pain management fellows preparing for board examinations and augment traditional resident educational methods. It proved to answer board exam questions at the pain fellow level with relatively high accuracy in questions involving simple knowledge recall and complex clinical scenarios. The usage of external information in 98% of questions also demonstrated the effectiveness of the tool, as ChatGPT provided detailed explanations, reasoning, and sources for most of its answers. Furthermore, unlike traditional question banks which simply discuss the correct answer and the incorrect options, ChatGPT also demonstrated how it would analyze and answer a question, using stepwise logic. Using an artificial intelligence system to analyze how to approach questions can also be a valuable tool in preparing for board exams. Future studies could use larger data sets and compare different versions of ChatGPT with other artificial intelligence systems to better determine which version of artificial intelligence would be superior to assist those preparing for their board examinations.