Introduction:

Since its initial conception in 1956, artificial intelligence (AI) has exponentially evolved¹. Large language models (LLMs) have become particularly popular, with one such device entitled Chat Generative Pre-trained Transformer (ChatGPT) quickly gaining 100 million users in the first two months following its launch². As it pertains to medicine, these LLMs may offer enormous potential, albeit with limitations.

Many LLMs are easy to both use and access at low costs. LLMs are capable of quickly integrating vast amounts of open access information and are able to promote personalized learning experiences for users³. Focusing within pain medicine, these LLMs may thus represent an exciting tool to be used in creating individualized learning experiences to teach trainees how to navigate complex clinical situations or to solidify their understandings of pain management. However, LLMs are not without their limitations, as they offer the potential to ‘hallucinate’, increase educational inequalities, and perpetuate bias, cheating, and plagiarism^4,5.

Currently, there is a paucity of literature examining the use of LLMs within pain medicine education. In this limited pilot study, we investigated the ability of LLMs to assist in pain medicine trainee education.

 

Materials/Methods:

The pilot project was approved by UPMC Quality Improvement Review Committee (#4547). At the time of the initiation of the pilot study, three of the most commonly utilized LLMs – ChatGPT Plus, Google Bard, and Bing AI – were selected to be asked a series of multiple choice questions (MCQ) to evaluate their ability to assist in learner education within pain medicine. Similar to ChatGPT Plus, BingAI utilized Open AI’s GPT-4 application programming interface (API), although it also incorporates Microsoft’s Prometheus model. Conversely, Google Bard was included as a third LLM that incorporated an entirely different API (Google Bard API). 

A list of 17 MCQ with five answers each were compiled in the style of pain medicine board questions by the authors. Correct answers were confirmed by two physicians board-certified in pain medicine. The questions included topics such as anatomy, neuroimaging, clinical diagnosis, patient management, medication mechanisms and complications. All 17 of these questions were input into ChatGPT Plus, Google Bard, and Bing AI with instructions to identify the correct answer. The LLMs were not primed with any information prior to the questions. 

As the case report is devoid of patient identifiable information, it is exempt from IRB review requirements as per UPMC policy.

 

Results:

Out of the 17 questions, 9 assessed basic fact recall (first-order questions), 5 required an integration of information (second-order questions), and 3 required multiple layers of analysis/critical thinking (third-order questions). 

As shown in Table 1, ChatGPT Plus answered the highest percentage of all questions correctly (16/17). Lowest correctness scores by LLMs were in answering first-order questions, with Google Bard and Bing AI answering 4/9 and 3/9 first-order questions correctly, respectively.

Zero LLMs consistently referenced existing literature in their answers. All LLMs frequently provided explanations for their selected MCQ answer, while only ChatGPT Plus consistently explained why the other MCQ answers were incorrect. Qualitative evaluation of these LLM-provided explanations in answering second- and third-order questions revealed some reasoning inconsistencies (e.g., providing flawed information in selecting the correct answer).

 

Discussion:

ChatGPT Plus answered the highest percentage of first-, second-, and third-order questions correctly. Despite utilizing a similar API, ChatGPT Plus answered a greater number of first- and second-order questions correctly as compared to Bing AI.

Poor answer accuracy by Google Bard and Bing AI in first-order questions suggests potential limitations in the information that LLMs were trained upon at the timing of this trial, and has implications for teaching trainees basic pain medicine concepts. However, this is challenged by the impressive accuracy of ChatGPT Plus in answering these questions, and the fact that both ChatGPT Plus and Bing AI utilize the same GPT-4 programming.

Interestingly, the LLMs performed better with higher-level questions, implying educational utility in critical thinking scenarios. However, there were reasoning inconsistencies or lack of explanations provided in choosing the correct answers (Table 2). Only ChatGPT Plus consistently provided explanations indicating why the other MCQ answers were incorrect. Given that these questions are designed to test pain medicine comprehension, developing an understanding of exactly why an answer is correct or incorrect is of critical importance in learning. Limited ad hoc testing demonstrated that these LLMs did provide additional answer choice explanations when asked to in the prompt. Thus, training learners to engage the LLMs with tailored prompts may play a pivotal role in maximizing the application of AI in medical education.

Similarly, none of the LLMs consistently referenced literature in their answers, which again may have resulted from the nature of our prompts. To analyze this, we performed another brief trial whereby we proposed the same questions and requested a citation. In these few questions, the citations were from websites and national organizations instead of medical texts. This may be due to the lack of LLM access to private domain information and indicates a limitation in the ability to educate trainees on salient or novel medical literature.

Our abbreviated pilot study had several limitations, mainly pertaining to the small sample size of unvalidated MCQ, potential for unintentionally training LLMs in the process of asking these questions, limited number of LLMs studied, and lack of medical trainee evaluation. Furthermore, since the conception of this pilot study, each LLM has undergone revisions, including the renaming of Bing AI as Copilot (Copilot and Copilot Pro versions) and Google Bard as Gemini (Gemini Nano, Pro, and Ultra models). Throughout this abstract we only reference Google Bard and Bing AI to most accurately describe the models that we tested in our pilot experiment.

In conclusion, LLMs represent a continually evolving and promising modality to assist trainees pursuing a career in pain medicine. Still, limitations currently exist that may hinder their independent use in this setting. Future research exploring how LLMs may overcome these challenges is thus required. Until then, LLMs should be utilized as supplementary tool within pain medicine trainee education and with caution.