INTRODUCTION
Large Language Models (LLMs) have gained widespread popularity
ChatGPT (OpenAI Inc., San Francisco, US) was released in November 2022
100 million active users within 2 months
1,500+ PubMed indexed articles within the first year
Passing benchmark for the United States Medical Licensing Exam (USMLE) [1]
Performance in Anesthesiology is similar
56.2% on American Board of Anesthesiology (ABA) Board Examination preparation book
Noted variability between subjects: 30.3% on “anesthetic equipment” question and 64.8% on “knowledge recall” questions [2]
“Fine-tuning” existing LLMs and utilizing “document-augmented retrieval” can improve performance on domain-specific knowledge
Chen et al. have demonstrated improvements of 10-15% on USMLE-style questions after fine-tuning LLMs on exam questions, PubMed Abstracts, and clinical guidelines [3]
In this study, we demonstrate a marked improvement in accuracy on a previous regional anesthesia question set from Anesthesia Toolbox [4] using unblinded document-augmented retrieval on articles from OpenAnesthesia

METHODS
ChatGPT from OpenAI was programmatically accessed (12/15/2023 - Python 3.9.13 on Mac OS 12.5) to insert and evaluate input prompts
“Select the single best answer:” was prepended to each of the 36 regional anesthesia question stems to obtain a succinct and unambiguous response
Incorrect answers from the original ChatGPT evaluation were categorized through human anesthesiologist sorting into a 3-6 word subject that encompassed the high-level learning objective of the question
OpenAnesthesia Keyword Articles [5] (if available) for each of the subjects were translated from HTML to text and fed into the ChatGPT “Knowledge” database before being re-evaluated with this new model

DISCUSSION
ChatGPT struggles with niche topics such as regional anesthesia and acute pain
Creating a domain-specific LLM from scratch would require not only substantial computational effort, but also require a corpus of validated input data and text which does not currently exist
Additionally, Beyond board preparations questions, no accepted benchmark exists for model evaluation
Document-augmented retrieval and fine-tuning LLMs may offer an effective methodology to optimize LLM performance on niche topics
Notable limitation of our study is the unblinded nature of document-retrieval on inaccurate questions only
Next steps to scrape the entire compendium of OpenAnesthesia Keyword Articles (approx. 1000)