49th Annual Regional Anesthesiology and Acute Pain Medicine Meeting

Large Language Models in Medical Literature Review: Automating the Analysis of Erector Spinae Plane Block Studies

Background:

•Literature is growing fast, with over 1.4M citations added to MEDLINE in 2022 alone.

•Large language models (LLMs), such as ChatGPT, are demonstrating improving performance in processing vast amounts of texts and question-and-answering.

•How well can ChatGPT conduct automated systematic review of Erector Spinae Plane block (ESPB) literature relative to traditional manual reviews?

Methods:

1. Manual review of ESPB literature published between August 1, 2018 and December 31, 2022 (gold standard answers)
2. Automated review of the same literature was done via advanced text-embedding techniques in ChatGPT
3. Comparison of ChatGPT outputs to the gold standard answers

Results:

•ChatGPT generally underperformed in conducting systematic literature review of ESPB literature, exhibiting moderate accuracy to notable discrepancies from gold standard answers.

•It performed well on answering factual questions at the article level and poorly on questions requiring aggregating studies.

Discussion:

•The study highlights the limitations of current ChatGPT in performing literature review on a clinical subspecialty topic.

•The study uniquely uses a manual review as benchmark for the LLM performance and focuses on a subspecialty topic (ESPB).

•Future directions include developing more holistic LLM evaluation metrics tailored to clinical medicine and evaluating different LLMs for biomedical literature review.

•We emphasize the important role of physicians in applying and validating LLM capabilities in clinical use cases.

References:

1.Shbeer A: Regional Anesthesia (2012–2021): A Comprehensive Examination Based on Bibliometric Analyses of Hotpots, Knowledge Structure and Intellectual Dynamics. J Pain Res 2022; 15:2337–50

2.Mao R, Chen G, Zhang X, Guerin F, Cambria E: GPTEval: A Survey on Assessments of ChatGPT and GPT-4 2023 at <http://arxiv.org/abs/2308.12488>

3.Cao Y, Li S, Liu Y, Yan Z, Dai Y, Yu PS, Sun L: A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT 2023 doi:10.48550/ARXIV.2303.04226

4.Kung TH, Cheatham M, Medenilla A, Sillos C, Leon LD, Elepaño C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V: Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2023; 2:e0000198

5.Ali R, Tang OY, Connolly ID, Zadnik Sullivan PL, Shin JH, Fridley JS, Asaad WF, Cielo D, Oyelese AA, Doberstein CE, Gokaslan ZL, Telfeian AE: Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations. Neurosurgery 2023; 93:1353–65

49th Annual Regional Anesthesiology and Acute Pain Medicine Meeting

LARGE LANGUAGE MODELS IN MEDICAL LITERATURE REVIEW: AUTOMATING THE ANALYSIS OF ERECTOR SPINAE PLANE BLOCK STUDIES

Poster Presenter

Authors

Affiliations

Part of Topic