334 posters, 125 sessions, 831 authors, 236 institutions
ePostersLive by SciGen Technologies S.A. All rights reserved.
78th Annual PostGraduate Assembly in Anesthesiology (PGA78) 2024
December 6 - 9, 2024 | New York City
4027
Sentiment Analysis of Letters of Recommendation for a United States Pain Medicine Fellowship from 2020-2023
Part of Topic
Sentiment Analysis of Letters of Recommendation for a United States Pain Medicine Fellowship from 2020-2023
INTRODUCTION
The pain medicine fellowship is competitive and draws applicants from multiple specialties, particularly from anesthesiology and PM&R. Letters of recommendation (LORs), key in selection, may reflect implicit biases that affect applicant evaluation. This study applies natural language processing (NLP) to analyze LORs for potential bias based on gender, race, specialty, and writer position. By quantifying sentiment in LORs, the study aims to provide programs with insights into how these biases may influence their perception of pain fellowship applicants and inform future candidates on application strategy.
METHODS
This IRB-approved study analyzed 3,488 de-identified letters of recommendation (LORs) for applicants to the Mount Sinai West and Morningside Pain Management Fellowship from 2020 to 2023. Applicant demographics included sex, self-identified race, residency specialty, medical school type, and chief resident status. Each LOR was de-identified using spaCy.
Sentiment was analyzed using VADER and Empath, two NLP tools. VADER assigns a normalized sentiment score from -1 to +1 by considering word valence and intensity, while Empath uses deep learning to score text across 200 sentiment categories. For this
study, 15 categories—achievement, positive emotion, negative
emotion, masculine, feminine, strength, optimism, work,
communication, trust, leader, cheerfulness, valuable, help,
and dominant personality—were selected based on prior research on fellowship applications.
Sentiment scores were compared across demographic groups using Wilcoxon rank-sum tests and ANOVA. Significance for VADER and Empath scores was set at 0.003, applying a Bonferroni correction, while word counts were analyzed at a standard threshold of 0.05. Trends from 2020-2023 were examined for significant sentiments (Figure 1).
RESULTS
From 2020-2023, the pain fellowship applicant pool was skewed male (782 males vs. 181 females). Applicant demographics, shown in Table I, reveal a notable drop in anesthesiology-trained applicants from 144 in 2022 to 98 in 2023. Figure I presents demographic trends in statistically significant sentiment categories. Across sex, race, medical school type, residency specialty, chief resident status, and interview status, some sentiment score disparities were observed, with some remaining significant after Bonferroni correction.
Program directors wrote shorter letters with lower VADER (sentiment) scores than department chairs or other writers. Empath analysis showed fewer “negative emotion” sentiments in department chair letters. Specialty-based analysis revealed lower “achievement” and “negative emotion” scores in PM&R writer letters. Initial significant sentiment differences by sex and race, including higher “strength” and “work” scores for females, lost significance post-Bonferroni, suggesting Type I errors. Anesthesiology applicants’ LORs scored higher in “achievement,” while PM&R scored lower in “negative emotion.” Chief residents’ LORs had lower “negative emotion” and higher “leader” scores, which remained significant post-correction. No differences were observed across medical school types.
CONCLUSION
No significant differences were found in the linguistic content of LORs written for applicants of different genders and races, which is an important finding given the persistence of demographic imbalances among pain physicians. Other causes may instead be responsible for this disparity. Sentiment differences were found for LORs written for applicants from different specialties and with/without chief resident status, and LOR sentiment was also affected by the academic status of the letter writer. These findings highlight a limitation of newly introduced standardized letters of recommendation, as they shift away from valuable emphasis on candidates' personal traits and may not adequately highlight residents' unique attributes. Further research into other characteristics or outcomes of applications such as matched versus unmatched status, ideally with data from multiple institutions, can help better characterize pain medicine fellowship applicants.