INTRODUCTION
Assess feasibility, safety, readability, and patient reported impact of LLM-converted clinic notes (AI notes) on patient understanding and satisfaction.
AIM
•
Prospective single center pilot study at Moffitt Cancer Center.
•
A HIPAA-compliant LLaMA 3.1 70B model prompt engineered and iteratively adapted using synthetic notes to convert clinic notes.
•
Targets: Flesch-Kincaid (FK) readability grade ≤9, Patient Education Materials Assessment Tool (PEMAT) Understandability and Actionability ≥70%, and zero high-risk, ≤2 low-risk errors; noncompliant notes regenerated.
•
Three physician error review; inter rater reliability via intraclass correlation coefficient (ICC).
•
Patient understanding and treatment satisfaction were assessed on 0–10 Likert scales before and after AI note review using survey responses.
•
Scores compared with one-sided paired Wilcoxon signed-rank tests since the metrics and survey outcomes were not normally distributed given small sample size.
•
Follow up patients with plasma cell dyscrasias or myeloid neoplasms. Thirty enrolled; 24 completed surveys. Median age 68.5 years; 67% female; median time since diagnosis 60 months.

 

Results:

•
AI notes (n=30): higher PEMAT Understandability (+23 points) and Actionability (+40 points), both p<0.01, similar readability to original notes (median FK difference −0.19; p=0.9) and were more concise (median −1,543 tokens; p<0.01) (Table 1).
•
Half of the AI notes met safety criteria on first attempt (median 1.5 attempts).
•
Overall low error rate (0.53 ± 0.67 per note) with 27% notes error free on first attempt; high risk errors limited to omissions (27%); no hallucinations.
•
Good physician agreement on error counts (ICC=0.76).
•
Patients reported improved understanding (p=0.01) and satisfaction (p=0.02) (Figure 2).
•
Among those with baseline scores ≤8 (25%), median improvements: +2 understanding (p=.03), +2.5 satisfaction (p=0.06).
•
All patients reported scores ≥8 post AI note review.

 

Conclusion:

LLMs can be leveraged to generate accurate, patient-centric clinic notes that enhance actionability of oncology information and potentially improve patient understanding and satisfaction.
•
Future efforts will aim to scale the approach and reduce errors.