4,068 posters, 219 videos, 101 topics, 69 sessions, 14,365 authors, 6,128 institutions

Alzheimer's Association International Conference 2026

July 12-15, 2026 | London, United Kingdom, and Online

7203

Automated Scoring Of A Free Speech Task Using A Large Language Model

View Poster Contact Presenter

Poster Presenter

Natasha Clarke

Authors

Natasha Clarke, Andres Salcedo, James Galvin, Michael Kleiman

Affiliations

University of Miami Miller School of Medicine, Universite de Montreal

Part of Topic

Background

•Free response questions capture rich cognitive information, but scoring open-ended questions objectively is difficult. Manual scoring is subjective and labor-intensive

•Automated linguistic and acoustic speech markers tell only part of the story: they miss content and are vulnerable to confounds (age, sex, education)

•Open-ended questions are easy for patients to answer

Objective

•Test whether targeted LLM prompts can score open-ended responses and detect MCI

•Determine whether a focused, few-item content score is robust to demographic matching and response length, where some linguistic features are not

Methods

Participants: Cognitive-aging research cohort (65+) with clinical diagnosis: 275 cognitively normal (CN), 91 MCI, 36 impaired-but-not-MCI (127 impaired cases)

Procedure: Each participant answered the question “Describe your typical morning routine”. Targeted LLM prompts scored specific content dimensions using Claude Sonnet 4.6. 30 questions were generated targeted 9 categories:

executive, genericity, lexical, temporal, density, scaffolding, monitoring,
discourse, and affective

Analysis: Discrimination measured by AUC, sensitivity, specificity in a 1:1 matched cohort on age, sex, and education. Robustness tested by adjusting for word count and comparing content vs linguistic/acoustic features

Results

• 5 question composite (“Rn5”) reached AUC = 0.676 for CN-vs-CI and survived word-count adjustment (p=.02); the 30-item score did not (p=.07)

•Both content scores added information over demographics (LR p <.001) but not over MoCA (LR p=.19)

•When excluding Impaired-not-MCI, linguistic/acoustic features collapsed toward chance (AUC .652 → .573) while the content score held (AUC .647 → .634)

Discussion

•Targeted LLM prompts make objective, scalable content scoring of a single open-ended answer feasible

•Carries signal that linguistic/acoustic speech features miss and survives correction

•Limitations: single exploratory cohort, item selection partly in-sample, a small heterogenous impaired-not-MCI subgroup, and moderate AUCs with wide confidence intervals

Comments

Email is required but is not shown publicly.

Sort comments

Loading comments...