Title: An automated animal fluency scorer: development and validation across languages

Introduction: Animal fluency tasks are core to dementia screening, diagnosis, and monitoring. Valid responses reveal deficits in Alzheimer’s disease (AD) and behavioral variant frontotemporal dementia (bvFTD), tracking decline and predicting clinical progression. However, standard scoring relies on inconsistent criteria and fallible manual work from experts unequally available across regions, compromising reliability and scalability. Here, we validate TELL’s digital fluency scorer, an AI tool automating end-to-end acquisition, preprocessing, scoring, and reporting.

Methods: We evaluated TELL’s scorer using animal fluency data from four datasets spanning tabulation-based and audio-based workflows. Tabulation-based analyses leveraged expert-tabulated English responses from 625 DementiaBank participants (58 AD patients, 567 healthy controls) and Spanish responses from an independent South American cohort of 182 participants (77 AD patients, 34 bvFTD patients, 71 healthy controls). Audio-based analyses used English recordings from 614 DementiaBank participants (57 AD patients, 557 healthy controls) and Spanish recordings from 2,324 ReDLat participants across five Latin American countries (943 AD patients, 171 bvFTD patients, 1,210 healthy controls). Tabulated responses were cleaned to remove scoring cues, and audio recordings were automatically transcribed. A locally deployed lightweight LLM (Gemma 4 E4B) first isolated the animal fluency task when necessary and identified candidate animal responses. For each candidate, the system retrieved close matches from a Wikidata-derived animal dictionary using hybrid lexical-semantic retrieval, and the same LLM then applied dataset-specific scoring criteria to classify the response as valid, a repetition, or an intrusion. Agreement with expert scores was assessed using intraclass correlation coefficients [ICC(A,1)], while absolute scoring error was quantified using mean absolute error (MAE). Country-level analyses were further performed for both Spanish datasets.

Results: TELL’s scorer showed excellent agreement with expert scores across English tabulation-based [ICC(A,1) = 0.984; MAE = 0.41], Spanish tabulation-based [ICC(A,1) = 0.988; MAE = 0.52], English audio-based [ICC(A,1) = 0.949; MAE = 1.12], and Spanish audio-based analyses [ICC(A,1) = 0.923; MAE = 1.49]. Within the Spanish datasets, country-level agreement remained strong and absolute error low for both tabulation-based [ICC(A,1) = 0.986 - 0.993; MAE = 0.30 - 0.62] and audio-based analyses [ICC(A,1) = 0.881 - 0.961; MAE = 1.22 - 1.63].

Conclusions: TELL’s fluency scorer consistently matched expert scoring across tabulation-based and audio-based workflows. Validation in English and Spanish covers two of the world’s most widely spoken languages, both globally and among people living with dementia. These findings show how LLMs can advance efficiency, scalability, and equity in digital health.