Title: Feasibility and Preliminary Validation of a Voice Agent-Administered Cognitive Screening Protocol in Healthy and Impaired Older Adults

Authors: Stephanie Ruth Young, MEd, PhD¹; Justin Mason, MBA². ¹Northwestern University Feinberg School of Medicine; ²CaringAI Inc.

Background: Cognitive impairment is frequently under-detected in primary care, and diagnostic delays disproportionately affect older Black Americans, who experience roughly twice the dementia rate of their White peers. Routine screening is constrained by limited time, training, and staffing. AI voice agents offer a scalable, structured, and familiar means of phone-based screening for older adults. We evaluated the scoring accuracy and triage recommendations of a telephone-based AI screening agent (CaringAI) through expert clinician audit, comparing in-clinic versus scheduled outbound call modes.

Methods: Sixty-eight older adults (≥60 years) were recruited from a family practice within an accountable care organization; the sample was predominantly Black/African American and Medicare/Medicaid-insured. The CaringAI Listen agent administered five instruments by phone — the 6-Item Cognitive Impairment Test (6CIT), a supplementary cognitive set, the Lawton IADL scale, GAD-2, and GDS-4 — during in-clinic calls at intake versus scheduled outbound calls approximately 14 days before the visit. A Certified Dementia Practitioner then confirmed or overrode each AI-generated score and triage recommendation. Agreement was assessed using ICC(2,1) for scores and Cohen's κ for triage decisions, with bootstrap 95% confidence intervals.

Results: AI–reviewer score agreement was good to excellent across instruments (ICC 0.84–1.00). The reviewer accepted 79.4% of the AI's triage recommendations overall (κ = 0.72). Acceptance was 86.7% for scheduled outbound calls (κ = 0.83) and 73.7% for in-clinic calls (κ = 0.56); the between-mode difference was not statistically significant (Δκ = 0.28; 95% CI −0.00 to 0.53). The 6CIT showed a tendency toward over-referral.

Conclusions: A Certified Dementia Practitioner accepted approximately 79% of the AI agent's triage recommendations, supporting the feasibility of telephone-based AI cognitive screening in primary care when paired with structured clinician oversight. Feasibility was demonstrated in a real-world, historically underrepresented sample of predominantly Black/African American older adults. 6CIT over-referral is a priority for model refinement. Limitations include a single site (n = 68), a single unblinded reviewer (raising the possibility of anchoring), and non-randomized deployment modes. Larger, multi-site, reviewer-blinded studies are warranted.