When prompted with eye disease cases based on the World Health Organizations(WHO) Primary Eye Care Manual (PEC).

Method

Clinical eye disease cases made from the WHO PEC manual domains:Loss of vision, Trauma, Paediatrics, Lumps and lashes and Red eye. 5 cases per domain equalled 25 cases in total. Each case had 3 presenting complaints labelled A, B and C. This allowed for 3 case variations to be made AB, BC and AC. Both platforms were prompted with each case variation three times. Analysis was performed based on the platforms suggested diagnosis, management, and referral outcomes in comparison to gold standard answers from a consultant ophthalmologist. Additionally, readability of the transcripts was determined by performing a Flesch-Kincaid(FK) analysis; an online readability calculator.

Results

• Significant differences between the diagnostic accuracies of the platforms was indicated through statistical testing. 

• Referral suggestions from the correctly diagnosed cases were correct in 39/55 for Alan and 30/35 for ChatGPT-4o mini, with no statistical significance found between referral suggestions. 

• Paired t-tests showed highly statistically significant linguistic differences between platforms, most notably between the lower average reading ease and word count per transcript from Alan in comparison to ChatGPT-4o mini.