Background: Free-text clinical records represent an untapped wealth of data for secondary use, but their potential is limited by operational barriers and resource demands necessary for accurate, large-scale data extraction. We introduce a scalable, resource-efficient, and high-performance pipeline leveraging large language models (LLMs) to address these challenges, using retinal disease as a proof-of-concept.

Methods: We developed a data extraction pipeline for nine retinal disease classes and laterality using 600 real-world ophthalmic clinical letters (dual-specialist annotated) from 600 adult patients seen at Moorfields Eye Hospital, United Kingdom. We performed structured experimentation and iterative prompt refinement using a modular strategy using a single LLM in the development phase. This approach was tested in external validation, and interoperability was examined within the same model family and seven other LLM families. Performance metrics were computed, with micro-averaged F1 scores (with 95% confidence intervals [CIs]) as the primary outcome. An error taxonomy was developed through qualitative analysis and used to categorise all model errors. Pareto frontier analysis was used to establish the optimal configuration for operationalisation, balancing performance, cost, and latency.

Results: Our pipeline achieved strong performance in the development phase, yielding a maximum micro-F1 score of 0.954 (95% CI 0.941-0.967), sensitivity 0.99 (0.98-1.00), specificity 0.99 (0.99-0.99), PPV 0.92 (0.89-0.94), NPV 1.00 (1.00-1.00) with the optimal prompt. Eight model- and data-centric error categories were identified - with errors in domain knowledge and inference being most common - and addressed through prompt refinement. This approach demonstrated strong generalisability (micro-F1 0.945, 95%CI 0.920-0.966) in external validation, extended to two other models in the same LLM family, and largely demonstrated robustness against model choice and deployment constraints in other local model families (for models >10B parameters), with gpt-oss 20b being the top performing local model (micro-F1 0.96, 95% CI 0.95–0.97). Pareto analysis identified gemini-2.5-flash as the optimal configuration for scaling up the pipeline to a 222,204-letter dataset: micro-F1 0.975 (95% CI 0.965-0.984); USD 0.00199/ letter; 7.8s (IQR 6.81-9.20)/letter.

Conclusion: Iterative prompt refinement alone achieved promising performance for data extraction from clinical letters. Beyond this, we propose and conduct a multi-dimensional assessment to map operational trade-offs, enabling the development of a scalable, resource-efficient, and high performing pipeline. A robust approach to operationalisation may help lay the foundation for next-generation data pipelines that can accelerate scientific discovery and power continuous learning health systems.