IMPLEMENTATION, USE, AND PERCEIVED IMPACT OF ARTIFICIAL INTELLIGENCE TOOLS FOR CHEST DIAGNOSTICS: QUALITATIVE ANALYSIS OF 10 NHS ORGANISATIONS IN ENGLAND

Ramsay et al

BACKGROUND

•AI for chest diagnostics may support

oFaster, more accurate diagnoses

oReduced pressure, reduced healthcare costs

•Limited evidence on implementation – experiences, effectiveness, costs

•June 2023: AIDF - NHS England programme to pilot AI in chest imaging

oNetworks of hospitals bid to be pilot sites; different models/approaches proposed

oAI only used as a support – treatment decisions always made by clinicians

•NIHR RSET commissioned to independently evaluate AIDF implementation

 

RESEARCH QUESTIONS

•How  were AI tools implemented and used?

•How did staff and patients experience and view AI tool use?

•What were the perceived impacts of AI tool use?

 

METHODS

•Part of larger mixed-method rapid evaluation (incl quantitative & health economic)

•Interviews, non-participant observations, documentary analysis (Jun 2025-Jan 2026)

•5 researchers collected and analysed data, working with wider team, clinicians, & PPIE

•RAP sheets for formative feedback; in-depth coding guided by Major System Change framework (Fulop et al, 2016)

 

DATA

•Trusts sampled: 3 in depth; 7 light touch (1 withdrew)

•Interviews:

oNational (incl AI suppliers): 8

oStaff: 38 (incl 9 follow-ups)

oPatients: 10

•Non-participant observations: 6

•Documents: 98

 

RESULTS: AI tool implementation & usage

•Sites used tools from four different AI suppliers

•Nine sites used AI to analyse x-rays, whilst one used it to analyse CT scans

•All sites used a ‘decision support’ function. Seven also used their AI tool for prioritisation.

•Six sites had changed their reporting workflow based on AI prioritisation

•In five sites, only radiology staff had access to the AI tool (vs. availability to all staff with PACS access)

•In eight sites, AI tools were used on scans from all referral routes, but active functions could vary by referral route

•Eligibility was primarily determined by age. Nine sites only used AI for patients aged 16/18+

•Four sites informed patients about the use of AI (e.g. outlined in SOP, advertised on posters)

 

RESULTS: implementation facilitators

•Engagement between suppliers & trusts – open communication and feedback loops

•Regular meetings & clear communication about processes & timelines

•Clinical leadership & dedicated project management 

•Sharing learning across trusts/networks

•Clear and effective training

•Openness to innovation

•Integrating AI and prioritisation into existing systems: planning/groundwork pre-implementation (e.g. shadow mode/ testing phase)

•Regular network meetings – community of practice for shared learning

 

RESULTS: implementation barriers

•Gaps in leadership to manage AI implementation (need balance of clinical and technical knowledge)

•Complex clinical governance & safety processes

•Managing AI implementation alongside clinical roles

•Varied engagement with AI training

•Staff felt they had insufficient knowledge

•Lack of national guidance, e.g. from MHRA, RCR & NHSE

•Integrating secondary capture and AI tool into PACS

•Integrating prioritisation into PACS, collecting comprehensive data to evaluate impact

•Varying AI literacy and resource/capacity across trusts

 

RESULTS: Staff and patient experience

•Staff had positive experiences of AI, regardless of how implemented

•felt AI could benefit less experienced reporters 

•Despite patients wanting to be told about use of AI, few aware of AI use in their care 

•Patients positive overall about use of AI, due to 

otrust in NHS generally & trust in NHS staff using the AI tools

ohope that abnormal cases could be processed more quickly

•Staff & patient concerns: technical performance, overcalling of abnormal scans, overreliance of staff, governance, & accountability

•Future use: staff & patients felt AI use would & should continue but wanted 

oongoing monitoring

ofurther evidence of effectiveness

 

RESULTS: Perceived impact

•Confidence/ overreliance

oLittle impact of AI tools on personal reading accuracy

oBut reassurance from AI meant fewer checks with colleagues were required

oConversely, some staff concerned that colleagues might over-rely on AI outputs

•Patient outcomes

oToo early to state impact on patient outcomes - lack of data

oNoted even small improvements may have clinically significant effects on patients. 

oHowever, poor aspects of tool performance (e.g. NG tube placement) were noted as posing risks to clinical outcomes. 

oSome sites evaluating AI-reported false negatives (NB all scans still checked by humans)

•Equality, diversity and inclusion

oFew concerns raised: broad view that AI processing of scans not affected by demographics 

oHowever, staff noted limited local data/capacity to evaluate this

 

IMPLICATIONS

•Implementation

oAI has potential to be implemented at scale, but requires

oResources, time, & collaboration between NHS staff & AI suppliers

oEarly/ongoing stakeholder engagement (incl staff, AI suppliers, patients)

oNeed to ensure that AI tools address challenges faced by services

oMandatory training may build staff confidence & appropriate AI tool usage

•Staff and patient views generally positive, but wanted

omore evidence of AI effectiveness

oongoing monitoring of AI performance

oEspecially if autonomous AI reporting to be introduced

oPatients would value being informed of AI use, e.g. posters, leaflets, appointments

 

STRENGTHS & LIMITATIONS

•Strengths: 

oOne of the first real world evaluations of AI tools for diagnostics in the UK

oCovered range of approaches to AI use/implementation

oIn-depth data on implementation, staff & patient experience, & perceived effectiveness

•Limitations:

oOnly conducted in 10 NHS Trusts: findings may not be generalised to all trusts

oOnly interviewed ten patients with limited diversity: findings an indication of patient views

oMore observations of different activities/settings would strengthen analysis of AI use & oversight

oUnable to evaluate long-term perceived impacts

POSTER TEXT

BACKGROUND

·       AI for chest diagnostics may support

o   Faster, more accurate diagnoses

o   Reduced pressure, reduced healthcare costs

·       Limited evidence on implementation – experiences, effectiveness, costs

·       June 2023: AIDF - NHS England programme to pilot AI in chest imaging

o   Networks of hospitals bid to be pilot sites; different models/approaches proposed

o   AI only used as a support – treatment decisions always made by clinicians

·       NIHR RSET commissioned to independently evaluate AIDF implementation

 

RESEARCH QUESTIONS

·       How  were AI tools implemented and used?

·       How did staff and patients experience and view AI tool use?

·       What were the perceived impacts of AI tool use?

 

METHODS

·       Part of larger mixed-method rapid evaluation (incl quantitative & health economic)

·       Interviews, non-participant observations, documentary analysis (Jun 2025-Jan 2026)

·       5 researchers collected and analysed data, working with wider team, clinicians, & PPIE

·       RAP sheets for formative feedback; in-depth coding guided by Major System Change framework (Fulop et al, 2016)

 

DATA

·       Trusts sampled: 3 in depth; 7 light touch (1 withdrew)

·       Interviews:       

o   National (incl AI suppliers): 8

o   Staff: 38 (incl 9 follow-ups)

o   Patients: 10

·       Non-participant observations: 6

·       Documents: 98

 

RESULTS: AI tool implementation & usage

·       Sites used tools from four different AI suppliers

·       Nine sites used AI to analyse x-rays, whilst one used it to analyse CT scans

·       All sites used a ‘decision support’ function. Seven also used their AI tool for prioritisation.

·       Six sites had changed their reporting workflow based on AI prioritisation

·       In five sites, only radiology staff had access to the AI tool (vs. availability to all staff with PACS access)

·       In eight sites, AI tools were used on scans from all referral routes, but active functions could vary by referral route

·       Eligibility was primarily determined by age. Nine sites only used AI for patients aged 16/18+

·       Four sites informed patients about the use of AI (e.g. outlined in SOP, advertised on posters)

 

RESULTS: implementation facilitators

·       Engagement between suppliers & trusts – open communication and feedback loops

·       Regular meetings & clear communication about processes & timelines

·       Clinical leadership & dedicated project management

·       Sharing learning across trusts/networks

·       Clear and effective training

·       Openness to innovation

·       Integrating AI and prioritisation into existing systems: planning/groundwork pre-implementation (e.g. shadow mode/ testing phase)

·       Regular network meetings – community of practice for shared learning

 

RESULTS: implementation barriers

·       Gaps in leadership to manage AI implementation (need balance of clinical and technical knowledge)

·       Complex clinical governance & safety processes

·       Managing AI implementation alongside clinical roles

·       Varied engagement with AI training

·       Staff felt they had insufficient knowledge

·       Lack of national guidance, e.g. from MHRA, RCR & NHSE

·       Integrating secondary capture and AI tool into PACS

·       Integrating prioritisation into PACS, collecting comprehensive data to evaluate impact

·       Varying AI literacy and resource/capacity across trusts

 

RESULTS: Staff and patient experience

·       Staff had positive experiences of AI, regardless of how implemented

·       felt AI could benefit less experienced reporters

·       Despite patients wanting to be told about use of AI, few aware of AI use in their care

·       Patients positive overall about use of AI, due to

o   trust in NHS generally & trust in NHS staff using the AI tools

o   hope that abnormal cases could be processed more quickly

·       Staff & patient concerns: technical performance, overcalling of abnormal scans, overreliance of staff, governance, & accountability

·       Future use: staff & patients felt AI use would & should continue but wanted

o   ongoing monitoring

o   further evidence of effectiveness

 

RESULTS: Perceived impact

·       Confidence/ overreliance

o   Little impact of AI tools on personal reading accuracy

o   But reassurance from AI meant fewer checks with colleagues were required

o   Conversely, some staff concerned that colleagues might over-rely on AI outputs

·       Patient outcomes

o   Too early to state impact on patient outcomes - lack of data

o   Noted even small improvements may have clinically significant effects on patients.

o   However, poor aspects of tool performance (e.g. NG tube placement) were noted as posing risks to clinical outcomes.

o   Some sites evaluating AI-reported false negatives (NB all scans still checked by humans)

·       Equality, diversity and inclusion

o   Few concerns raised: broad view that AI processing of scans not affected by demographics

o   However, staff noted limited local data/capacity to evaluate this

 

IMPLICATIONS

·       Implementation

o   AI has potential to be implemented at scale, but requires

o   Resources, time, & collaboration between NHS staff & AI suppliers

o   Early/ongoing stakeholder engagement (incl staff, AI suppliers, patients)

o   Need to ensure that AI tools address challenges faced by services

o   Mandatory training may build staff confidence & appropriate AI tool usage

·       Staff and patient views generally positive, but wanted

o   more evidence of AI effectiveness

o   ongoing monitoring of AI performance

o   Especially if autonomous AI reporting to be introduced

o   Patients would value being informed of AI use, e.g. posters, leaflets, appointments

 

STRENGTHS & LIMITATIONS

·       Strengths:

o   One of the first real world evaluations of AI tools for diagnostics in the UK

o   Covered range of approaches to AI use/implementation

o   In-depth data on implementation, staff & patient experience, & perceived effectiveness

·       Limitations:

o   Only conducted in 10 NHS Trusts: findings may not be generalised to all trusts

o   Only interviewed ten patients with limited diversity: findings an indication of patient views

o   More observations of different activities/settings would strengthen analysis of AI use & oversight

o   Unable to evaluate long-term perceived impacts