Introduction: While systematic reviews (SR) are imperative for qualitative/quantitative
assessment and synthesis of existing evidence, they are increasingly time-consuming given the
ever-increasing amount of literature combined with adoption of stricter methodological

requirements. The emergence of artificial intelligence (AI) offers a promising, Cochrane-
approved solution to make this process more efficient. In this post-hoc analysis using data from a

recently finalized systematic review by our group, we tested an AI approach and determined its
efficacy.
Methods: This was a post-hoc analysis using data from a recently finalized systematic review by
our group: ICAROS II (PROSPERO: CRD42024475582). Manual screening (using Covidence)
was compared to an AI-assisted approach (AS Review Lab), a tool that uses active learning
techniques and learns from decisions on the inclusion or exclusion of publications. Based on
relevant/irrelevant decisions, non-evaluated publications are continuously ranked from most to
least relevant; at any point a screener can opt to stop screening if they do not expect to identify
any more relevant publications. A deduplicated .csv file of all studies included in the titles &
abstract screening (using the manual approach) was exported from Covidence and uploaded to
AS Review Lab. AS Review Lab uses an active learning model which consists of a feature
extraction technique, a classifier, a query strategy and a balance strategy. We opted for the
default setup (term frequency-inverse document frequency [TD-IDF], Naive Bayes [NB],

Maximum, Dynamic resampling) as suggested by the creators. The software requires prior
knowledge to build an initial algorithm, which consisted of 2 relevant and 2 irrelevant articles,
handpicked by the screener/reviewer (PG). After commencing the review, the primary outcome
was total labeled records (relevant/irrelevant), until all relevant articles (included in data
extraction through the manual process) were retrieved. To test predefined stopping criteria in the
literature, the secondary outcome was maximum number of consecutive irrelevant records since
last relevant.

Results: We retrieved all n = 133 included studies out of a total N = 6,247 records (Figure 1)
after screening n = 1,363 (21.8%) studies (Figure 2). The maximum number of consecutive
irrelevant records since last relevant was n = 195, and happened between the 131st and 132nd
relevant studies.
Discussion: The utilized AI tool considerably reduced the number of articles in the screening
process. However, 2 studies may have been missed if we had opted to use predetermined
stopping criteria reported, i.e. to stop after a certain amount of consecutive irrelevant studies.
Factors such as the research question, inclusion/exclusion criteria, characteristics of articles
retrieved (number, relevance, abstract available in .csv file, etc.) massively influence the
algorithm as TD-IDF/NB can overemphasize rare but irrelevant terms, leading to early “false
positives” and late “false negatives”. Other limitations include TD-IDF/NB’s inability to predict
dataset balance (relevant/irrelevant studies ratio) in its decision-making algorithm, and
insufficient accounting for semantics, interchangeable terms and context.
In conclusion, AI algorithms can make the SR screening process more efficient, however,
algorithms are influenced by numerous factors that may mitigate this initial efficiency.