Chaptered Recording: The Past, Present, and Futures of Pragmatic and Responsible Adoption of AI in Systematic Reviews

• Here is the list of sections so you can choose which parts to listen to/see. Select the time tag from below to listen to your preferred section(s).

Thank you for supporting the future content by subscribing to the YouTube channel.

• I do my best to use Chaptering for all my YouTube videos so as to save time for ~1000 subscribers.
• Some complain about YouTube ads; I found Adblock Plus and AdBlock extensions, as well as VPNs,to be very good at blocking all ads.
• If you have any comments or questions, please add them under the video, and I will definitely reply.

00:00 Introduction by Professor Angèle Gayet-Ageron

03:42 Start of lecture and declaring conflict of interests

04:24 Systematic Reviewing as Process vs Systematic Review as Product

08:24 Classification of (Semi)Automation and AI in Systematic Review Context: Rule-based, machine learning (active learning, classifiers), predictive AI, discriminative AI, generative AI, LLMs, and agentic AI

10:26 Welcomers of Automation: Librarians & Statisticians

11:23 Timeline of automation of systematic reviews from 1991 to 2026

17:28 LEMASyR Map and BIMS-ARINES Newsletter

17:51 Number of papers published on automation of SR and LLMs/GenAI

18:37 Systematic review steps targeted by the automation

19:12 Organizations’ actions when facing the challenge; organizations with released documents related to the use of AI in Evidence Synthesis

19:25 Who reads the documents released by the organizations? From PRISMA and GRADE to the RAISE: Responsible Use of AI in Evidence Synthesis

20:17 LLMs used in the automation of systematic review steps

21:13 Promping, Prompt Engineering, Prompting Framework, Prompt Libraries, and Fixed Models

22:28 Prompting features and structure in Elicit (Columns) and Nested Knowledge (Tags)

23:55 Prompting Development and Testing Process (Prompt Engineering)

24:13 Large Language Models’ Features and Facets

24:50 Factors affecting the performance of LLMs

25:09 Issues and problems reported in the literature when using large language models in systematic reviews

25:44 Use case of LLMs in Protocol Development: Refining Research Question, and Exploration & Scoping

26:16 Undermind as an example for Refining Research Question

26:54 Classification of Search in Systematic Review Context: Exploratory, Scoping, Supplementary, Systematic Searching (list of databases and synonyms/keywords), and Top-Up and Update Searches

29:39 What information sources are behind LLM-based tools? Knowledge Graphs: Semantic Scholar, OpenAlex, and Lens.org

30:20 Retrieval-Augmented Generation (RAG) + Knowledge Graph = Safety RAG

31:05 Screening: Active Learning (Relevancy Ranking, Priority Screening, and Stopping Rules or Switching Points), Classifiers, PICO Data Extraction in Elicit

32:20 Active Machine Learning can have issues in the relevancy ranking of records without an abstract or non-standard records (Rayyan)

33:06 Active Machine Learning (Relevancy Ranking) in Covidence

33:12 Priority Screening with Active Learning in ASReview and Stopping Rule or Switching Point: Systematic Review vs Scoping Review or Mapping Review

34:00 Stopping Rule Classification (Switching Point and Switch Back)

34:41 Classifiers

34:53 RCT Classifier in Covidence and Classifiers in EPPI-Reviewer

36:01 Classification of LLMs’ Errors in Data Extraction for Systematic Reviews

40:26 Preventing/Detecting LLM Errors in Data Extraction

42:24 High-Risk Data for Data Extraction using LLM

43:17 Explainability (Seeing Is Believing = SIB) in PICO Portal, DistillerSR, Nested Knowledge, Laser AI, and Elicit

45:04 Data Extraction from Images, Figures, and Tables in PICO Portal, Nested Knowledge, and Elicit

45:33 Meta-Analysis in ChatGPT 4.5 vs ChatGPT 5.5 Extended (Function Calling, Chain-of-Thoughts)

46:58 Should I use AI in Systematic Reviews?

48:52 What AI tool to use for systematic reviews? AI (ML, LLM) features within existing SR tools, Assistive Use

49:10 Futures

50:14 Iterative & Living SR in Elicit

51:03 Golden Rule in Using LLMs

51:23 Golden Rule in Evaluation of LLMs

51:34 Golden Rule In Developing New AI Tools

51:53 To Do Homework

52:39 QA Start. Minab 168, Makan Nasiri

53:00 Question 1: Can AI help in reducing the Publication Bias?

53:53 Question 2: What can we learn from the classification of errors in data extraction? Why is it important to know why AI gets it wrong?

56:59 Question 3: EPPI-Reviewer can do automatic update searches using OpenAlex. How about a baseline search?

58:58 Question 4: Going directly from searching to data extraction means you are replacing/skipping the screening. Screening as Classification or Data Extraction Problem

01:03:07 Question 5: Tools highlight PDFs. Not all these PDFs are CC-BY. Is there a problem with uploading copyrighted paywalled PDFs to LLMs?

01:06:16 Question 6: Safety RAG solved the hallucination problem. What about problems of missing studies?

01:08:48 Question 7: You referred to switching points (stopping rules). What tools support it?

01:11:04 Question 8: How effective is it to use Elicit for data extraction?

01:13:36 Question 9: Are there any journal policies on using AI for systematic reviews?

01:15:54 Statement: AI is like a tool or medicine & has to be tested in empirical studies to determine which tool is suitable for which task and what the risks are. Humanities (ethics and law) are two decades behind science &technology.