How effectively do Large Language Models (LLMs) and AI-based automation tools assist in Writing and Summarising Evidence Synthesis?

A further webinar from Cochrane Germany, this time with Dr Riaz Qureshi, Associate Professor, Department of Ophthalmology, University of Colorado Anschutz Medical Campus, presenting on Large Language Models (referencing ChatGPT primarily) and their role in evidence synthesis. With the realisation that systematic reviews are expensive, human resource-intensive and time-consuming, Dr Qureshi suitably addressed the issues surrounding GenAI regarding its application to writing Protocols, Synthesis and Guidance (in using AI).

It was acknowledged that in writing protocols, ChatGPT was a good starting point, and saved time, though for generating a search strategy, the Librarian or Information Specialist is still the expert, especially when peer-reviewing. This is especially important to note given the potential danger of over-reliance on AI-generated searches and the assumption that the results are sufficient without consulting information professionals.

Although he cited numerous references in his introductory words, he acknowledged that more research (especially large-scale case studies and prospective studies) is necessary to review the impact and potential of LLM in this context. In particular, Dr Qureshi would like to see more direct comparisons between the human elements and the AI-generated elements in the evidence synthesis process.

That being said, LLMs, such as ChatGPT, can be utilised as follows to:

Perform a more inclusive search
Summarise content
Refine written English for readability (especially for non-native English speakers)
Improve scientific coding
Jumpstart scientific writing skills

Ultimately, as was reiterated in the Q&A session, transparency in the use of LLMs is the key issue with reporting standards currently being worked on, especially with PRISMA in view and Cochrane having also updated their AI policy in recent times. This is also crucial for publishers, who, for the most part, accept the application of AI tools in the process but expect explicit references in terms of how and when it is, with the appropriate disclosures made.