The right way to use AI is pioneered by Lefty

It's Peer Review Week 2023!

This is the first post of this week, so we're discussing reading other researchers' work, how this impacts our own, and the broader community.

Stick around and follow daily!

If you want to know how i do it, check it here.
If you want to know why you should care, read on.

Eleftherios Teperikidis is the lead author of the first AI-powered systematic review.

 

One of the most interesting articles I read in recent weeks has been one on using artificial inteligence (AI) for research... It's called Prompting ChatGPT to perform an umbrella review. For me, the most remarkable feature of this publication is it dispassionate analysis of the advantages and disadvantages of using AI tools in scientific writing.  

ChatGPT was successfully prompted to execute nearly every step of the systematic review process, by using PICO (Problem, Intervention, Comparison, Outcome) as a universal technique to approach the scientific writing process.

There has been a lot of hype, people fearing their jobs would be at risk, and a new term was coined —hallucinations. For some weird reason, there was a discussion on AI being named as an author. As if academics were ever keen to share credit...

Gupta et al. state:

As pioneers in our field, we must realize that ChatGPT is a tool designed to assist clinicians rather than a substitute for human expertise. All in all, the ethical accountability of utilizing tools such as ChatGPT lies with its user, who is responsible for designing the study, collecting data, and formulating a manuscript.

This seems like a perfectly reasonable approach, especially for researchers that are time deprived. Tools that save us time are always welcomed. Nevertheless, when you are me and see something like this in a scientific paper, you stop and take a look. 

Eleftherios Teperikidis, Aristi Boulmpou, Victoria Potoupni, Satyabrata Kundu, Balpreet Singh & Christodoulos Papadopoulos (2023)  
Does the long-term administration of proton pump inhibitors increase the risk of adverse cardiovascular outcomes? A ChatGPT powered umbrella review, Acta Cardiologica, DOI: 10.1080/00015385.2023.2231299

According to Lefty, ChatGPT 3.5 has the advantage of beeing an aggregator tool, where the whole process can be automated and conducted in record time (minutes rather than days). Nevertheless, it stumbles at the risk of bias assessment using A MeaSurement Tool to Assess systematic Reviews. 

This little checklist (AMSTAR 2) prompts authors to reflect on the design and quality of the studies they plan on including in their analysis. A simple question such as "Did the review authors report on the sources of funding for the studies included in the review?" is often left unaddressed. Even if there's a little note stating "Reporting that the reviewers looked for this information but it was not reported by study authors also qualifies".

This is a question I've seen before. In a form. Repeatedly. 

Do you want to know where? Post-publication audits, that's where... it's heartbreaking to perform them (but I do), and I view it as part of my work as a reviewer. 

Trust in science starts with the people doing it everyday. We want to be able to trust the scholarly record. 

Join me for more this week, as we uncover the novel ways in which trust and truth are achieved. 

It starts with transparency.


 

 

Comments

Post a Comment

Leave a comment!

How to spot fake reviewers: a beginner's guide

Image
For those unfamiliar with or new to academic publishing, peer review manipulation is hard to spot. This is mainly because most peer review reports are still confidential documents that stay buried deep in academic journal's editorial offices. As long as this continues to be the default, recognising good and bad peer reviews is difficult enough... now we have to worry about fake peer review as well?! 😒 Well, yes. Yes, we do. The peer review process in academic publishing is unlike any other. The style and content of a manuscript are criticized, as are the thought processes that birthed it. Everything from word choice to significance of the research focus is under scrutiny. In traditional peer review formats, reviewers recommend that manuscripts be accepted or rejected. This much power may make some less scrupulous people guiddy... or yearning to rig the system. The refereeing work that is supposed to be one of the garantors of research integrity is itself highly susceptible to ma
Read more

Auditing published papers (part I)

Image
After they have been published, I audit research papers.  Image by @freepik Why? How does that work? 😕 A new revised version of The European Code of Conduct for Research Integrity (ECCRI) has been produced by Allea , and recognised by the European Commission, one of the world's largest research funding bodies. This revision focuses on a heightened awareness of issues pertaining to Equity, Diversity, and Inclusion ( EDI ), but also the General Data Protection Regulation ( GDPR ), and even Open Science . I know.. too much legal speak, too many acronyms... 😬 So, let me sum it up for you. 🔊 All institutions, organizations, and individuals that engage in or interact with research must always abide by the principles of: Honesty Respect   Reliability   Accountability  Simple enough, right? Humm.. I analyse papers that have been flagged as "of concern". 😟 The issues I most commonly see have to do with lack of funding or data availability statements, inappropriate citati
Read more

IMHO: why open science should adopt double anonymous peer review

Image
Let's get opinionated... Peer review is still the cornerstone of quality research. This has been repeated by so many during #PeerReviewWeek2023 , it almost lost its meaning to me. I participated in this conversation wholeheartedly, as a freelancer with a variety of roles, from content review to integrity audits.  I’m no longer in academia, but I did my stint there: it’s hard to imagine that many PhD students and postdocs get excited by performing their first peer review. It often results from an invitation trickle-down and, although it can be regarded as a learning opportunity, peer review is hard, time-consuming, unpaid, and often unrecognized work. Open participation is often regarded by researchers as the fairer, more constructive and productive process, leading to improvement in the quality of the finished version of the works. This is the iterative version of having a preprint published in a stable server, with access to all versions of the work, and “comments on”.   The
Read more