LibGuides: Systematic Reviews: Conduct Quality Assessment

Assessing for Quality and Bias

After screening your articles, you will evaluate them for quality and bias. Decide on a quality assessment tool to grade each article and create a summary of the quality of the literature to include in your review.

According to the Center for Evidence-Based Medicine (CEBM): "Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity, generalizability and relevance."

From the Prisma Explanation and Elaboration Document (Park et al., 2021)

Item 18. Present assessments of risk of bias for each included study

Explanation: For readers to understand the internal validity of a systematic review’s results, they need to know the risk of bias in results of each included study. Reporting only summary data (such as “two of eight studies successfully blinded participants”) is inadequate because it fails to inform readers which studies had each particular methodological shortcoming.

Page M J, Moher D, Bossuyt P M, Boutron I, Hoffmann T C, Mulrow C D et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews BMJ 2021; 372 :n160 doi:10.1136/bmj.n160

Quality Assessment Tools

✅ Best Practice:

Use AI to support the appraisal process — for example:

Pre-fill information (e.g., study design, sample size, intervention)
Help triage which studies need more in-depth review
Highlight potential bias areas for further scrutiny

⚠️ Key Cautions:

AI is not yet capable of fully applying critical appraisal tools like CASP, JBI, or AMSTAR with accuracy.
These tools may misclassify study types, bias levels, or overlook methodological flaws.
There's a risk of over-reliance, especially if the tool’s algorithm is a “black box.”