what is an example of a systematic review

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on June 22, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

A faster, more affordable way to improve your paper

Scribbr’s new AI Proofreader checks your document and corrects spelling, grammar, and punctuation mistakes with near-human accuracy and the efficiency of AI!

Proofread my paper

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, June 22). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved November 21, 2023, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

Systematic Reviews

• Introduction
• Guidelines and procedures
• Management tools
• Define the question
• Check the topic
• Determine inclusion/exclusion criteria
• Develop a protocol
• Identify keywords
• Databases and search strategies
• Grey literature
• Manage and organise
• Screen & Select
• Locate full text
• Extract data

Example reviews

• Examples of systematic reviews
• Accessing help This link opens in a new window
• Systematic Style Reviews Guide This link opens in a new window

Please choose the tab below for your discipline to see relevant examples.

For more information about how to conduct and write reviews, please see the Guidelines section of this guide.

• Health & Medicine
• Social sciences
• Vibration and bubbles: a systematic review of the effects of helicopter retrieval on injured divers. (2018).
• Nicotine effects on exercise performance and physiological responses in nicotine‐naïve individuals: a systematic review. (2018).
• Association of total white cell count with mortality and major adverse events in patients with peripheral arterial disease: A systematic review. (2014).
• Do MOOCs contribute to student equity and social inclusion? A systematic review 2014–18. (2020).
• Interventions in Foster Family Care: A Systematic Review. (2020).
• Determinants of happiness among healthcare professionals between 2009 and 2019: a systematic review. (2020).
• Systematic review of the outcomes and trade-offs of ten types of decarbonization policy instruments. (2021).
• A systematic review on Asian's farmers' adaptation practices towards climate change. (2018).
• Are concentrations of pollutants in sharks, rays and skates (Elasmobranchii) a cause for concern? A systematic review. (2020).
• << Previous: Write
• Next: Publish >>
• Last Updated: Aug 17, 2023 3:31 PM
• URL: https://libguides.jcu.edu.au/systematic-review

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 17 October 2022.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesise the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimise research b ias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinised by others.
• They’re thorough : they summarise all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomised control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective(s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgement of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

• Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2022, October 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 21 November 2023, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

• Ask a Librarian

Systematic Reviews

• Choosing a Review
• Getting Started
• Further Resources
• Sys Review Request Form

What is a Systematic Review?

A Systematic Review is a very specialized and in-depth literature review.  It is a research method in its own right, where information from research studies is aggregated, analyzed,  and synthesized in an unbiased and reproducible manner to answer a research question. The SR can provide evidence for practice and policy-making as well as gaps in research. In the medical field, where systematic reviews are most common, there are a variety of standard protocols for conducting a SR.

Generally, the systematic review looks at a very specific question.  For example, how effective a particular medical treatment is for a specific population with a stipulated ailment.  How effective a teaching method is for a certain topic in a particular setting.

Systematic reviews are very time intensive and typically require a multi-person research team.  Thus, it is important for you to determine whether a systematic review is right for you.

SR Process Diagram

Librarians can...

Generally speaking, our SR consultations entail 1 or 2 hour-long sessions. We ask that you prepare by filling out our request form .

There are three levels of consultation support we provide. The first two levels are part of our standard service, but assistance at the third level is available only as our capacity allows, and at the discretion of the librarian . Co-authorship acknowledgement is expected for third-level assistance.   

First Level/Session: Guidance on the feasibility of a systematic review of the topic.

• Provide guidance on choosing a review type
• Advise on constructing a research question
• Identify relevant databases for you to search
• Consult on initial search strategies to improve your results
• Make suggestions on reference management tools
• Locate existing SR's that could be models for your work

Second Level/Session:  Follow-up on initial work

• Feedback on your initial searches, database selection, including appropriateness of terms, strategies, completeness, and accuracy.

Third Level Support:  If you need further support, you may request the librarian participate as a co-author on the project.  These co-author level activities include

• Contributing to completing the SR process flow chart for the project
• Designing and refining in-depth search strategies for the team
• Train the team on using a screening platformlike Covidence or Rayyan
• Running searches in appropriate databases
• Continually updating search results using alerts
• Writing Methods section of SR paper
• Reviewing final manuscript
• Next: Choosing a Review >>
• Last Edited: Nov 21, 2023 8:17 AM
• URL: https://guides.lib.purdue.edu/sys-review

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard
• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: Nov 13, 2023 5:05 PM
• URL: https://lib.guides.umd.edu/SR

Library Services

UCL LIBRARY SERVICES

• Library skills
• Systematic reviews

What are systematic reviews?

• Types of systematic reviews
• Formulating a research question
• Identifying studies
• Searching databases
• Describing and appraising studies
• Synthesis and systematic maps
• Software for systematic reviews
• Online training and support
• Live and face to face training
• Individual support
• Further help

Systematic reviews are a type of literature review of research which require equivalent standards of rigour as primary research. They have a clear, logical rationale that is reported to the reader of the review. They are used in research and policymaking to inform evidence-based decisions and practice. They differ from traditional literature reviews particularly in the following elements of conduct and reporting.

Systematic reviews: 

• use explicit and transparent methods
• are a piece of research following a standard set of stages
• are accountable, replicable and updateable
• involve users to ensure a review is relevant and useful.

For example, systematic reviews (like all research) should have a clear research question, and the perspective of the authors in their approach to addressing the question is described. There are clearly described methods on how each study in a review was identified, how that study was appraised for quality and relevance and how it is combined with other studies in order to address the review question. A systematic review usually involves more than one person in order to increase the objectivity and trustworthiness of the reviews methods and findings.

Research protocols for systematic reviews may be peer-reviewed and published or registered in a suitable repository to help avoid duplication of reviews and for comparisons to be made with the final review and the planned review.

• History of systematic reviews to inform policy (EPPI-Centre)
• Six reasons why it is important to be systematic (EPPI-Centre)
• Evidence Synthesis International (ESI): Position Statement Describes the issues, principles and goals in synthesising research evidence to inform policy, practice and decisions

On this page

Should all literature reviews be 'systematic reviews', different methods for systematic reviews, reporting standards for systematic reviews.

Literature reviews provide a more complete picture of research knowledge than is possible from individual pieces of research. This can be used to: clarify what is known from research, provide new perspectives, build theory, test theory, identify research gaps or inform research agendas.

A systematic review requires a considerable amount of time and resources, and is one type of literature review.

If the purpose of a review is to make justifiable evidence claims, then it should be systematic, as a systematic review uses rigorous explicit methods. The methods used can depend on the purpose of the review, and the time and resources available.

A 'non-systematic review' might use some of the same methods as systematic reviews, such as systematic approaches to identify studies or quality appraise the literature. There may be times when this approach can be useful. In a student dissertation, for example, there may not be the time to be fully systematic in a review of the literature if this is only one small part of the thesis. In other types of research, there may also be a need to obtain a quick and not necessarily thorough overview of a literature to inform some other work (including a systematic review). Another example, is where policymakers, or other people using research findings, want to make quick decisions and there is no systematic review available to help them. They have a choice of gaining a rapid overview of the research literature or not having any research evidence to help their decision-making. 

Just like any other piece of research, the methods used to undertake any literature review should be carefully planned to justify the conclusions made. 

Finding out about different types of systematic reviews and the methods used for systematic reviews, and reading both systematic and other types of review will help to understand some of the differences. 

Typically, a systematic review addresses a focussed, structured research question in order to inform understanding and decisions on an area. (see the  Formulating a research question  section for examples). 

Sometimes systematic reviews ask a broad research question, and one strategy to achieve this is the use of several focussed sub-questions each addressed by sub-components of the review.  

Another strategy is to develop a map to describe the type of research that has been undertaken in relation to a research question. Some maps even describe over 2,000 papers, while others are much smaller. One purpose of a map is to help choose a sub-set of studies to explore more fully in a synthesis. There are also other purposes of maps: see the box on  systematic evidence maps  for further information. 

Reporting standards specify minimum elements that need to go into the reporting of a review. The reporting standards refer mainly to methodological issues but they are not as detailed or specific as critical appraisal for the methodological standards of conduct of a review.

A number of organisations have developed specific guidelines and standards for both the conducting and reporting on systematic reviews in different topic areas.  

• PRISMA PRISMA is a reporting standard and is an acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses. The Key Documents section of the PRISMA website links to a checklist, flow diagram and explanatory notes. PRISMA is less useful for certain types of reviews, including those that are iterative.
• eMERGe eMERGe is a reporting standard that has been developed for meta-ethnographies, a qualitative synthesis method.
• ROSES: RepOrting standards for Systematic Evidence Syntheses Reporting standards, including forms and flow diagram, designed specifically for systematic reviews and maps in the field of conservation and environmental management.

Useful books about systematic reviews

Systematic approaches to a successful literature review

An introduction to systematic reviews

Cochrane handbook for systematic reviews of interventions

Systematic reviews: crd's guidance for undertaking reviews in health care.

Finding what works in health care: Standards for systematic reviews

Systematic Reviews in the Social Sciences

Meta-analysis and research synthesis.

Research Synthesis and Meta-Analysis

Doing a Systematic Review

Literature reviews.

• What is a literature review?
• Why are literature reviews important?
• << Previous: Systematic reviews
• Next: Types of systematic reviews >>
• Last Updated: Nov 8, 2023 10:55 AM
• URL: https://library-guides.ucl.ac.uk/systematic-reviews

Systematic Reviews

• What is a Systematic Review?

A systematic review is an evidence synthesis that uses explicit, reproducible methods to perform a comprehensive literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.

Key Characteristics of a Systematic Review:

Generally, systematic reviews must have:

• a clearly stated set of objectives with pre-defined eligibility criteria for studies
• an explicit, reproducible methodology
• a systematic search that attempts to identify all studies that would meet the eligibility criteria
• an assessment of the validity of the findings of the included studies, for example through the assessment of the risk of bias
• a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

A meta-analysis is a systematic review that uses quantitative methods to synthesize and summarize the pooled data from included studies.

Additional Information

• How-to Books
• Beyond Health Sciences

• Cochrane Handbook For Systematic Reviews of Interventions Provides guidance to authors for the preparation of Cochrane Intervention reviews. Chapter 6 covers searching for reviews.
• Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care From The University of York Centre for Reviews and Dissemination: Provides practical guidance for undertaking evidence synthesis based on a thorough understanding of systematic review methodology. It presents the core principles of systematic reviewing, and in complementary chapters, highlights issues that are specific to reviews of clinical tests, public health interventions, adverse effects, and economic evaluations.
• Cornell, Sytematic Reviews and Evidence Synthesis Beyond the Health Sciences Video series geared for librarians but very informative about searching outside medicine.
• << Previous: Getting Started
• Next: Levels of Evidence >>
• Getting Started
• Levels of Evidence
• Locating Systematic Reviews
• Searching Systematically
• Developing Answerable Questions
• Identifying Synonyms & Related Terms
• Using Truncation and Wildcards
• Identifying Search Limits/Exclusion Criteria
• Keyword vs. Subject Searching
• Where to Search
• Search Filters
• Sensitivity vs. Precision
• Core Databases
• Other Databases
• Clinical Trial Registries
• Conference Presentations
• Databases Indexing Grey Literature
• Web Searching
• Handsearching
• Citation Indexes
• Documenting the Search Process
• Managing your Review

Research Support

• Last Updated: Nov 17, 2023 11:47 AM
• URL: https://guides.library.ucdavis.edu/systematic-reviews

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Advanced Search
• Journal List
• Korean J Anesthesiol
• v.71(2); 2018 Apr

Introduction to systematic review and meta-analysis

1 Department of Anesthesiology and Pain Medicine, Inje University Seoul Paik Hospital, Seoul, Korea

2 Department of Anesthesiology and Pain Medicine, Chung-Ang University College of Medicine, Seoul, Korea

Systematic reviews and meta-analyses present results by combining and analyzing data from different studies conducted on similar research topics. In recent years, systematic reviews and meta-analyses have been actively performed in various fields including anesthesiology. These research methods are powerful tools that can overcome the difficulties in performing large-scale randomized controlled trials. However, the inclusion of studies with any biases or improperly assessed quality of evidence in systematic reviews and meta-analyses could yield misleading results. Therefore, various guidelines have been suggested for conducting systematic reviews and meta-analyses to help standardize them and improve their quality. Nonetheless, accepting the conclusions of many studies without understanding the meta-analysis can be dangerous. Therefore, this article provides an easy introduction to clinicians on performing and understanding meta-analyses.

Introduction

A systematic review collects all possible studies related to a given topic and design, and reviews and analyzes their results [ 1 ]. During the systematic review process, the quality of studies is evaluated, and a statistical meta-analysis of the study results is conducted on the basis of their quality. A meta-analysis is a valid, objective, and scientific method of analyzing and combining different results. Usually, in order to obtain more reliable results, a meta-analysis is mainly conducted on randomized controlled trials (RCTs), which have a high level of evidence [ 2 ] ( Fig. 1 ). Since 1999, various papers have presented guidelines for reporting meta-analyses of RCTs. Following the Quality of Reporting of Meta-analyses (QUORUM) statement [ 3 ], and the appearance of registers such as Cochrane Library’s Methodology Register, a large number of systematic literature reviews have been registered. In 2009, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [ 4 ] was published, and it greatly helped standardize and improve the quality of systematic reviews and meta-analyses [ 5 ].

Levels of evidence.

In anesthesiology, the importance of systematic reviews and meta-analyses has been highlighted, and they provide diagnostic and therapeutic value to various areas, including not only perioperative management but also intensive care and outpatient anesthesia [6–13]. Systematic reviews and meta-analyses include various topics, such as comparing various treatments of postoperative nausea and vomiting [ 14 , 15 ], comparing general anesthesia and regional anesthesia [ 16 – 18 ], comparing airway maintenance devices [ 8 , 19 ], comparing various methods of postoperative pain control (e.g., patient-controlled analgesia pumps, nerve block, or analgesics) [ 20 – 23 ], comparing the precision of various monitoring instruments [ 7 ], and meta-analysis of dose-response in various drugs [ 12 ].

Thus, literature reviews and meta-analyses are being conducted in diverse medical fields, and the aim of highlighting their importance is to help better extract accurate, good quality data from the flood of data being produced. However, a lack of understanding about systematic reviews and meta-analyses can lead to incorrect outcomes being derived from the review and analysis processes. If readers indiscriminately accept the results of the many meta-analyses that are published, incorrect data may be obtained. Therefore, in this review, we aim to describe the contents and methods used in systematic reviews and meta-analyses in a way that is easy to understand for future authors and readers of systematic review and meta-analysis.

Study Planning

It is easy to confuse systematic reviews and meta-analyses. A systematic review is an objective, reproducible method to find answers to a certain research question, by collecting all available studies related to that question and reviewing and analyzing their results. A meta-analysis differs from a systematic review in that it uses statistical methods on estimates from two or more different studies to form a pooled estimate [ 1 ]. Following a systematic review, if it is not possible to form a pooled estimate, it can be published as is without progressing to a meta-analysis; however, if it is possible to form a pooled estimate from the extracted data, a meta-analysis can be attempted. Systematic reviews and meta-analyses usually proceed according to the flowchart presented in Fig. 2 . We explain each of the stages below.

Flowchart illustrating a systematic review.

Formulating research questions

A systematic review attempts to gather all available empirical research by using clearly defined, systematic methods to obtain answers to a specific question. A meta-analysis is the statistical process of analyzing and combining results from several similar studies. Here, the definition of the word “similar” is not made clear, but when selecting a topic for the meta-analysis, it is essential to ensure that the different studies present data that can be combined. If the studies contain data on the same topic that can be combined, a meta-analysis can even be performed using data from only two studies. However, study selection via a systematic review is a precondition for performing a meta-analysis, and it is important to clearly define the Population, Intervention, Comparison, Outcomes (PICO) parameters that are central to evidence-based research. In addition, selection of the research topic is based on logical evidence, and it is important to select a topic that is familiar to readers without clearly confirmed the evidence [ 24 ].

Protocols and registration

In systematic reviews, prior registration of a detailed research plan is very important. In order to make the research process transparent, primary/secondary outcomes and methods are set in advance, and in the event of changes to the method, other researchers and readers are informed when, how, and why. Many studies are registered with an organization like PROSPERO ( http://www.crd.york.ac.uk/PROSPERO/ ), and the registration number is recorded when reporting the study, in order to share the protocol at the time of planning.

Defining inclusion and exclusion criteria

Information is included on the study design, patient characteristics, publication status (published or unpublished), language used, and research period. If there is a discrepancy between the number of patients included in the study and the number of patients included in the analysis, this needs to be clearly explained while describing the patient characteristics, to avoid confusing the reader.

Literature search and study selection

In order to secure proper basis for evidence-based research, it is essential to perform a broad search that includes as many studies as possible that meet the inclusion and exclusion criteria. Typically, the three bibliographic databases Medline, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) are used. In domestic studies, the Korean databases KoreaMed, KMBASE, and RISS4U may be included. Effort is required to identify not only published studies but also abstracts, ongoing studies, and studies awaiting publication. Among the studies retrieved in the search, the researchers remove duplicate studies, select studies that meet the inclusion/exclusion criteria based on the abstracts, and then make the final selection of studies based on their full text. In order to maintain transparency and objectivity throughout this process, study selection is conducted independently by at least two investigators. When there is a inconsistency in opinions, intervention is required via debate or by a third reviewer. The methods for this process also need to be planned in advance. It is essential to ensure the reproducibility of the literature selection process [ 25 ].

Quality of evidence

However, well planned the systematic review or meta-analysis is, if the quality of evidence in the studies is low, the quality of the meta-analysis decreases and incorrect results can be obtained [ 26 ]. Even when using randomized studies with a high quality of evidence, evaluating the quality of evidence precisely helps determine the strength of recommendations in the meta-analysis. One method of evaluating the quality of evidence in non-randomized studies is the Newcastle-Ottawa Scale, provided by the Ottawa Hospital Research Institute 1) . However, we are mostly focusing on meta-analyses that use randomized studies.

If the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) system ( http://www.gradeworkinggroup.org/ ) is used, the quality of evidence is evaluated on the basis of the study limitations, inaccuracies, incompleteness of outcome data, indirectness of evidence, and risk of publication bias, and this is used to determine the strength of recommendations [ 27 ]. As shown in Table 1 , the study limitations are evaluated using the “risk of bias” method proposed by Cochrane 2) . This method classifies bias in randomized studies as “low,” “high,” or “unclear” on the basis of the presence or absence of six processes (random sequence generation, allocation concealment, blinding participants or investigators, incomplete outcome data, selective reporting, and other biases) [ 28 ].

The Cochrane Collaboration’s Tool for Assessing the Risk of Bias [ 28 ]

Data extraction

Two different investigators extract data based on the objectives and form of the study; thereafter, the extracted data are reviewed. Since the size and format of each variable are different, the size and format of the outcomes are also different, and slight changes may be required when combining the data [ 29 ]. If there are differences in the size and format of the outcome variables that cause difficulties combining the data, such as the use of different evaluation instruments or different evaluation timepoints, the analysis may be limited to a systematic review. The investigators resolve differences of opinion by debate, and if they fail to reach a consensus, a third-reviewer is consulted.

Data Analysis

The aim of a meta-analysis is to derive a conclusion with increased power and accuracy than what could not be able to achieve in individual studies. Therefore, before analysis, it is crucial to evaluate the direction of effect, size of effect, homogeneity of effects among studies, and strength of evidence [ 30 ]. Thereafter, the data are reviewed qualitatively and quantitatively. If it is determined that the different research outcomes cannot be combined, all the results and characteristics of the individual studies are displayed in a table or in a descriptive form; this is referred to as a qualitative review. A meta-analysis is a quantitative review, in which the clinical effectiveness is evaluated by calculating the weighted pooled estimate for the interventions in at least two separate studies.

The pooled estimate is the outcome of the meta-analysis, and is typically explained using a forest plot ( Figs. 3 and ​ and4). 4 ). The black squares in the forest plot are the odds ratios (ORs) and 95% confidence intervals in each study. The area of the squares represents the weight reflected in the meta-analysis. The black diamond represents the OR and 95% confidence interval calculated across all the included studies. The bold vertical line represents a lack of therapeutic effect (OR = 1); if the confidence interval includes OR = 1, it means no significant difference was found between the treatment and control groups.

Forest plot analyzed by two different models using the same data. (A) Fixed-effect model. (B) Random-effect model. The figure depicts individual trials as filled squares with the relative sample size and the solid line as the 95% confidence interval of the difference. The diamond shape indicates the pooled estimate and uncertainty for the combined effect. The vertical line indicates the treatment group shows no effect (OR = 1). Moreover, if the confidence interval includes 1, then the result shows no evidence of difference between the treatment and control groups.

Forest plot representing homogeneous data.

Dichotomous variables and continuous variables

In data analysis, outcome variables can be considered broadly in terms of dichotomous variables and continuous variables. When combining data from continuous variables, the mean difference (MD) and standardized mean difference (SMD) are used ( Table 2 ).

Summary of Meta-analysis Methods Available in RevMan [ 28 ]

The MD is the absolute difference in mean values between the groups, and the SMD is the mean difference between groups divided by the standard deviation. When results are presented in the same units, the MD can be used, but when results are presented in different units, the SMD should be used. When the MD is used, the combined units must be shown. A value of “0” for the MD or SMD indicates that the effects of the new treatment method and the existing treatment method are the same. A value lower than “0” means the new treatment method is less effective than the existing method, and a value greater than “0” means the new treatment is more effective than the existing method.

When combining data for dichotomous variables, the OR, risk ratio (RR), or risk difference (RD) can be used. The RR and RD can be used for RCTs, quasi-experimental studies, or cohort studies, and the OR can be used for other case-control studies or cross-sectional studies. However, because the OR is difficult to interpret, using the RR and RD, if possible, is recommended. If the outcome variable is a dichotomous variable, it can be presented as the number needed to treat (NNT), which is the minimum number of patients who need to be treated in the intervention group, compared to the control group, for a given event to occur in at least one patient. Based on Table 3 , in an RCT, if x is the probability of the event occurring in the control group and y is the probability of the event occurring in the intervention group, then x = c/(c + d), y = a/(a + b), and the absolute risk reduction (ARR) = x − y. NNT can be obtained as the reciprocal, 1/ARR.

Calculation of the Number Needed to Treat in the Dichotomous table

Fixed-effect models and random-effect models

In order to analyze effect size, two types of models can be used: a fixed-effect model or a random-effect model. A fixed-effect model assumes that the effect of treatment is the same, and that variation between results in different studies is due to random error. Thus, a fixed-effect model can be used when the studies are considered to have the same design and methodology, or when the variability in results within a study is small, and the variance is thought to be due to random error. Three common methods are used for weighted estimation in a fixed-effect model: 1) inverse variance-weighted estimation 3) , 2) Mantel-Haenszel estimation 4) , and 3) Peto estimation 5) .

A random-effect model assumes heterogeneity between the studies being combined, and these models are used when the studies are assumed different, even if a heterogeneity test does not show a significant result. Unlike a fixed-effect model, a random-effect model assumes that the size of the effect of treatment differs among studies. Thus, differences in variation among studies are thought to be due to not only random error but also between-study variability in results. Therefore, weight does not decrease greatly for studies with a small number of patients. Among methods for weighted estimation in a random-effect model, the DerSimonian and Laird method 6) is mostly used for dichotomous variables, as the simplest method, while inverse variance-weighted estimation is used for continuous variables, as with fixed-effect models. These four methods are all used in Review Manager software (The Cochrane Collaboration, UK), and are described in a study by Deeks et al. [ 31 ] ( Table 2 ). However, when the number of studies included in the analysis is less than 10, the Hartung-Knapp-Sidik-Jonkman method 7) can better reduce the risk of type 1 error than does the DerSimonian and Laird method [ 32 ].

Fig. 3 shows the results of analyzing outcome data using a fixed-effect model (A) and a random-effect model (B). As shown in Fig. 3 , while the results from large studies are weighted more heavily in the fixed-effect model, studies are given relatively similar weights irrespective of study size in the random-effect model. Although identical data were being analyzed, as shown in Fig. 3 , the significant result in the fixed-effect model was no longer significant in the random-effect model. One representative example of the small study effect in a random-effect model is the meta-analysis by Li et al. [ 33 ]. In a large-scale study, intravenous injection of magnesium was unrelated to acute myocardial infarction, but in the random-effect model, which included numerous small studies, the small study effect resulted in an association being found between intravenous injection of magnesium and myocardial infarction. This small study effect can be controlled for by using a sensitivity analysis, which is performed to examine the contribution of each of the included studies to the final meta-analysis result. In particular, when heterogeneity is suspected in the study methods or results, by changing certain data or analytical methods, this method makes it possible to verify whether the changes affect the robustness of the results, and to examine the causes of such effects [ 34 ].

Heterogeneity

Homogeneity test is a method whether the degree of heterogeneity is greater than would be expected to occur naturally when the effect size calculated from several studies is higher than the sampling error. This makes it possible to test whether the effect size calculated from several studies is the same. Three types of homogeneity tests can be used: 1) forest plot, 2) Cochrane’s Q test (chi-squared), and 3) Higgins I 2 statistics. In the forest plot, as shown in Fig. 4 , greater overlap between the confidence intervals indicates greater homogeneity. For the Q statistic, when the P value of the chi-squared test, calculated from the forest plot in Fig. 4 , is less than 0.1, it is considered to show statistical heterogeneity and a random-effect can be used. Finally, I 2 can be used [ 35 ].

I 2 , calculated as shown above, returns a value between 0 and 100%. A value less than 25% is considered to show strong homogeneity, a value of 50% is average, and a value greater than 75% indicates strong heterogeneity.

Even when the data cannot be shown to be homogeneous, a fixed-effect model can be used, ignoring the heterogeneity, and all the study results can be presented individually, without combining them. However, in many cases, a random-effect model is applied, as described above, and a subgroup analysis or meta-regression analysis is performed to explain the heterogeneity. In a subgroup analysis, the data are divided into subgroups that are expected to be homogeneous, and these subgroups are analyzed. This needs to be planned in the predetermined protocol before starting the meta-analysis. A meta-regression analysis is similar to a normal regression analysis, except that the heterogeneity between studies is modeled. This process involves performing a regression analysis of the pooled estimate for covariance at the study level, and so it is usually not considered when the number of studies is less than 10. Here, univariate and multivariate regression analyses can both be considered.

Publication bias

Publication bias is the most common type of reporting bias in meta-analyses. This refers to the distortion of meta-analysis outcomes due to the higher likelihood of publication of statistically significant studies rather than non-significant studies. In order to test the presence or absence of publication bias, first, a funnel plot can be used ( Fig. 5 ). Studies are plotted on a scatter plot with effect size on the x-axis and precision or total sample size on the y-axis. If the points form an upside-down funnel shape, with a broad base that narrows towards the top of the plot, this indicates the absence of a publication bias ( Fig. 5A ) [ 29 , 36 ]. On the other hand, if the plot shows an asymmetric shape, with no points on one side of the graph, then publication bias can be suspected ( Fig. 5B ). Second, to test publication bias statistically, Begg and Mazumdar’s rank correlation test 8) [ 37 ] or Egger’s test 9) [ 29 ] can be used. If publication bias is detected, the trim-and-fill method 10) can be used to correct the bias [ 38 ]. Fig. 6 displays results that show publication bias in Egger’s test, which has then been corrected using the trim-and-fill method using Comprehensive Meta-Analysis software (Biostat, USA).

Funnel plot showing the effect size on the x-axis and sample size on the y-axis as a scatter plot. (A) Funnel plot without publication bias. The individual plots are broader at the bottom and narrower at the top. (B) Funnel plot with publication bias. The individual plots are located asymmetrically.

Funnel plot adjusted using the trim-and-fill method. White circles: comparisons included. Black circles: inputted comparisons using the trim-and-fill method. White diamond: pooled observed log risk ratio. Black diamond: pooled inputted log risk ratio.

Result Presentation

When reporting the results of a systematic review or meta-analysis, the analytical content and methods should be described in detail. First, a flowchart is displayed with the literature search and selection process according to the inclusion/exclusion criteria. Second, a table is shown with the characteristics of the included studies. A table should also be included with information related to the quality of evidence, such as GRADE ( Table 4 ). Third, the results of data analysis are shown in a forest plot and funnel plot. Fourth, if the results use dichotomous data, the NNT values can be reported, as described above.

The GRADE Evidence Quality for Each Outcome

N: number of studies, ROB: risk of bias, PON: postoperative nausea, POV: postoperative vomiting, PONV: postoperative nausea and vomiting, CI: confidence interval, RR: risk ratio, AR: absolute risk.

When Review Manager software (The Cochrane Collaboration, UK) is used for the analysis, two types of P values are given. The first is the P value from the z-test, which tests the null hypothesis that the intervention has no effect. The second P value is from the chi-squared test, which tests the null hypothesis for a lack of heterogeneity. The statistical result for the intervention effect, which is generally considered the most important result in meta-analyses, is the z-test P value.

A common mistake when reporting results is, given a z-test P value greater than 0.05, to say there was “no statistical significance” or “no difference.” When evaluating statistical significance in a meta-analysis, a P value lower than 0.05 can be explained as “a significant difference in the effects of the two treatment methods.” However, the P value may appear non-significant whether or not there is a difference between the two treatment methods. In such a situation, it is better to announce “there was no strong evidence for an effect,” and to present the P value and confidence intervals. Another common mistake is to think that a smaller P value is indicative of a more significant effect. In meta-analyses of large-scale studies, the P value is more greatly affected by the number of studies and patients included, rather than by the significance of the results; therefore, care should be taken when interpreting the results of a meta-analysis.

When performing a systematic literature review or meta-analysis, if the quality of studies is not properly evaluated or if proper methodology is not strictly applied, the results can be biased and the outcomes can be incorrect. However, when systematic reviews and meta-analyses are properly implemented, they can yield powerful results that could usually only be achieved using large-scale RCTs, which are difficult to perform in individual studies. As our understanding of evidence-based medicine increases and its importance is better appreciated, the number of systematic reviews and meta-analyses will keep increasing. However, indiscriminate acceptance of the results of all these meta-analyses can be dangerous, and hence, we recommend that their results be received critically on the basis of a more accurate understanding.

1) http://www.ohri.ca .

2) http://methods.cochrane.org/bias/assessing-risk-bias-included-studies .

3) The inverse variance-weighted estimation method is useful if the number of studies is small with large sample sizes.

4) The Mantel-Haenszel estimation method is useful if the number of studies is large with small sample sizes.

5) The Peto estimation method is useful if the event rate is low or one of the two groups shows zero incidence.

6) The most popular and simplest statistical method used in Review Manager and Comprehensive Meta-analysis software.

7) Alternative random-effect model meta-analysis that has more adequate error rates than does the common DerSimonian and Laird method, especially when the number of studies is small. However, even with the Hartung-Knapp-Sidik-Jonkman method, when there are less than five studies with very unequal sizes, extra caution is needed.

8) The Begg and Mazumdar rank correlation test uses the correlation between the ranks of effect sizes and the ranks of their variances [ 37 ].

9) The degree of funnel plot asymmetry as measured by the intercept from the regression of standard normal deviates against precision [ 29 ].

10) If there are more small studies on one side, we expect the suppression of studies on the other side. Trimming yields the adjusted effect size and reduces the variance of the effects by adding the original studies back into the analysis as a mirror image of each study.

• Research Consultation
• How-to Guides
• People Search
• Library Databases
• Research Guides
• Find Periodicals
• Google Scholar
• Library Map
• Texts for Vets
• Interlibrary Loan - ILL
• Research & Instruction
• Computers and Laptops
• Services For...
• Reserve a Room
• APSU Records Management
• Library Collections
• Digital Collections
• Special Collections
• Veterans' Oral History
• Government Resources
• Library Information
• Woodward Library Society
• New @ Woodward Library
• Celebration of Scholarship
• Join Woodward Library Society

Systematic Reviews: What is a systematic review?

• What is a systematic review?
• How do you perform a systematic review?
• Tools & Resources

What is a Systematic Review?

In the health-related professions, systematic reviews are considered the most reliable resources.  They are studies that make up the top level of the evidence-based information pyramid, and as a result, they are the most sought after information for questions about health. Systematic reviews are not quite the same as literature reviews. Literature reviews, also known as narrative reviews, attempt to find all published materials on a subject, whereas systematic reviews try to find everything that focuses on answering a specific question.  Since systematic reviews are generally associated with health related fields, their main objective is to ensure the results of the review provide accurate evidence that answers relevant questions.  If you are looking for information about literature reviews, please check the library's guide on the topic  here .

Why use systematic reviews to answer questions?

When looking for answers to health questions, systematic reviews are considered the best resources to use for evidence-based information.  The predefined protocols, the amount of information reviewed, the evaluation process involved, and the efforts to eliminate bias are all a part of what makes health professionals consider systematic reviews to be the highest level of evidence based information available.  As a part of the process, systematic reviews tend to look at and evaluate all the randomized controlled trials, or all the cohort studies, for their specific topic.  By looking at and evaluating a vast amount of comparable studies, a systematic review is able to provide answers that have a much stronger level of evidence than any individual study.

Why perform a systematic review?

These reviews collect large amounts of information that fit within the predetermined parameters, so performing a systematic review is an excellent way to develop expertise on a topic.  Setting up the criteria, searching for the information, and evaluating the information found, gives the reviewer an extremely strong understanding of the process needed to create a review as well as how to evaluate its various elements.  Creating a systematic review gives the reviewer an opportunity to further the discussion on a topic.  In the health fields, performing and then publishing these reviews provides more evidence on topics that can be used for making decisions in a clinical environment.

How to find a systematic review

There are a number of databases that focus on health related resources, and most of them search through journals that include systematic reviews.  In these cases, you can include the words “systematic review” and the results will include entries that have the words “systematic review” in them somewhere.  Many of these results will be systematic reviews; however, some of the results may include these words, but are not systematic reviews.  A few databases that are used by researchers have added in limitation features that make it easier to find systematic reviews, and ensure a specific article/document/publication type are found.  Here are three examples of databases and how to limit their search results to systematic reviews:

• Scroll down the main page and in the middle column, click on Clinical Queries.  In the provided search box type in your search terms.  The middle row of results is systematic reviews.  You can type your search terms in on the main page, and limit the results after they are listed.  To do this, find the section for “article types” on the left side of the page and click “customize”.  On the list that pops up, clear all the check boxes except the one for “systematic reviews”, then click “Show”.  When the main page is displayed again, click on “systematic reviews” to reset the search using that limit.
• In CINAHL (EBSCO version) it is easy to limit search results to systematic reviews from the initial search page.  Scroll down to find “Publication types” on the right side of the page.  In the Publication Types box, search through the list until you find systematic reviews and highlight it.  After this has been done, add in any other limits needed and then search as usual.
• This is the easiest of the three to use for finding systematic reviews.  On the main search page, type in the terms, run the search and wait for the results page.  On the results page, the right side has a limit that says “systematic review”.  Click on it and the results will all be systematic reviews.

Types of systematic reviews

These are a few of the various types of systematic reviews.

Critical review

Critical reviews are often thought to be the same as a literature review.  They involve a comprehensive review of a topic that involves a high level of analysis that seeks to identify the most important aspects of a subject.

The overview is a look at the literature available on the topic.  They may or may not have an analytical component that provides a synthesis of the information found, but this depends on how systematic the review has been.

Qualitative systematic review

This type of study looks at as many qualitative studies and tries to find themes among these studies that lie across the range of reviews. The information is then organized into a narrative that is frequently accompanied by a conceptual model.

Rapid review

The number of studies included in a rapid review are dictated by time constraints. Most often these reviews are conducted to determine what is known about policies or practices, and then determine how much these are based on evidence.

Scoping review

These reviews are intended to quickly assess the availability of research on a specific topic, and then determine whether the evidence provided in these research articles is sufficient and appropriate, 

Umbrella review

Umbrella reviews research broad topic, and look at a wide range of reviews that have been done for a number of potential interventions. It then triest to establish the pros and cons of each treatment and what the potential results would be.

Subject Guide

• Next: How do you perform a systematic review? >>
• Last Updated: Jun 30, 2023 2:03 PM
• URL: https://libguides.apsu.edu/systematic-reviews

Systematic Reviews

Describes what is involved with conducting a systematic review of the literature for evidence-based public health and how the librarian is a partner in the process.

Several CDC librarians have special training in conducting literature searches for systematic reviews.  Literature searches for systematic reviews can take a few weeks to several months from planning to delivery.

Fill out a search request form here  or contact the Stephen B. Thacker CDC Library by email  [email protected] or telephone 404-639-1717.

Campbell Collaboration

Cochrane Collaboration

Eppi Centre

Joanna Briggs Institute

McMaster University

PRISMA Statement

Systematic Reviews – CRD’s Guide

Systematic Reviews of Health Promotion and Public Health Interventions

The Guide to Community Preventive Services

Look for systematic reviews that have already been published. 

• To ensure that the work has not already been done.
• To provides examples of search strategies for your topic

Look in PROSPERO for registered systematic reviews.

Search Cochrane and CRD-York for systematic reviews.

Search filter for finding systematic reviews in PubMed

Other search filters to locate systematic reviews

A systematic review attempts to collect and analyze all evidence that answers a specific question.  The question must be clearly defined and have inclusion and exclusion criteria. A broad and thorough search of the literature is performed and a critical analysis of the search results is reported and ultimately provides a current evidence-based answer  to the specific question.

Time:  According to Cochrane , it takes 18 months on average to complete a Systematic Review.

The average systematic review from beginning to end requires 18 months of work. “…to find out about a healthcare intervention it is worth searching research literature thoroughly to see if the answer is already known. This may require considerable work over many months…” ( Cochrane Collaboration )

Review Team: Team Members at minimum…

• Content expert
• 2 reviewers
• 1 tie breaker
• 1 statistician (meta-analysis)
• 1 economist if conducting an economic analysis
• *1 librarian (expert searcher) trained in systematic reviews

“Expert searchers are an important part of the systematic review team, crucial throughout the review process-from the development of the proposal and research question to publication.” ( McGowan & Sampson, 2005 )

*Ask your librarian to write a methods section regarding the search methods and to give them co-authorship. You may also want to consider providing a copy of one or all of the search strategies used in an appendix.

The Question to Be Answered: A clearly defined and specific question or questions with inclusion and exclusion criteria.

Written Protocol: Outline the study method, rationale, key questions, inclusion and exclusion criteria, literature searches, data abstraction and data management, analysis of quality of the individual studies, synthesis of data, and grading of the evidience for each key question.

Literature Searches:  Search for any systematic reviews that may already answer the key question(s).  Next, choose appropriate databases and conduct very broad, comprehensive searches.  Search strategies must be documented so that they can be duplicated.  The librarian is integral to this step of the process. Before your librarian creates a search strategy and starts searching in earnest you should write a detailed PICO question , determine the inclusion and exclusion criteria for your study, run a preliminary search, and have 2-4 articles that already fit the criteria for your review.

What is searched depends on the topic of the review but should include…

• At least 3 standard medical databases like PubMed/Medline, CINAHL, Embase, etc..
• At least 2 grey literature resources like Clinicaltrials.gov, COS Conference Papers Index, Grey Literature Report,  etc…

Citation Management: EndNote is a bibliographic management tools that assist researchers in managing citations.  The Stephen B. Thacker CDC Library oversees the site license for EndNote.

To request installation:   The library provides EndNote  to CDC staff under a site-wide license. Please use the ITSO Software Request Tool (SRT) and submit a request for the latest version (or upgraded version) of EndNote. Please be sure to include the computer name for the workstation where you would like to have the software installed.

EndNote Training:   CDC Library offers training on EndNote on a regular basis – both a basic and advanced course. To view the course descriptions and upcoming training dates, please visit the CDC Library training page .

For assistance with EndNote software, please contact [email protected]

Vendor Support and Services:   EndNote – Support and Services (Thomson Reuters)  EndNote – Tutorials and Live Online Classes (Thomson Reuters)

Getting Articles:

Articles can be obtained using DocExpress or by searching the electronic journals at the Stephen B. Thacker CDC Library.

IOM Standards for Systematic Reviews: Standard 3.1: Conduct a comprehensive systematic search for evidence

The goal of a systematic review search is to maximize recall and precision while keeping results manageable. Recall (sensitivity) is defined as the number of relevant reports identified divided by the total number of relevant reports in existence. Precision (specificity) is defined as the number of relevant reports identified divided by the total number of reports identified.

Issues to consider when creating a systematic review search:   

• All concepts are included in the strategy
• All appropriate subject headings are used
• Appropriate use of explosion
• Appropriate use of subheadings and floating subheadings
• Use of natural language (text words) in addition to controlled vocabulary terms
• Use of appropriate synonyms, acronyms, etc.
• Truncation and spelling variation as appropriate
• Appropriate use of limits such as language, years, etc.
• Field searching, publication type, author, etc.
• Boolean operators used appropriately
• Line errors: when searches are combined using line numbers, be sure the numbers refer to the searches intended
• Check indexing of relevant articles
• Search strategy adapted as needed for multiple databases

For more information on how librarians can assist you with systematic review searching, please check out the following presentation slides given by CDC librarians in 2015 about librarians as collaborators in systematic reviews: Course_LibrarianasCollaborator_SystematicReviews_2015 [PDF – 2 MB]

• Cochrane Handbook: Searching for Studies See Part 2, Chapter 6

A step-by-step guide to systematically identify all relevant animal studies

Materials listed in these guides are selected to provide awareness of quality public health literature and resources. A material’s inclusion does not necessarily represent the views of the U.S. Department of Health and Human Services (HHS), the Public Health Service (PHS), or the Centers for Disease Control and Prevention (CDC), nor does it imply endorsement of the material’s methods or findings. HHS, PHS, and CDC assume no responsibility for the factual accuracy of the items presented. The selection, omission, or content of items does not imply any endorsement or other position taken by HHS, PHS, and CDC. Opinion, findings, and conclusions expressed by the original authors of items included in these materials, or persons quoted therein, are strictly their own and are in no way meant to represent the opinion or views of HHS, PHS, or CDC. References to publications, news sources, and non-CDC Websites are provided solely for informational purposes and do not imply endorsement by HHS, PHS, or CDC.

Exit Notification / Disclaimer Policy

• The Centers for Disease Control and Prevention (CDC) cannot attest to the accuracy of a non-federal website.
• Linking to a non-federal website does not constitute an endorsement by CDC or any of its employees of the sponsors or the information and products presented on the website.
• You will be subject to the destination website's privacy policy when you follow the link.
• CDC is not responsible for Section 508 compliance (accessibility) on other federal or private website.

Help us improve our Library guides with this 5 minute survey . We appreciate your feedback!

• UOW Library
• Key guides for researchers

Systematic Review

What is a systematic review.

• Five other types of systematic review
• How is a literature review different?
• Search tips for systematic reviews
• Controlled vocabularies
• Grey literature
• Transferring your search
• Documenting your results
• Support & contact

A systematic review is an authoritative account of existing evidence using reliable, objective, thorough and reproducible research practices.

It is a method of making sense of large bodies of information and contributes to the answers to questions about what works and what doesn't.

Systematic reviews map areas of uncertainty and identify where little or no relevant research has been done, but where new studies are needed.

It is a good idea to familiarise yourself with the systematic review process before beginning your review. You can do this by searching for other systematic reviews to look at as examples, by  reading a glossary of commonly used terms , and by learning how to distinguish between  types of systematic review.

Characteristics of a systematic review

Some characteristics, or features, of systematic reviews are:

• Clearly stated set of objectives with pre-defined eligibility criteria
• Explicit, reproducible methodology
• A systematic search that attempts to identify all studies that would meet the eligibility criteria
• Assesses the validity of the findings, for example assessing the risk of bias
• Systematic presentation and synthesis of the findings of the included studies. (Cochrane Handbook for Systematic Reviews of Interventions, 2008, p. 6).

Watch this video from the Cochrane Library for more information about systematic reviews.

• Previous: Introduction to systematic reviews
• Next: Five other types of systematic review
• Last Updated: Oct 23, 2023 1:59 PM
• URL: https://uow.libguides.com/systematic-review

Insert research help text here

LIBRARY RESOURCES

Library homepage

Library SEARCH

A-Z Databases

STUDY SUPPORT

Learning Co-Op (academic skills support)

Referencing and citing

Digital Skills Hub

MORE UOW SERVICES

UOW homepage

Student support and wellbeing

IT Services

On the lands that we study, we walk, and we live, we acknowledge and respect the traditional custodians and cultural knowledge holders of these lands.

Copyright & disclaimer | Privacy & cookie usage

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 14, Issue 3
• What is a systematic review?
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Jane Clarke
• Correspondence to Jane Clarke 4 Prime Road, Grey Lynn, Auckland, New Zealand; janeclarkehome{at}gmail.com

http://dx.doi.org/10.1136/ebn.2011.0049

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

A high-quality systematic review is described as the most reliable source of evidence to guide clinical practice. The purpose of a systematic review is to deliver a meticulous summary of all the available primary research in response to a research question. A systematic review uses all the existing research and is sometime called ‘secondary research’ (research on research). They are often required by research funders to establish the state of existing knowledge and are frequently used in guideline development. Systematic review findings are often used within the healthcare setting but may be applied elsewhere. For example, the Campbell Collaboration advocates the application of systematic reviews for policy-making in education, justice and social work.

Systematic reviews can be conducted on all types of primary research. Many are reviews of randomised trials (addressing questions of effectiveness), cross-sectional studies (addressing questions about prevalence or diagnostic accuracy, for example) or cohort studies (addressing questions about prognosis). When qualitative research is reviewed systematically, it may be described as a systematic review, but more often other terms such as meta-synthesis are used.

Systematic review methodology is explicit and precise and aims to minimise bias, thus enhancing the reliability of the conclusions drawn. 1 , 2 The features of a systematic review include:

■ clear aims with predetermined eligibility and relevance criteria for studies;

■ transparent, reproducible methods;

■ rigorous search designed to locate all eligible studies;

■ an assessment of the validity of the findings of the included studies and

■ a systematic presentation, and synthesis, of the included studies. 3

The first step in a systematic review is a meticulous search of all sources of evidence for relevant studies. The databases and citation indexes searched are listed in the methodology section of the review. Next, using predetermined reproducible criteria to screen for eligibility and relevance assessment of titles and the abstracts is completed. Each study is then assessed in terms of methodological quality.

Finally, the evidence is synthesised. This process may or may not include a meta-analysis. A meta-analysis is a statistical summary of the findings of independent studies. 4 Meta-analyses can potentially present more precise estimates of the effects of interventions than those derived from the individual studies alone. These strategies are used to limit bias and random error which may arise during this process. Without these safeguards, then, reviews can mislead, such that we gain an unreliable summary of the available knowledge.

The Cochrane Collaboration is a leader in the production of systematic reviews. Cochrane reviews are published on a monthly basis in the Cochrane Database of Systematic Reviews in The Cochrane Library (see: http://www.thecochranelibrary.com ).

• Antman EM ,
• Kupelnick B ,
• Higgins JPT ,

Competing interests None.

Read the full text or download the PDF:

Systematic Reviews

What is a systematic review.

• Types of Systematic Reviews
• Assembling Your Research Team
• Step 1: Identify Your Research Question
• Step 2: Define Inclusion/Exclusion Criteria
• Step 3: Search the Literature
• Step 4: Selecting Studies For Inclusion
• Step 5: Quality Assessment & Data Extraction
• PRISMA Checklist and Flow Diagram
• Helpful Documents & Links
• Help From the Library

"A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. The key characteristics of a systematic review are: a clearly defined question with inclusion & exclusion criteria; rigorous & systematic search of the literature; critical appraisal of included studies; data extraction and management; analysis & interpretation of results; and report for publication."   --  Duke University Medical Center Library & Archives

A systematic review differs from a traditional literature review or narrative review, in that it aims to be as thorough and unbiased as possible, and also provides detailed information about how the studies were identified and why they were included.

Prior to starting your own research, you will want to look at existing systematic reviews - this is especially important so that you don't duplicate existing work. It can also be helpful to look at the approaches taken for systematic reviews similar to your own topic or discipline.  You can find existing systematic reviews a number of ways: 

• They can published as journal articles – to identify them, add "systematic review" as an additional search term in databases, or look for limits if available. Here’s an example of a  systematic review  published as a journal article
• Cochrane Database of Systematic Reviews : includes the full text of the regularly updated systematic reviews of the effects of healthcare prepared by The Cochrane Collaboration.
• PROSPERO  is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome. PROSPERO aims to provide a comprehensive listing of systematic reviews registered at inception to help avoid duplication and reduce opportunity for reporting bias by enabling comparison completed review with what was planned in the protocol.
• The Campbell Collaboration  is an international network which publishes high quality systematic reviews of social and economic interventions around the world

Below are some books available at the University Library discussing systematic reviews:

• Next: Types of Systematic Reviews >>
• Last Updated: Sep 12, 2023 3:51 PM
• URL: https://franu.libguides.com/systematicreviews

How to write a systematic review

Affiliations.

• 1 The Methodist Orthopedics and Sports Medicine Center, Houston, Texas [email protected].
• 2 Sports Medicine Center, The Ohio State University, Columbus, Ohio Department of Orthopaedic Surgery, The Ohio State University, Columbus, Ohio.
• 3 Department of Orthopaedic Surgery, The Ohio State University, Columbus, Ohio.
• 4 Sports Medicine Center, The Ohio State University, Columbus, Ohio Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, Ohio.
• PMID: 23925575
• DOI: 10.1177/0363546513497567

Background: The role of evidence-based medicine in sports medicine and orthopaedic surgery is rapidly growing. Systematic reviews and meta-analyses are also proliferating in the medical literature.

Purpose: To provide the outline necessary for a practitioner to properly understand and/or conduct a systematic review for publication in a sports medicine journal.

Study design: Review.

Methods: The steps of a successful systematic review include the following: identification of an unanswered answerable question; explicit definitions of the investigation's participant(s), intervention(s), comparison(s), and outcome(s); utilization of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines and PROSPERO registration; thorough systematic data extraction; and appropriate grading of the evidence and strength of the recommendations.

Results: An outline to understand and conduct a systematic review is provided, and the difference between meta-analyses and systematic reviews is described. The steps necessary to perform a systematic review are fully explained, including the study purpose, search methodology, data extraction, reporting of results, identification of bias, and reporting of the study's main findings.

Conclusion: Systematic reviews or meta-analyses critically appraise and formally synthesize the best existing evidence to provide a statement of conclusion that answers specific clinical questions. Readers and reviewers, however, must recognize that the quality and strength of recommendations in a review are only as strong as the quality of studies that it analyzes. Thus, great care must be used in the interpretation of bias and extrapolation of the review's findings to translation to clinical practice. Without advanced education on the topic, the reader may follow the steps discussed herein to perform a systematic review.

Keywords: PRISMA; PROSPERO; evidence-based medicine; meta-analysis; systematic review.

© 2013 The Author(s).

Publication types

• Systematic Review
• Evidence-Based Medicine
• Medical Writing / standards*
• Meta-Analysis as Topic
• Orthopedics*
• Publishing / standards*
• Review Literature as Topic*
• Sports Medicine*

• Search by keyword
• Search by citation

Page 1 of 55

Interprofessional diagnostic management teams: a scoping review protocol

Diagnostic errors are a major problem in healthcare. In 2015, the report “Improving Diagnosis in Health Care” by the National Academies of Sciences, Engineering, and Medicine (NASEM) stated that it is likely t...

• View Full Text

Quality of measurement properties of medication adherence instruments in cardiovascular diseases and type 2 diabetes mellitus: a systematic review and meta-analysis

Medication adherence has a major impact on reducing mortality and healthcare costs related to the treatment of cardiovascular diseases and diabetes mellitus. Selecting the best patient-reported outcome measure...

Resuscitation-associated endotheliopathy (RAsE): a conceptual framework based on a systematic review and meta-analysis

Shock-induced endotheliopathy (SHINE), defined as a profound sympathoadrenal hyperactivation in shock states leading to endothelial activation, glycocalyx damage, and eventual compromise of end-organ perfusion...

Maternal, newborn, and children under-five health surveillance system: a scoping review protocol

Public health surveillance is crucial in monitoring the progress of maternal, newborn, and children under-five health outcomes (MNCH). Consequently, mapping the existing surveillance system from countries with...

Prehabilitation programs for individuals with cancer: a systematic review of randomized-controlled trials

Prehabilitation programs focusing on exercise training as the main component are known as a promising alternative for improving patients’ outcomes before cancer surgery. This systematic review determined the b...

Correction of symbrachydactyly: a systematic review of surgical options

Symbrachydactyly is a rare congenital malformation of the hand characterized by short or even absent fingers with or without syndactyly, mostly unilaterally present. The hand condition can vary from a small ha...

Associations between nature exposure, screen use, and parent–child relations: a scoping review protocol

Today’s youth are growing up in an evolving digital world, and concerns about the potential detrimental effects of excessive screen use on biopsychosocial outcomes in childhood are mounting. Parents worry abou...

Does haste make waste? Prevalence and types of errors reported after publication of studies of COVID-19 therapeutics

The COVID-19 pandemic spurred publication of a rapid proliferation of studies on potential therapeutic agents. While important for the advancement of clinical care, pressure to collect, analyze, and report dat...

Efficacy of nano encapsulated herbal extracts in the treatment of induced wounds in animal models: a systematic review protocol

Wounds inflict pain and affect human health causing high expenditure on treatment and management. Herbal crude extracts are used in traditional medicine as a treatment for wounds and other illnesses. However, ...

Mode of HIV exposure and excess burden of neurocognitive impairment in people living with HIV: a protocol for systematic review and meta-analysis of controlled studies

Chronic HIV infection significantly elevates the risk of brain pathology, precipitating neurocognitive impairment (NCI) among people living with HIV (PLWH). The diagnosis of NCI in PLWH hinges on evaluating de...

Correction: Gestational diabetes mellitus, follow-up of future maternal risk of cardiovascular disease and the use of eHealth technologies – a scoping review

The original article was published in Systematic Reviews 2023 12 :178

Global prevalence of COVID-19-induced acute respiratory distress syndrome: systematic review and meta-analysis

Acute respiratory distress syndrome (ARDS) is potentially a fatal form of respiratory failure among COVID-19 patients. Globally, there are inconsistent findings regarding ARDS among COVID-19 patients. Therefor...

Title-plus-abstract versus title-only first-level screening approach: a case study using a systematic review of dietary patterns and sarcopenia risk to compare screening performance

Conducting a systematic review is a time- and resource-intensive multi-step process. Enhancing efficiency without sacrificing accuracy and rigor during the screening phase of a systematic review is of interest...

Preoperative education in patients undergoing foot and ankle surgery: a scoping review

International guidelines promote preoperative education for patients undergoing orthopedic surgery. However, the evidence sustaining these recommendations comes mainly from studies for hip and knee replacement...

Estimating and visualising the trade-off between benefits and harms on multiple clinical outcomes in network meta-analysis

The relative treatment effects estimated from network meta-analysis can be employed to rank treatments from the most preferable to the least preferable option. These treatment hierarchies are typically based o...

Photoactivated chromophore-corneal cross-linking accelerates corneal healing in fungal keratitis: an updated meta-analysis

To determine the effectiveness and safety of photoactivated chromophore-corneal cross-linking (PACK-CXL) adjuvant in infectious keratitis by April 5, 2022.

Beyond the bedside: protocol for a scoping review exploring the experiences of non-practicing healthcare professionals within health professions education

The number of healthcare professionals leaving clinical practice and transitioning to alternative careers in health professions education is increasing. Among these non-practicing healthcare professionals, con...

Correction: What can the citations of systematic reviews of ethical literature tell us about their use?—an explorative empirical analysis of 31 reviews

The original article was published in Systematic Reviews 2023 12 :173

Efficacy of pharmacological and non-pharmacological therapy on pain intensity and disability of older people with chronic nonspecific low back pain: a protocol for a network meta-analysis

The purpose of this network meta-analysis (NMA) is to investigate the efficacy of pharmacological and non-pharmacological therapy on pain intensity and disability of older people with chronic nonspecific low b...

The influence of assistive technology and home modifications on falls in community-dwelling older adults: a systematic review protocol

Fall-related injuries can reduce older adults’ independence and result in economic burdens. The assistive technologies and home modifications explored in this review are suggested to reduce the risk of falls o...

Trauma-informed prevention programmes for depression, anxiety, and substance use among young people: protocol for a mixed-methods systematic review

Mental ill-health and substance use bear a substantial burden and harm on young people and often arise from co-occurring and compounding risk factors, such as traumatic stress. Trauma-informed prevention of me...

Multivariable prediction models of caries increment: a systematic review and critical appraisal

Multivariable prediction models are used in oral health care to identify individuals with an increased likelihood of caries increment. The outcomes of the models should help to manage individualized interventi...

Systematic review and meta-analysis: the efficiency of bacteriophages previously patented against pathogenic bacteria on food

Food-borne diseases are a global public health issue with 1 in 10 people falling ill after eating contaminated food every year. In response, the food industry has implemented several new pathogen control strat...

Effects and mechanisms of Tai Chi on mild cognitive impairment and early-stage dementia: a scoping review

Dementia is associated with cognitive and functional decline that significantly impacts quality of life. There is currently no cure for dementia, thus, it is important to manage dementia in the early stages an...

Peer support interventions in maternal and child healthcare delivery in sub-Saharan Africa: protocol for a realist review

Peer support has been proposed as a promising policy intervention for addressing adverse maternal and child healthcare (MCH) outcomes in sub-Saharan Africa (SSA). Existing reviews on peer support largely draw ...

Interventions to prevent unintended pregnancies among adolescents: a rapid overview of systematic reviews

Risks associated with unintended pregnancy include unsafe abortions, poor maternal health-seeking behaviour, poor mental health, and potentially, maternal and infant deaths. Adolescent girls with unintended pr...

Dual antiplatelet management in the perioperative period: updated and expanded systematic review

Antiplatelet agents are central in the management of vascular disease. The use of dual antiplatelet therapy (DAPT) for the management of thromboembolic complications must be weighed against bleeding risk in th...

The PRISMATIC project: protocol for a research programme on novel methods to improve reporting and peer review of systematic reviews of health evidence

Incomplete reporting about what systematic reviewers did and what they found prevents users of the report from being able to fully interpret the findings and understand the limitations of the underlying eviden...

Graduate and postgraduate educational challenges during the COVID-19 pandemic period: its impact and innovations—a scoping review

The coronavirus disease 2019 (COVID-19) pandemic has transformed the global view of education, including graduate and postgraduate education making the development of an alternative approach in times of social...

Relational practice in health, education, criminal justice, and social care: a scoping review

Establishing and maintaining relationships and ways of connecting and being with others is an important component of health and wellbeing. Harnessing the relational within caring, supportive, educational, or c...

A protocol for ongoing systematic scoping reviews of World Trade Center Health research

The World Trade Center (WTC) Health Program (“Program”) seeks to assess the inventory, quality, and impact of its funded research in the context of all clinical and translational research involving WTC populat...

Experienced barriers in the use of ICT for social interaction in older adults ageing in place: a qualitative systematic review protocol (SYSR-D-22–00848)

Several information and communication technologies (ICT) have been developed to enhance social connectedness of older adults aging in place, although they are not accessible for all. Barriers in using ICT migh...

Efficacy of homoeopathic treatment: Systematic review of meta-analyses of randomised placebo-controlled homoeopathy trials for any indication

Since 1997, several meta-analyses (MAs) of placebo-controlled randomised efficacy trials of homoeopathy for any indication (PRETHAIs) have been published with different methods, results and conclusions. To dat...

Implementation strategies, facilitators, and barriers to scaling up and sustaining task-sharing in family planning: a protocol for a mixed-methods systematic review

Ensuring access to quality family planning (FP) services is fundamental to achieving the Sustainable Development Goals (SDG) targets 3.1, 3.7, and 5.6, including universal access to reproductive health service...

Diversity when interpreting evidence in network meta-analyses (NMAs) on similar topics: an example case of NMAs on diabetic macular oedema

Different network meta-analyses (NMAs) on the same topic result in differences in findings. In this review, we investigated NMAs comparing aflibercept with ranibizumab for diabetic macular oedema (DME) in the ...

Exploring the impact of evaluation on learning and health innovation sustainability: protocol for a realist synthesis

Within the Learning Health System (LHS) model, learning routines, including evaluation, allow for continuous incremental change to take place. Within these learning routines, evaluation assists in problem iden...

Evaluation of a prototype machine learning tool to semi-automate data extraction for systematic literature reviews

Evidence-based medicine requires synthesis of research through rigorous and time-intensive systematic literature reviews (SLRs), with significant resource expenditure for data extraction from scientific public...

Factors influencing emotional support of older adults living in the community: a scoping review protocol

Emotional support is key to improve older adults’ subjective health, and psychological, social and emotional well-being. However, many older adults living in the community lack emotional support, increasing th...

Definitions, terminology, and related concepts of “racial health equity”: a scoping review protocol

In the USA, access to quality healthcare varies greatly across racial and ethnic groups, resulting in significant health disparities. A new term, “racial health equity” (RHE), is increasingly reported in the m...

Identity development, attraction, and behaviour of heterosexually identified men who have sex with men: scoping review protocol

Heterosexually identified men who have sex with men (H-MSM) are distinct from other heterosexual men and from gay, bisexual, and other sexual minority men. Specifically, H-MSM experience discordance between th...

Patient-reported outcome (PRO) instruments used in patients undergoing adoptive cell therapy (ACT) for the treatment of cancer: a systematic review

Adoptive cell therapy (ACT) is a rapidly evolving field. Patient-reported outcomes (PROs) allow patients to report the impact of treatment on their quality of life during and after treatment. The systematic re...

The effectiveness of nurse-led interventions to manage frailty in community-dwelling older people: a systematic review

The global increase in the number of frail older people and the accompanying increase in chronic conditions underline the need to develop effective health promotion and preventive interventions for these popul...

Prospective longitudinal associations between adverse childhood experiences and adult mental health outcomes: a protocol for a systematic review and meta-analysis

Research cites a strong, dose–response relationship between adverse childhood experiences (ACEs) and poor adult mental health outcomes including anxiety, depression, post-traumatic stress disorder (PTSD), self...

Systematic scoping review protocol of Stroke Patient and Stakeholder Engagement (SPSE)

This protocol describes a systematic scoping review of Stroke Patient and Stakeholder Engagement (SPSE), concepts, definitions, models, implementation strategies, indicators, or frameworks. The active engageme...

Saturated fat, the estimated absolute risk and certainty of risk for mortality and major cancer and cardiometabolic outcomes: an overview of systematic reviews

To assess the impact of reducing saturated fat or fatty foods, or replacing saturated fat with unsaturated fat, carbohydrate or protein, on the risk of mortality and major cancer and cardiometabolic outcomes i...

Gestational diabetes mellitus, follow-up of future maternal risk of cardiovascular disease and the use of eHealth technologies—a scoping review

Globally, gestational diabetes mellitus complicates 1 in 6 pregnancies and increases future risk of type 2 diabetes and cardiovascular disease in the affected women. There is a lack of consensus on the optimal...

The Correction to this article has been published in Systematic Reviews 2023 12 :213

Neuroprotective effect and possible mechanism of edaravone in rat models of spinal cord injury: a protocol for a systematic review and meta-analysis

Spinal cord injury (SCI) is one of the most disabling neurological conditions, afflicting thousands of human beings. Edaravone, a well-known reactive oxygen species scavenger, is expanding its new scope in fie...

Women’s experiences with yoga after a cancer diagnosis: a qualitative meta-synthesis—part I

Qualitative research on women’s experiences participating in yoga after a cancer diagnosis is growing; systematic synthesis and integration of results are necessary to facilitate the transfer and implementatio...

Conflict of interest and risk of bias in systematic reviews on methylphenidate for attention-deficit hyperactivity disorder: a cross-sectional study

Systematic reviews (SRs) are pivotal to evidence-based medicine, yet there is limited research on conflicts of interest in SRs. Our aim was to investigate financial conflicts of interest and risk of bias (RoB)...

Clinical practice guideline recommendations for diagnosis and management of anxiety and depression in hospitalized adults with delirium: a systematic review

Delirium commonly occurs in hospitalized adults. Psychiatric disorders such as anxiety, depression, and post-traumatic stress disorder (PTSD) can co-occur with delirium, and can be recognized and managed by cl...

• Editorial Board
• Manuscript editing services
• Instructions for Editors
• Sign up for article alerts and news from this journal
• Follow us on Twitter

Annual Journal Metrics

2022 Citation Impact 3.7 - 2-year Impact Factor 3.8 - 5-year Impact Factor 1.561 - SNIP (Source Normalized Impact per Paper) 1.269 - SJR (SCImago Journal Rank)

2022 Speed 153 days submission to first editorial decision for all manuscripts (Median) 340 days submission to accept (Median)

2022 Usage  3,104,570 downloads 15,353 Altmetric mentions 

• More about our metrics

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 08 November 2023

Childhood vaccine refusal and what to do about it: a systematic review of the ethical literature

• Kerrie Wiley 1 ,
• Maria Christou-Ergos 1 ,
• Chris Degeling 2 ,
• Rosalind McDougall 3 ,
• Penelope Robinson 1 ,
• Katie Attwell 4 ,
• Catherine Helps 1 ,
• Shevaun Drislane 4 &
• Stacy M Carter 2  

BMC Medical Ethics volume  24 , Article number:  96 ( 2023 ) Cite this article

666 Accesses

9 Altmetric

Metrics details

Parental refusal of routine childhood vaccination remains an ethically contested area. This systematic review sought to explore and characterise the normative arguments made about parental refusal of routine vaccination, with the aim of providing researchers, practitioners, and policymakers with a synthesis of current normative literature.

Nine databases covering health and ethics research were searched, and 121 publications identified for the period Jan 1998 to Mar 2022. For articles, source journals were categorised according to Australian Standard Field of Research codes, and normative content was analysed using a framework analytical approach.

Most of the articles were published in biomedical journals (34%), bioethics journals (21%), and journals that carry both classifications (20%). Two central questions dominated the literature: (1) Whether vaccine refusal is justifiable (which we labelled ‘refusal arguments’); and (2) Whether strategies for dealing with those who reject vaccines are justifiable (‘response arguments’). Refusal arguments relied on principlism, religious frameworks, the rights and obligations of parents, the rights of children, the medico-legal best interests of the child standard, and the potential to cause harm to others. Response arguments were broadly divided into arguments about policy, arguments about how individual physicians should practice regarding vaccine rejectors, and both legal precedents and ethical arguments for vaccinating children against a parent’s will. Policy arguments considered the normative significance of coercion, non-medical or conscientious objections, and possible reciprocal social efforts to offset vaccine refusal. Individual physician practice arguments covered nudging and coercive practices, patient dismissal, and the ethical and professional obligations of physicians. Most of the legal precedents discussed were from the American setting, with some from the United Kingdom.

Conclusions

This review provides a comprehensive picture of the scope and substance of normative arguments about vaccine refusal and responses to vaccine refusal. It can serve as a platform for future research to extend the current normative literature, better understand the role of cultural context in normative judgements about vaccination, and more comprehensively translate the nuance of ethical arguments into practice and policy.

Peer Review reports

Introduction

Vaccine rejection has existed for as long as vaccines [ 1 ]. Despite the significant contribution of childhood vaccination to reductions in global child morbidity and mortality [ 2 ], some parents continue to reject vaccines for their children. Parents’ reasons for rejection vary widely, and often depend on their social settings. For example, in high-income settings where around 2–3% of parents reject routine childhood vaccines [ 3 , 4 ], reasons can include previous bad experiences with vaccines or the medical system, concerns about vaccine safety, doubt about the effectiveness or necessity of vaccines, alternative health approaches, and participation in particular social groups or communities. These reasons can be grounded in deeply held religious beliefs or general philosophical approaches to health, views on freedom of choice, or mistrust in government and/or the vested interests of vaccine producers, among other things [ 5 , 6 , 7 , 8 ].

Vaccination plays a dual role in disease prevention: it serves to protect the vaccinated individual from disease, and when vaccination rates reach a high enough threshold for some diseases, also protects the broader community—including those who remain unvaccinated—by disrupting disease transmission through herd immunity. This dual role of vaccination, providing benefit to both the individual and community, complicates ethical questions regarding vaccine refusal, specifically, whether vaccine rejection is ethically justifiable.

Health care providers, communities, and governments encourage uptake and discourage vaccine rejection by various means, and the dual role of vaccination is also relevant to an evaluation of these practice and policy responses. Vaccine acceptance is encouraged with interventions like incentives, health provider recommendations and “nudges” directed at individual families, as well as by facilitating easier access to vaccination through strategies such as cost reduction and making clinic locations and opening times convenient, with many of these interventions supported by varying levels of evidence [ 9 ]. Governments often discourage vaccine rejection via the imposition of mandates that can vary in type and severity [ 10 ] and are not always well-supported by evidence [ 11 ]. These can include punitive measures, such as limiting unvaccinated children’s access to early childhood education or daycare. A thorough understanding of the ethical dimensions of childhood vaccine rejection and responses to it is important when navigating vaccine rejection in the clinical setting, and when formulating policy [ 12 ]. Systematic reviews of the evidence are considered best practice for informing vaccine practice and policy however, to our knowledge there have not yet been any published systematic reviews of the literature on the ethics of childhood vaccine rejection despite there being a broad literature on the subject. We sought to systematically explore and characterise the normative arguments made about parental refusal of routine vaccination, with the aim of better informing vaccine policy and practice.

We searched nine databases for literature that discussed normative positions on childhood vaccine rejection. Refer to the PRISMA flow chart (Fig.  1 .)

PRIMSA Flow Diagram of Review

Search strategy

We searched Medline, Embase, Philosophers Index, Philpapers, Project Muse, Cinahl, The Global Digital Library on Ethics (globethics.net), The Bioethics Literature Database (BELIT), and Pubmed using the general search strategy listed in Fig.  2 for articles published between January 1998 and March 2022.

Inclusion criteria

We included any publication which provided a substantive normative argument about parental refusal of routine vaccines for children aged five and under. We used a broad definition of ‘normative’ to mark anything that goes beyond mere description to consider right and wrong, good and bad, justifiable and unjustifiable, or legitimate and illegitimate actions or ways of being in the world. Our broad conception included textual forms such as ethical reflections, prudential and legal norms, and accounts of rationality. We used ’substantive’ to mark publications where the authors’ main purpose was to make an argument about whether vaccine refusal is morally justifiable. This included empirical research that explicitly examined normative dimensions of vaccine refusal. We were limited to reviewing publications published in English.

Exclusion criteria

We excluded publications where authors made a normative claim in passing, but the publication’s main purpose was to report non-normative empirical findings. We also excluded: publications on adult vaccination (including COVID vaccination) and the HPV vaccine (which is administered in adolescence, not childhood); empirical research such as surveys or interviews, unless they expressly explored normative arguments; and descriptive publications about the characteristics of the anti-vaccination movement that provided no normative position.

Screening and data extraction

After search execution and duplicate removal, a screening triangulation exercise was undertaken to ensure consistency among the screeners. A set of 20 titles and abstracts were independently screened by six authors, and the results compared. The inclusion and exclusion criteria were refined in a subsequent group discussion, and a sub-set of full text articles were then screened and evaluated by the same group of people, and results again compared. A discussion of this second triangulation step resulted in a refined and standardized screening approach.

The authorship group were then divided into four pairs, and the remaining titles and abstracts divided among the pairs. Each individual screened titles and abstracts against inclusion criteria, and then met with their screening partner to compare results and discuss and resolve any differences.

Full text was sought for each record screened for inclusion, and a second screening then removed articles which didn’t meet the inclusion criteria once the full text was read, articles that could not be sourced, and duplicates not identified in the initial screening.

The final list of full text publications was then divided among four authors (SC, RM, CD and KW) for data extraction using the concept of “information units” described by Mertz and colleagues [ 13 ]. In this context an information unit was defined as a normative issue or argument, and each of the four ‘extracting’ authors summarized each of the relevant information units in the papers they were assigned.

For included journal articles, Australian Standard Field of Research (FoR) codes for the journal that each article appeared in were sourced as a proxy for the disciplinary location of the article (e.g. bioethics, medicine, law). We used the Australian and New Zealand Standard Research Classification (ANZSRC) 2008, as this was the current standard when analysis commenced [ 14 ]. We used two digit FoR codes (division codes) to identify the source journal as either being Medical and Health Sciences (code 11), Ethics and Philosophy (code 22) Law (code 18) or other codes grouped as “other”. In some cases, the journal was assigned a combination of these codes (refer to Fig.  3 ).

Respective percentages of included articles falling under various ANZSRC FoR Codes (2008)

Quality assessment in systematic reviews of normative literature remains a contested area, with various options and no established best practice approach [ 15 ]. In this review, we took a satisficing approach to quality appraisal [ 16 ]: publication in a peer-reviewed journal or by a reputable academic publisher was taken as a sufficient level of quality to justify inclusion in the review. The peer review process undergone by PhD theses was also taken to be a sufficient indictor of quality to justify inclusion. Further quality appraisal of individual publications was not undertaken. This aligns with the purpose of the review which was to map and synthesize the current literature on this topic.

A framework approach was used to organise and synthesise the data [ 17 ]. The extracted information units were read by one author (KW), and a coding frame inductively developed to summarise and classify the information units extracted by the group. The publications were then independently coded according to this framework by two authors (KW and PR). Following this, the two authors met and compared their coding, discussing any differences and resolving them by consensus. The data were then synthesized into themes. In addition, for journal articles, the ANZSRC Field of Research codes for the journal each article appeared in were descriptively analysed to assess the distribution of the included literature across various disciplines.

Search results

Five thousand, two hundred and thirty-one publications were returned by the searches (see Fig.  1 ). Eight hundred and twenty-two duplicates were removed in the first instance, leaving 4409 records to be screened by title and abstract. During this screening process 4058 records were excluded, leaving 351 full text publications to be assessed. Of these a further 230 records were excluded (due to not meeting the inclusion criteria, previously unidentified duplicates, or inability to source the full text), leaving 121 publications for inclusion in the review. These included 117 journal articles, three theses and one book.

Literature source type

Analysis of the ANZSRC Field of Research codes of the source journals of included articles revealed three main areas, or a combination of them (Fig.  3 ). Around half were coded to medicine (63%); of these, just over half were dual coded to ethics (20%) or another code (9%). 21% of articles were from the philosophy or ethics literature alone; another 25% were from ethics and medicine or ethics and law. Law was the least dominant discipline, with only 12% of articles being coded to law (alone or in combination with other disciplines). This pattern suggests active concern within medicine regarding non-vaccination, but also widespread overlap in concern between medicine, ethics, and law.

Main themes found in the literature

Articles addressed two central questions (see Table  1 ):

Whether vaccine refusal was justified (henceforth ‘refusal’ arguments).

Whether various policy or practice responses to those who reject vaccines are justified (henceforth ‘response’ arguments).

Descriptive analysis of content

The literature was dominated by papers focused on ‘response’ arguments (61%). A smaller group of papers address ‘refusal’ arguments (19%), and about 18% considered both ‘refusal’ and ‘response’, usually making normative arguments about vaccine refusal as background to arguments regarding ‘response’ (See Fig.  4 ). Less than 2% of papers had a different focus.

Comparative frequencies of themes occurring among included articles

‘Response’ arguments were more common in the medical and health sciences literature (ERA FoR code 11, see Fig.  5 ). Although the ethics/philosophy (FoR code 22) and law literatures (FoR code 18) were also dominated by ‘response’ arguments, these journals—unlike medical journals—were more likely to include ‘refusal’ arguments.

Comparative frequency of overarching themes across the different disciplines of the included articles

As would be expected, authors made ‘response’ and ‘refusal’ arguments in different ways. In the following sections we consider the detail of how arguments were made. We refer to each included article by its unique reference listed in Table  1 .

‘Refusal’ arguments: whether or not vaccine rejection by individual parents is justifiable

Arguments about whether vaccine refusal by individual parents is justifiable included consideration of parents’ rights, the interests of the child (including the legal ‘best interests of the child standard’), the value of herd immunity, the epistemic basis for ethical claims, and the relevance of religious views. Our sampling period included a special issue of Narrative Inquiry in Bioethics which published narratives written by parents to communicate their normative positions on vaccination. Most of these were written by non-vaccinating parents, and they make up over one third of all arguments in the identified literature that support refusal. On balance, most of the literature argues that it is not justifiable for parents to refuse routine vaccination for their children.

Some arguments within the literature were absolute in their position on whether vaccine rejection is justifiable; others weighed competing values in a situation-specific approach. Irrespective of the arguments used to justify a position, most of the literature frames the question of whether vaccine rejection is justifiable based on three key areas of concern: (i) Respect for autonomy, the doctrine of informed consent and the value of liberty, (ii) Consequences for the child and others, and/or (iii) The normative significance of parental trust, distrust, and uncertainty. We explore the main arguments within these concepts below. As the discussion shows, these concepts are not discrete – they are often weighed against one another, linked by causal claims, or held in tension in the arguments made. Figure  6 represents proportionally the ’refusal’ arguments made in the reviewed literature.

‘Refusal’ arguments made in the literature on the ethics of vaccine refusal

Respect for autonomy, the doctrine of informed consent and the value of liberty

Fifteen papers from this sample present arguments that vaccine refusal is justified based on respect for parental autonomy, rights, or liberties (21, 23, 25, 31, 32, 35, 36, 39, 68, 71, 75, 80, 94, 100, 121). Some argue that vaccine refusal is justified on the basis of preserving legal rights (31, 80) or expression of religious freedom [ 23 ]. Opposing positions (including from four of the authors who also offer arguments justifying refusal) argue that, on balance, considerations regarding respect for autonomy are, or can be, outweighed by the potential harm caused to the child and others by not vaccinating though the increased risk of vaccine preventable diseases (21, 36, 20, 23, 110). This includes legal perspectives arguing that the freedom to choose is not unfettered [ 25 ] and that courts can override parental autonomy if this is in the child’s best interest (75, 85), as well as arguments from religious perspectives that the freedom to exercise religious beliefs needs to be weighed against harm caused to others (21,91). Those who argue that vaccine refusal is justified counter that disrespecting parental autonomy can also cause harm to the child through loss of trust and possible disengagement of the child from the healthcare system (100), and that the increased risk of disease is a price worth paying to ensure that political values are preserved (71). Of note: non-vaccinating parents also assert a right to make choices for their children in support of their refusal [ 14 , 18 ], but unlike others, their arguments are based primarily on epistemic claims about vaccine effectiveness, necessity and safety rather than moral or ethical positions. However, they assert that these doubts necessitate respect for their decision.

Consequences for others and the child

Most of the literature argues for or against the justifiability of vaccine refusal based on consequences. These include potential harms from vaccine preventable diseases or vaccines themselves, or conversely, potential benefits from herd immunity. The concept of herd immunity is deployed in different ways. Those justifying vaccine refusal in certain circumstances argue that in settings where there is a high level of herd immunity, the risk posed by an unvaccinated child is not great enough to override respect for parental autonomy (62, 65, 94, 98), and that the benefits of community protection do not justify the individual risk posed by the vaccine and borne by the child who is already protected through herd immunity (72, 96, 97, 17, 93, 108). Perspectives of non-vaccinators echo these ideas by asserting that some diseases are not harmful enough to proscribe vaccine refusal [ 14 ] and that vaccine injury contributes to and justifies refusal [ 16 ].

In contrast, those who argue that refusal is not justifiable propose a duty to contribute to herd immunity because it is a public good (7,80, 19,120, 33, 48, 68,115), or that free-riding (allowing one’s child to enjoy the benefits of herd immunity provided by others, while avoiding the risk of vaccinating) is unfair (37,46, 48). On this account, the vaccine refusal of a few may undermine herd immunity and thus cause harm to the many by increasing disease risks (9, 11, 26, 37, 59, 76, 81, 86); further, these risks are borne by the most vulnerable (43). These arguments about harm to others include those made by authors writing from religious perspectives (8, 81, 84, 92, 98). Finally, an account by a vaccinating parent suggests that harms resulting from non-vaccination are blameworthy because they are an intentional act of aggression against vaccinated children [ 19 ].

The concept of the child’s interests arises frequently in these publications. Pursuing or protecting these interests generally combines concern about the consequences of non-vaccination for the child with concern for autonomy, in the broad sense of being able to direct one’s life in accordance with one’s values or aims. Authors write about the interests of the child in both a general sense (i.e. the interests of the child outside of a legal context) and in a legal sense (the formal ‘best interests of the child standard’). The legal construction is used both to support (31, 6, 93) and to oppose vaccine refusal. Arguments that receiving a vaccine is in the legal ‘best interests of the child’ (21,39) posit that any deviation from a widely accepted legal view of the interests of a child should weigh the risk of harm to the child (68) irrespective of the parent’s beliefs (78), or that non-vaccination constitutes negligence or child endangerment [ 28 ]. On the other hand, some authors argue that, from a legal perspective, parents have the right to consent to or refuse vaccination ostensibly using the ‘child’s best interests standard’(93) and that there is insufficient legal precedent to argue that non-vaccination constitutes medical neglect [ 6 ].

Arguing from distrust and uncertainty

As previously noted, the sample included a set of papers written from the perspective of non-vaccinating parents. Most of these contributions seek to justify vaccine refusal, and many justifications were grounded in distrust. They call into question vaccine safety and effectiveness [ 12 , 13 , 14 , 18 ], and the accuracy of the reporting of adverse events following immunization (96). They claim financial conflicts, constructing clinicians, clinical medicine, and/or regulatory agencies as untrustworthy or non-credible [ 12 , 14 , 16 ]. They cite empirical studies of non-vaccinators to support parental preferences for natural infection over a vaccine (97). Non-vaccinating parents were not the only authors to make arguments in this vein. Some other authors cite the lack of absolute certainty of vaccine safety as justification for parents refusing vaccines in the interests of their children (28,76), especially regarding newer vaccines for which efficacy is not well-established (34). This line of argument depicts vaccine proponents as driven by commercial interests, thus justifying parental mistrust and refusal (34). Contra this, one paper asserts that refusal on the grounds of mistrust of government or medicine is not justifiable, as it is inconsistent with the scientific evidence and the well-established regulatory processes in place, such as the rigorous clinical testing required to develop and approve vaccines, and the systems established to report adverse events and ensure safety [ 8 ].

‘Response’ arguments: claims regarding the justifiability of different responses to non-vaccination

The literature examines four main responses to non-vaccination (i) government mandate policies (such as legal ramifications for refusing vaccination and vaccination as a school entry requirement), and other coercive policies, (ii) exemptions to mandate policies, (iii) individual practitioner and medical practice responses (including patient dismissal from practice for vaccine refusal, vaccinating against parents’ will, and nudging), and (iv) withholding health resources. The literature includes authors who argue that these responses are justifiable and others who argue that they are not. Much like the refusal arguments, some response arguments are absolute in their position, while others advocate weighing competing values in a context -specific way. Like refusal arguments, most arguments for and against particular responses to non-vaccinating parents draw from respect for autonomy, the doctrine of informed consent and the value of liberty, as well as considering consequences for the child and others. Other concepts appearing in these arguments include inequity, and the duties of governments and practitioners. Figure  7 represents proportionally the ’response’ arguments made in the reviewed literature.

‘Response’ arguments made in the literature on the ethics of vaccine refusal

As in the literature on refusal, many arguments about policy or practice responses to non-vaccinating parents depend on the interrelated concepts of respect for autonomy, informed consent and liberty. Five papers engage with the issue of practitioners vaccinating against parents’ will with respect to these concepts. They argue that forced vaccination by healthcare providers violates parents’ autonomy and/or the ethical requirement for informed consent, because vaccination carries risks (80,119), and clinicians have legal obligations to obtain valid consent for procedures (94). Some authors propose alternatives to forced vaccination, including focusing on rebuilding trust (rather than violating negative liberty) (32), and accepting that views on vaccination derive from plural and culturally-specific values [ 29 ]. On the other hand, proponents of forced vaccination do not engage with these concepts, instead deploying the harm principle and the legal ‘best interests of the child standard’ to justify their position. We explore this argument in the following section “Consequences for the child and others”.

Another set of papers make arguments about vaccine mandates that also draw on autonomy or liberty justifications, often weighing these against harm or risk of harm. Arguments justifying mandates are often legal in nature and use, for example, the harm principle or case law to argue that the freedom or liberty to choose not to vaccinate is limited by the risk of ill health and/or death to the child or others in the community, including vulnerable persons (83,91). One author argues that legal actions should be brought against those who harm others by refusing vaccination, as this would both discourage refusal and, in the case of any successful claims, compensate victims (55). Some authors argue that mandates are justifiable if the exercise of liberty rights poses a threat to public health (53,82,83,91,119). While those arguing that mandates are not justifiable sometimes rely on arguments about risk of harm—i.e. that in a low-incidence (and therefore low-risk) setting mandates cannot be justified (45, 87,104)—most make their arguments from autonomy, informed consent, and personal liberty and do not weigh these against the potential for harm (12,16,61,82,89,107,114). One author argues that even if mandates improve vaccination rates, they damage trust with parents and make refusers more steadfast in their decision (121), so are not sustainable. Finally, some authors present middle-ground positions that—in their view—are more autonomy- or liberty-preserving, including persuasion (121) or weakly enforced mandates (71), or argue that policy responses should take the least coercive approach that is feasible and effective to balance the needs of the individual with public health (117).

Those supporting conscientious objection to mandates argue that such provisions contribute to the collective good of a culture of respect for autonomy (82), or reflect the “American ideal” of personal freedom (66). Contra this, those opposed to conscientious objection provisions argue that challenges to mandates based in religious freedom have failed in case law, as the right to practice religion freely does not include the liberty to expose children or communities to disease (20,92). One author provides a qualified view of conscientious objection on religious grounds, arguing that such liberties could be justified only while high vaccination rates are maintained (109).

Authors disagree about whether certain policy or practice responses do, or do not, respect autonomy or uphold important liberties. For example, authors disagree on the effect of both nudges and conscientious objection policies on parental autonomy or liberty. With respect to nudges, some argue they are autonomy-preserving because they steer parents in a certain direction while allowing choice (106), do not override or challenge the strong views of deeply opposed opponents (42, 44) and uphold informed consent (121). Some supporters of nudging weigh multiple normative considerations, arguing that nudges that appeal to social responsibilities in a medical practice setting are justified because they appropriately balance parental autonomy against the practitioner’s responsibility to promote trust and collective benefits (3,80). Those opposed to nudges for vaccination decisions argue that the invasive nature of immunization increases the need for independent and informed decision making (60,113). These authors argue against a presumptive consultation style in general practice, proposing participatory clinical encounters (114), and using persuasion (42), as alternatives to more coercive approaches.

Consequences for the child and others

Many of the arguments in this literature consider individual and collective consequences—benefits, harms, burdens, and costs to society — and propose that these may override other normative considerations. The risk and prevention of harm is particularly pertinent here. For example, a parental decision can be overruled in cases where there is a significant risk of harm to the child (78), or nudges become more justifiable when the risk of harm to others is higher (3, 75).

Arguments about mandates often include concern about consequences, since it is inherent in a vaccine mandate that there will be some costs associated with non-vaccination. Mandate proponents argue that mandates ensure high vaccination rates, thus preventing disease outbreaks (39) and associated harms (97), so are in the best interest of individual children (28, 73, 111) and serve the greater good (4,28,73,79). Some justify mandates by proposing a duty to contribute to herd immunity, including under the “clean hands principle”, that is, an obligation not to participate in collectively harmful activities [ 1 , 5 ]. Conversely, some authors argue that mandates are not necessary to achieve high levels of population immunity, so state coercion is unjustified at a collective level or at the level of the individual child because each child receives limited benefit (94). Those opposing mandates also argue that vaccine safety is not absolute (88) and that mandates are a disutility, carrying associated costs with surveillance and enforcement (95). Other authors sought to balance these kinds of consequences against other normative considerations with respect to mandates, including the level of herd immunity, the risks of non-vaccination to the child and/or society, and respect for parental autonomy (32,53,88,119). One author argues that mandates protect ‘victims’ of the anti-vaccination movement from harms so long as certain conditions are met (43): that the vaccine can prevent infection and transmission, that individuals minimize their risk of exposure, and that the right of self-defense is preserved (e.g. in the case of allergy to vaccines).

Consequences are also important to arguments about conscientious objection, but here it is generally concerns about the impact on the collective. Some argue that exemptions should not be allowed because they may increase rates of disease or undermine individual or community health (20, 87, 118); others argue that if disease risk is low, exemptions are justified because those few individuals with exemptions do not pose a risk to others or herd immunity (20, 82, 105).

Consequences to the child and others are used to justify whether responses should be applied in general practice settings. As mentioned in the previous section, some authors justify healthcare workers vaccinating against a parent’s will using both the harm principle (69) and the legal ‘best interests of the child standard’ [ 25 ]; others suggest it is against the legal best interests of an older child to be forcibly vaccinated, as this may have a more detrimental impact than being unvaccinated (25,51). The best interests of the child are also invoked extensively to argue that non-vaccinating families should not be dismissed from medical practices (98,104, 26, 75). Here authors note that an unvaccinated child is more vulnerable to vaccine preventable diseases (9, 49), practice dismissal limits opportunities to access health care (31,52, 56,79,116) and the increased risk of harm from vaccine preventable diseases is transferred to other practices (9,47,49). One paper makes an argument about the consequences of treating non-vaccinating families for general practitioners, suggesting that practices caring for unvaccinated children should disclose this to other patients to minimize medicolegal risks, and should receive legal protection to account for the increased liability and risk of caring for these patients (40).

A small body of literature employs claims about who is responsible for the consequences of non-vaccination to make arguments about responses to non-vaccination. For example, one article seeks to justify discriminating against unvaccinated children with a vaccine preventable disease by limiting their access to health resources, relying on precedents such as coronary bypass surgery being withheld from obese people and smokers, and arguing that those who contribute to their own ill-health (in this case by not vaccinating) do not deserve healthcare (80). A related argument focuses on managing refugee camps during outbreaks that pose a direct and imminent threat of harm, proposing that the state is justified in withholding humanitarian aid from non-vaccinating refugees because the state is responsible for setting conditions that provide protection to (or prevent harm to) aid givers and public health [ 30 ].

Some critiques of policy or practice responses to non-vaccination emphasise that these responses can have inequitable effects and argue that this is unjustifiable. Exemption policies are a key focus here. Five papers argue against exemptions to vaccine mandates on the grounds that these unevenly distribute the risks and benefits of vaccinations (27,61,66, 73,118). These authors propose that the inaction of a few compromises the health of the most vulnerable community members (118) and disenfranchises those with medical contraindications for vaccines [ 27 ]. One author particularly focuses on home-schooled children, arguing that exempting them from vaccine mandates exposes both those children and society to harm, and that it is in the interests of these children and society that they be protected through vaccination (73). Some authors suggest that policy exemptions could be made justifiable by imposing conditions that offset potential inequities. On this view, exemptions could be justified so long as the refuser is prepared to make a financial or other contribution to help offset the potential financial burden of the diseases they may cause, or to otherwise contribute to social good [ 2 , 22 ].

Similarly, some opponents of coercive mandates or practice dismissal for non-vaccination critique these responses for having inequitable effects. It is argued that coercion risks creating a group of disenfranchised people (113) and that different people have different capacities to resist coercive policies (114). Similarly, dismissal leaves vulnerable children without advocacy (64), leads to patients not being treated equally (63) and marginalizes children from health care (74). One paper argues that family dismissal should be strongly discouraged, and an alternative mutually beneficial solution sought after considering the interests of the patient, physician, family, community, and society at large (74).

The duty of practitioners and the state

Some papers address the duties of practitioners and the duties of the state to respond to non-vaccination, in ways that go beyond simply weighing up consequences, implications for autonomy or freedom, or equity of impacts.

A variety of duties of practitioners are proposed. The first of these is to protect a child from their parent’s beliefs if those beliefs are likely to cause significant harm, which is used to justify initiating child protection proceedings to vaccinate against a parent’s will (67). Another is to protect patients in the waiting room from the risks posed by non-vaccinating patients, which is used to justify dismissing non-vaccinating patients from practice (9,26,38, 40,45). Counter-obligations are used to argue against practice dismissal. These include a health professional’s obligation to provide healthcare in the best interest of the child despite the parent’s decisions, and to deal with infectious disease as a part of their role (9,26,45,47, 56,101). Authors also argue that physicians’ obligations exclude enforcing parental accountability through dismissal, especially if that means the child is held accountable for the actions of their parents (47), and that continuing to provide care to a non-vaccinating family does not make the physician complicit in their decision (116).

It is sometimes asserted that the state is obliged to discourage non-vaccination on a number of grounds. This includes a fundamental duty of states to protect society [ 21 ], a responsibility of states to protect herd immunity as a common good or to reduce social and financial burdens and costs (53,119), and the state’s role to protect the common good in the face of risks to public health and the fallibility of individuals’ risk perception (54). Some of these arguments focus on exemptions from mandatory vaccination policies, proposing that states can not justify such exemptions because the government’s interest in protecting society outweighs the individual’s interest [ 21 ] or because vaccination is a social and moral good owed by a society to its children (118).

This review systematically explored and characterised the normative arguments made about parental refusal of routine childhood vaccination. Included publications addressed two types of arguments (i) ‘Refusal’ arguments (whether vaccine refusal is justified) and (ii) ‘Response’ arguments (whether various policy or practice responses to those who reject vaccines are justified). There were more ‘response’ arguments than ‘refusal’ arguments in the literature. On balance, most of the literature on ‘refusal’ arguments contended that it is not justifiable for parents to refuse vaccination for their children. Most of the ‘response’ argument literature argued against the various responses to non-vaccination put forward. However, compared to ‘refusal’ arguments, ‘response’ arguments were more varied and nuanced, and often came with caveats (e.g. exemptions to mandates are permissible if the disease burden is low).

The included articles predominantly originated from medical journals: these accounted for most of the papers focused on ‘response’ arguments. This may arise from the broader distribution of academic literature – there are more papers published in medicine than in the other disciplines represented in this review. It may also reflect the needs of readers of medical literature for guidance on how they should respond to non-vaccinating parents, highlighting the importance of making literature addressing the ethical dimensions of vaccine refusal accessible to immunization practitioners. Although there were some interdisciplinary perspectives, the dominance of the medical literature relating to ‘response’ arguments suggests that knowledge in this field may be advanced by incorporating more voices with expertise in ethics, law, and policy. This is especially important for deciding how to implement policy and practice responses to non-vaccination.

‘Refusal’ arguments were more common in the comparatively smaller collection of ethics/philosophy literature identified by this search, which may be, in part, a product of the differences in disciplinary traditions. While ethics/philosophy texts explore counterarguments and reach conclusions that are nuanced, and often with caveats, medical disciplines are primarily guided by practical considerations and a tradition of arguing from evidence rather than from ethical or philosophical principles. This privileging of evidence over principles may make it difficult to explore differing vaccination positions within the medical arena, potentially contributing to the adversarial clinical immunisation encounters described by vaccine-refusing parents and clinicians alike [ 7 , 18 , 19 ]. This pattern needs attention if ethical arguments are to have an impact in practice. As shown, most ethical arguments pay attention to evidence, as most ethical arguments include consequences in some way (see below). Ethical arguments can add nuance to biomedical thinking about consequences (e.g. consequences for individuals vs. the collective) and also about competing values (e.g. balancing consequences against concerns regarding autonomy, consent and liberty). The challenge for ethicists is to provide these arguments in an accessible and compelling form.

In fact, (i) consequences for the child and others, and (ii) respect for autonomy, the doctrine of informed consent and the value of liberty were dominant themes in both ‘refusal’ and ‘response’ arguments. Arguments were guided by common concepts such as the value of herd immunity, the prospect of harm to the child or others in the community and legal perspectives and precedents. The normative significance of parental trust, distrust, and uncertainty was a consideration unique to the ‘refusal’ arguments literature, driven in part by the five parental accounts from the special issue of Narrative Inquiry in Bioethics included in our sample. The concepts of inequity, and the duties of governments and practitioners only appeared in ‘response’ arguments. This is unsurprising: it reflects the purpose and perspective of these writers. An analysis of policy options is often required to bring inequity into view, and both clinicians and policymakers have obligations by virtue of their roles that can inform thinking about the right thing to do.

Many of the arguments justifying vaccine refusal aligned with the wider literature on the perspectives of non-vaccinating parents who value the freedom to make health decisions as caregivers, in what they perceive to be the best interest of their children [ 20 , 21 ]. These decisions are often based on doubts about vaccine safety or efficacy and are commonly initiated by a negative experience [ 19 , 20 , 22 ]. Unsurprisingly, arguments against rejecting childhood vaccines reflected the broader literature on how vaccine-supporting people view non-vaccination— including views that non-vaccinators are misinformed and disrupt social order, and that their actions are not based on reason or shared social values [ 23 ]. Common negative descriptors such as “anti-vaxxer” have similar valence in social discourse [ 24 ]. Those writing about vaccination should be aware of the potential for stigmatization and “othering” that can result by framing non-vaccination as a failure of parents [ 25 ]. When such arguments are used to inform policy and practice responses to non-vaccination, it introduces the potential for negative psychosocial impacts and further alienation of non-vaccinating parents.

Most ‘response’ arguments dealt with the justifiability of mandates and coercive policy. Generally, authors in favour of mandates prioritised the good of society; those against mandates prioritised individual choice. The large number of papers we found on mandates is unsurprising, given that these policies have been contentious. In Australia, federal and most state governments have mandates that require children to be vaccinated to be enrolled in childcare and for their families to be eligible for government financial assistance [ 26 ] Key political, academic and industry stakeholders argue that these mandates are designed to increase vaccination rates for the benefit of society [ 27 ]. On the other hand, Australian non-vaccinating parents express a belief that their children do not pose a threat to society, that all children should be treated in the same way, and that all parents should be able to make decisions for their children, regardless of vaccination status [ 28 ]. These perceptions of policy makers and non-vaccinating parents broadly represent the opposing arguments about mandates presented in this review. Facilitating a middle-ground approach to policy implementation may require closer attention to the values underlying these opposing views, and using a procedurally just approach to weigh them against one another.

In the context of an increasing number of systematic reviews in the field of bioethics, there has been recent criticism emerging about the use of these methods in bioethics. For example, Birchley and Ives (2022) argue that such methods are designed and therefore better suited to aggregation of quantitative data and not the complex and subjective nature of bioethical concepts and the theory-generating and interpretive approaches they require [ 29 ]. We argue that our application of the framework systematic review method - one of many well-established methods for systematic review and synthesis of qualitative and conceptual data - is appropriate for this research question and the application of our findings. Vaccine policy and practice requires a synthesis of what is known on relevant issues, and a systematic approach such as that used here provides a useful summary of the breadth of relevant ethical issues in a format that is accessible to policymakers. Our review has some limitations. Our aim was to map the range of normative arguments about vaccination refusal and policy. We did not have scope to present a novel ethical argument in response to our findings; this is an aim for future empirical and theoretical research. Most of the included literature focuses on high-income settings, predominantly the United States and the United Kingdom. In low-income settings, health services are often harder to access and levels of and reasons for vaccine rejection also differ in these settings. For example, political and cultural factors have been implicated in polio vaccine rejection in Nigeria [ 30 ], while low literacy, unemployment, and owning a mobile phone have been associated with polio vaccine refusal in Pakistan [ 31 ]. Our sampling period included a special issue of Narrative Enquiry in Bioethics which published narratives written by parents to communicate their normative positions on vaccination. These were mostly written by non-vaccinating parents and made up over one third of all arguments in the literature that support refusal. This is a strength in that it expanded the range of views represented in the review. However, it is also a limitation in that if this special issue had not been published within our sampling period, the range of arguments would have been more strongly skewed against vaccine refusal. These papers artificially increased the proportion of arguments in the scholarly domain that argue for vaccine refusal. It is a strength of our methodology that we were able to identify the unique perspective from which they were written and position them separately in our literature synthesis so that our representation of the literature distribution is not artificially skewed.

This review highlights an opportunity for interdisciplinary collaboration to widen the scope and reach of normative arguments about non-vaccination. Such collaboration can facilitate a broader understanding of and engagement with the ethical issues that may be relevant for practitioners, policymakers, and researchers in deciding how to respond to non-vaccinating parents. Arguments about the justifiability of non-vaccination and what should be done about it have the potential to positively influence routine childhood vaccination rates but can also alienate non-vaccinating families if not deployed with their perspectives in mind. There is an avenue for future work to further understand the influence of cultural context on normative arguments, especially within low- and middle-income settings. Moreover, there is an opportunity to further explore the influence and translation of scholarly ethical arguments into policy and practice responses to childhood non-vaccination.

Data Availability

The datasets generated and/or analysed during the current review are not publicly available, however the search terms used to generate the dataset are included in this published article.

Spier RE. Perception of risk of vaccine adverse events: a historical perspective. Vaccine. 2001;20:78–S84. https://doi.org/10.1016/S0264-410X(01)00306-1

Article   Google Scholar  

Nandi A, Shet A. Why vaccines matter: understanding the broader health, economic, and child development benefits of routine vaccination. Hum Vaccines Immunotherapeutics. 2020;16(8):1900–04. https://doi.org/10.1080/21645515.2019.1708669

Siddiqui M, Salmon DA, Omer SB. Epidemiology of vaccine hesitancy in the United States. Hum Vaccines Immunotherapeutics. 2013;9(12):2643–48. https://doi.org/10.4161/hv.27243

Beard FH, Hull BP, Leask J, et al. Trends and patterns in vaccination objection, Australia, 2002–2013. Med J Aust. 2016;204(7):275–75.

Ward PR, Attwell K, Meyer SB, et al. Understanding the perceived logic of care by vaccine-hesitant and vaccine-refusing parents: a qualitative study in Australia. PLoS ONE. 2017;12(10):e0185955.

Ward PR, Attwell K, Meyer SB, et al. Risk, responsibility and negative responses: a qualitative study of parental trust in childhood vaccinations. J Risk Res. 2017;1–14. https://doi.org/10.1080/13669877.2017.1391318

Wiley KE, Leask J, Attwell K, et al. Parenting and the vaccine refusal process: a new explanation of the relationship between lifestyle and vaccination trajectories. Soc Sci Med. 2020;263:113259.

Yaqub O, Castle-Clarke S, Sevdalis N, et al. Attitudes to vaccination: a critical review. Soc Sci Med. 2014;112:1–11.

Brewer NT, Chapman GB, Rothman AJ, et al. Increasing vaccination: putting psychological science into action. Psychol Sci Public Interest. 2017;18(3):149–207.

Attwell K, Navin MC, Lopalco PL, et al. Recent vaccine mandates in the United States, Europe and Australia: a comparative study. Vaccine. 2018;36(48):7377–84.

Attwell K, Seth R, Beard F et al. Financial interventions to increase vaccine coverage. Pediatrics 2020;146(6).

Attwell K, Navin M. How policymakers employ ethical frames to design and introduce new policies: the case of childhood vaccine mandates in Australia. Policy & Politics. 2022;50(4):526–47. https://doi.org/10.1332/030557321x16476002878591

Mertz M, Strech D, Kahrass H. What methods do reviews of normative ethics literature use for search, selection, analysis, and synthesis? In-depth results from a systematic review of reviews. Syst Rev. 2017;6(1):261. https://doi.org/10.1186/s13643-017-0661-x . [published Online First: 20171219].

Australian Bureau of Statistics., 2008, Australian and New Zealand Standard Research Classification (ANZSRC), availabe at: https://www.abs.gov.au/ausstats/[email protected]/0/4ae1b46ae2048a28ca25741800044242?opendocument

Kahrass H, Borry P, Gastmans C et al. PRISMA-Ethics – Reporting Guideline for Systematic Reviews on Ethics Literature: development, explanations and examples 2021 [Available from: OSF Preprints at https://osf.io/g5kfb

McDougall RJ, Notini L. Overriding parents’ medical decisions for their children: a systematic review of normative literature. J Med Ethics. 2014;40(7):448–52. https://doi.org/10.1136/medethics-2013-101446

Brunton G, Oliver S, Thomas J. Innovations in framework synthesis as a systematic review method. Res Synthesis Methods. 2020;11(3):316–30. https://doi.org/10.1002/jrsm.1399

Berry NJ, Henry A, Danchin M, et al. When parents won’t vaccinate their children: a qualitative investigation of Australian primary care providers’ experiences. BMC Pediatr. 2017;17(1):19. https://doi.org/10.1186/s12887-017-0783-2

Helps C, Leask J, Barclay L, et al. Understanding non-vaccinating parents’ views to inform and improve clinical encounters: a qualitative study in an Australian community. BMJ Open. 2019;9(5):e026299. https://doi.org/10.1136/bmjopen-2018-026299

Wiley KE, Leask J, Attwell K, et al. Parenting and the vaccine refusal process: a new explanation of the relationship between lifestyle and vaccination trajectories. Soc Sci Med. 2020;263:113259. https://doi.org/10.1016/j.socscimed.2020.113259

Attwell K, Leask J, Meyer SB, et al. Vaccine rejecting parents’ Engagement with Expert systems that inform Vaccination Programs. J Bioeth Inq. 2017;14(1):65–76. https://doi.org/10.1007/s11673-016-9756-7 . [published Online First: 20161201].

Christou-Ergos M, Leask J, Wiley KE. How the experience of medical trauma shapes Australian non-vaccinating parents’ vaccine refusal for their children: a qualitative exploration. SSM - Qualitative Research in Health. 2022;2:100143. https://doi.org/10.1016/j.ssmqr.2022.100143

Rozbroj T, Lyons A, Lucke J. The mad leading the blind: perceptions of the vaccine-refusal movement among australians who support vaccination. Vaccine. 2019;37(40):5986–93. https://doi.org/10.1016/j.vaccine.2019.08.023 . [published Online First: 20190823].

Court J, Carter SM, Attwell K, et al. Labels matter: use and non-use of ‘anti-vax’ framing in Australian media discourse 2008–2018. Soc Sci Med. 2021;291:114502. https://doi.org/10.1016/j.socscimed.2021.114502

Wiley KE, Leask J, Attwell K, et al. Stigmatized for standing up for my child: a qualitative study of non-vaccinating parents in Australia. SSM Popul Health. 2021;16:100926. https://doi.org/10.1016/j.ssmph.2021.100926 . [published Online First: 20210916].

Attwell K, Drislane S. Australia’s ‘No jab no play’ policies: history, design and rationales. Aust N Z J Public Health. 2022;46(5):640–46. https://doi.org/10.1111/1753-6405.13289

Attwell K, Navin M. How policymakers employ ethical frames to design and introduce new policies: the case of childhood vaccine mandates in Australia. Policy and Politics. 2022;1–22. https://doi.org/10.1332/030557321X16476002878591

Wiley K, Robinson P, Degeling C, et al. Get your own house in order’: qualitative dialogue groups with nonvaccinating parents on how Measles outbreaks in their community should be managed. Health Expect. 2022;25(4):1678–90. https://doi.org/10.1111/hex.13511

Birchley G, Ives J. Fallacious, misleading and unhelpful: the case for removing ‘systematic review’ from bioethics nomenclature. Bioethics. 2022;36(6):635–47. https://doi.org/10.1111/bioe.13024

Yahya M. Polio vaccines—no thank you! Barriers to polio eradication in Northern Nigeria. Afr Affairs. 2007;106(423):185–204. https://doi.org/10.1093/afraf/adm016

Khattak FA, Rehman K, Shahzad M, et al. Prevalence of parental refusal rate and its associated factors in routine immunization by using WHO Vaccine Hesitancy tool: a Cross sectional study at district Bannu, KP, Pakistan. Int J Infect Dis. 2021;104:117–24. https://doi.org/10.1016/j.ijid.2020.12.029

Download references

This review was funded by the Australian National Health and Medical Research Council, grant number GNT1126543.

Author information

Authors and affiliations.

Sydney School of Public Health, The University of Sydney, Edward Ford Building A27, Sydney, 2006, Australia

Kerrie Wiley, Maria Christou-Ergos, Penelope Robinson & Catherine Helps

Australian Centre for Health Engagement, Evidence and Values, The University of Wollongong, Wollongong, 2522, Australia

Chris Degeling & Stacy M Carter

Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, 3010, Australia

Rosalind McDougall

School of Social Sciences, Asian Studies & Politics, International Relations, University of Western Australia, Perth, 6009, Australia

Katie Attwell & Shevaun Drislane

You can also search for this author in PubMed   Google Scholar

Contributions

KW contributed to study design and search strategy development, ran the searches, managed the screening and inclusion process, screened articles for inclusion, extracted data, analysed and interpreted data and co-led manuscript drafting; MC ran updated searches, screened articles for inclusion and extracted data, assisted with analysis and interpretation and co-led manuscript drafting; CD contributed to study design and search strategy development, provided technical guidance, screened articles for inclusion and contributed to manuscript drafts; RM contributed to study design and search strategy development, provided technical guidance, screened articles for inclusion and contributed to manuscript drafts; PR screened articles for inclusion, extracted data, assisted with analysis and contributed to manuscript drafts; KA contributed to search strategy development, screened articles for inclusion and contributed to manuscript drafts; CH screened articles for inclusion and contributed to manuscript drafts; SD screened articles for inclusion and contributed to manuscript drafts; SMC contributed to study design and search strategy development, provided technical guidance, screened articles for inclusion and contributed to manuscript drafts.

Corresponding author

Correspondence to Kerrie Wiley .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Cite this article.

Wiley, K., Christou-Ergos, M., Degeling, C. et al. Childhood vaccine refusal and what to do about it: a systematic review of the ethical literature. BMC Med Ethics 24 , 96 (2023). https://doi.org/10.1186/s12910-023-00978-x

Download citation

Received : 20 February 2023

Accepted : 31 October 2023

Published : 08 November 2023

DOI : https://doi.org/10.1186/s12910-023-00978-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Vaccination
• Immunization
• Vaccine refusal
• Systematic review
• Normative literature
• Medical ethics

BMC Medical Ethics

ISSN: 1472-6939

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 16 November 2023

Methods, strategies, and incentives to increase response to mental health surveys among adolescents: a systematic review

• Julia Bidonde   ORCID: orcid.org/0000-0001-7535-678X 1 ,
• Jose F. Meneses-Echavez   ORCID: orcid.org/0000-0003-4312-6909 1 , 2 ,
• Elisabet Hafstad   ORCID: orcid.org/0009-0001-6296-410X 1 ,
• Geir Scott Brunborg   ORCID: orcid.org/0000-0002-1382-2922 3 , 4 &
• Lasse Bang   ORCID: orcid.org/0000-0002-3548-5234 3  

BMC Medical Research Methodology volume  23 , Article number:  270 ( 2023 ) Cite this article

152 Accesses

10 Altmetric

Metrics details

This systematic review aimed to identify effective methods to increase adolescents’ response to surveys about mental health and substance use, to improve the quality of survey information.

We followed a protocol and searched for studies that compared different survey delivery modes to adolescents. Eligible studies reported response rates, mental health score variation per survey mode and participant variations in mental health scores. We searched CENTRAL, PsycINFO, MEDLINE and Scopus in May 2022, and conducted citation searches in June 2022. Two reviewers independently undertook study selection, data extraction, and risk of bias assessments. Following the assessment of heterogeneity, some studies were pooled using meta-analysis.

Fifteen studies were identified, reporting six comparisons related to survey methods and strategies. Results indicate that response rates do not differ between survey modes (e.g., web versus paper-and-pencil) delivered in classroom settings. However, web surveys may yield higher response rates outside classroom settings. The largest effects on response rates were achieved using unconditional monetary incentives and obtaining passive parental consent. Survey mode influenced mental health scores in certain comparisons.

Conclusions

Despite the mixed quality of the studies, the low volume for some comparisons and the limit to studies in high income countries, several effective methods and strategies to improve adolescents’ response rates to mental health surveys were identified.

Peer Review reports

Globally, one in seven adolescents (aged 10–19 years) experiences a mental disorder, accounting for 13% of the health burden in this age group [ 1 ]. The Global Burden of Diseases Study reports that anxiety disorders, depressive disorders and self-harm are among the top ten leading causes of adolescent health loss [ 2 ]. Understanding the magnitude and determinants of mental health problems among adolescents may inform initiatives to improve their health.

Survey research methods are often used to investigate the prevalence and incidence of mental health problems and associated risk factors and outcomes [ 3 , 4 , 5 ]. Prevalence estimates are based on responses from a sample of the target population. A major priority is to ensure that invited adolescents participate in the survey. In survey research, the response rate (also known as completion rate or return rate) is a crucial metric that indicates the proportion of individuals who participated in the survey divided by the total number of people in the selected sample. Non-response reduces the sample size and statistical precision of the estimates and may also induce non-response bias [ 6 , 7 ]. Consequently, survey response rate is often considered an indicator of the quality and representativeness of the obtained data [ 6 , 8 ].

Non-response is a particular concern in surveys of adolescents as this age-group is hard to reach and motivate to participate in research. Furthermore, response rates for health-related surveys are declining [ 3 , 5 ]. For example, the response rate for a repeated household survey conducted in the US dropped by 35 percentage points between 1971 and 2017 [ 9 ]. Similarly, response rates for the National Health and Nutrition Examination Survey (NHANES) dropped by 15 percentage points from 2011/2012 to 2017/2018 [ 10 ]. There is an increasing need for surveys to be designed and administered in ways that maximise response rates. Multiple published reviews [ 11 , 12 , 13 ] provide evidence of methods and strategies to increase response rates (primarily among adults). These point to several factors associated with increased response rate, including the use of monetary incentives, short questionnaires and notifying participants before sending questionnaires. However, none of these focuses specifically on adolescent samples. Survey characteristics may impact response rates differently in adult and adolescent samples due to age-specific attitudes. For example, adolescents may find web surveys more acceptable and appealing than telephone or postal surveys. Attitudes towards incentives or the topic of surveys (e.g., mental health) may also differ between adults and adolescents. Furthermore, surveys of adolescents are often conducted in class-room settings which exerts a strong contextual influence on response rates. Such contextual factors may moderate the effect of methods and strategies that have been shown to influence response rates among adults.

Features that boost response rates may also influence the mental health outcomes obtained. For example, web-based surveys may improve response rates due to the relative ease of participation when compared with in-person surveys. But they may also impact mental health scores, leading to higher or lower estimates of the prevalence of mental health problems. For example, this can occur because of reluctance to disclose mental health problems to an interviewer, or because web-surveys elicit careless responses. Some studies suggest that mental health indicators differ according to the mode of data collection [ 14 , 15 , 16 ]. Consequently, we need to know which strategies and methods improve adolescents' response rates to mental health surveys and how these might impact mental health scores.

Many factors may positively affect response rates in surveys, including how potential participants are approached and informed about the survey (e.g., pre-notifications), incentives (e.g., financial compensation), data collection mode (e.g., web-based vs. paper-and-pencil), survey measure composition and design (e.g., questionnaire length), using follow-up reminders, and practical issues such as time and location [ 11 , 16 ].

This review aims to identify effective methods and strategies to increase adolescents’ response rates (which may improve the quality of information gathered) to surveys that include questions about mental health, alcohol, and substance use. It also explores how different modes of survey delivery may impact on mental health scores. To accommodate recent trends in technological improvements and attitudes we focus on studies that have been published after 2007. By choosing 2007 we covered advances in technology since the development of the smart phone, and the literature after a previous review [ 13 ] whose search was completed in 2008. Furthermore, to provide the best quality evidence we focus on studies with randomised controlled designs.

This systematic review used the Cochrane approach to methodology reviews [ 17 ]. The full protocol was peer reviewed and is publicly available [ 18 ], but was not registered. The review is reported according to the PRISMA guidelines [ 19 ]. Amendments to the protocol can be found in Additional file 7 : Appendix G.

Eligibility criteria

This review evaluates the effectiveness of survey methods, strategies, and incentives (hereafter “survey mode”) to improve adolescents’ response rates for surveys containing mental health, alcohol, and substance use questions. Adolescents were defined as those aged 12–19 years. It focuses on research conducted in a community setting published since 2007 (when smart phones were introduced). The outcome measures are:

Survey response rates: the percentage of individuals who returned a completed survey, by survey mode;

Mental health variation (i.e., self-reported prevalence) by survey mode. For example, depression scores or alcohol use rates reported for survey modes;

Participant variations (e.g., gender differences) in self-reported mental health scores by survey mode.

Additional file 1 : Appendix A present the review’s eligibility criteria and a glossary of definitions.

Search strategy

One information specialist (EH) developed the search strategy, and a second peer reviewed it using the six domains of the PRESS guidelines [ 20 ]. Following a pilot search in the Cochrane Central Database of Controlled Clinical Trials (Wiley), an adapted search strategy was run in APA PsycINFO (Ovid), MEDLINE (Ovid) and Scopus (Elsevier) on May 13, 2022. Backwards and forwards citation searching were undertaken with last searches undertaken on June 28, 2022. Full searches are presented in Additional file 2 : Appendix B.

Study selection

We deduplicated records in EndNote and screened records in EPPI Reviewer 4 [ 21 ]. Two reviewers (JB, JFME) independently piloted the screening, using machine learning functions in EPPI-Reviewer combined with human assessment (see Additional file 2 : Appendix B). Randomised controlled trials (RCTs) and non-randomised studies of interventions were screened first, and once we identified more than five (pre-specified) RCTs, screening for other study designs was stopped. The two reviewers screened titles and abstracts, and then each relevant full text, independently against the eligibility criteria. A third reviewer adjudicated disagreements. Figure  1 shows the search and screening, and Additional file 2 : Appendix B lists the excluded studies.

PRISMA diagram for the study identification and selection

For studies reported in several documents, all related documents were identified and grouped together to ensure participants were only counted once.

Data extraction

The two reviewers conducted double independent data extraction into Excel forms. A third reviewer adjudicated disagreements. We piloted data extraction on five studies (see Additional file 3 : Appendix C).

Risk of bias (quality assessment)

The two reviewers assessed studies’ risk of bias (RoB) independently using Cochrane’s RoB 2.0 [ 22 ]. Any financial and non-financial conflicts of interest reported in the studies were collected as a separate bias category outside of RoB 2.0 (see Additional file 3 : Appendix C).

Data synthesis

The protocol provides full details of the planned data synthesis [ 18 ]. We present a summary here.

We grouped studies by the type of survey modes. When two or more studies reported the same outcome and survey modes were deemed sufficiently homogeneous, we checked that the data direction permitted pooling. Where necessary to make the values meaningful, we arithmetically reversed scales. We included studies in the meta-analyses regardless of their RoB rating.

To assess statistical heterogeneity, we first checked our data for mistakes and then used the Chi 2 test (threshold P  < 0.10) and the I 2 statistic following Cochrane Handbook recommendations [ 23 ]. In cases of considerable statistical heterogeneity (I 2  > 70%) we did not conduct meta-analysis. Where there was less heterogeneity (I 2  <  = 70%), we performed random effects meta-analysis using Review Manager 5.4.1. We also assessed studies’ clinical and methodological heterogeneity (participants, survey processes, outcomes, and other study characteristics) to determine whether meta‐analysis was appropriate.

Where statistical pooling was not feasible, we followed the Synthesis Without Meta-analysis guideline to report the results narratively [ 24 ]. For dichotomous outcomes (e.g., response rates and adolescents’ self-reported alcohol use) we calculated odds ratios (ORs) and their 95% confidence intervals (CIs) to estimate between-mode differences. We used the default weighting technique (e.g., Mantel–Haenszel) for dichotomous outcomes in RevMan software. For continuous outcomes, we estimated the difference between survey modes using Mean Differences (MDs) or Standardized Mean Differences (SMDs) if the same outcome was measured with different questionnaires. The standard deviation was not modified [ 25 ]. We planned subgroup analyses and a GRADE assessment [ 18 ]. Amendments to the protocol are in Additional file 7 : Appendix G.

Search and screening results

Database searches retrieved 12,054 records. We removed 1,892 duplicates. EPPI-reviewer 4 marked 6,841 records as ineligible (see Additional file 2 : Appendix B). The team screened the titles and abstracts of 3,321 records and the full text of 48 documents, identifying ten eligible documents. Citation searches on ten eligible documents retrieved a further 740 records, which yielded six eligible documents. We identified one further document from reference lists. In total, this review included 15 studies (17 documents). Additional file 2 : Appendix B shows the excluded studies. We did not identify any studies in languages we could not translate.

Figure  1 shows the PRISMA diagram.

Details of included studies

Table 1 provides details of the included studies and Additional file 3 : Appendix C shows the data extraction tables. The age distribution of participants in the studies varied, but most were aged 14 to 16 years. A smaller proportion of participants were aged < 14 years or > 16 years. The sex distribution in studies were generally even and ranged from 32% [ 26 ] to 58% [ 27 ]. Studies were conducted in both rural and urban areas and included a range of national and racial/ethnic representation. Although most studies took place within school settings, four of them [ 26 , 28 , 29 , 30 ] were conducted in non-school environments. All the studies involved community (i.e., non-clinical) samples, but we note that the Pejtersen’s study [ 26 ] focused on a group of vulnerable children and youth.

The fifteen studies investigated six comparisons:

Paper-and-pencil (PAPI) survey administration versus web administration ( n  = 9 in 10 documents)

Telephone interviews versus postal questionnaires ( n  = 2)

Active versus passive parental consent ( n  = 1)

Web first versus in-person first interviews ( n  = 1)

Vouchers versus no vouchers ( n  = 1 in 2 documents)

Internal supervision versus external supervision ( n  = 1)

Risk of bias

Overall, study authors provided little information on their research methods resulting in several unclear domains that raised concerns about risk of bias. The main issues identified related to the randomisation process, measurement of the outcomes, and selective reporting of results. We classified three cluster RCTs [ 31 , 32 , 38 , 40 ] and three parallel RCTs [ 26 , 35 , 37 , 39 ] as high RoB. There were some concerns with nine [ 14 , 16 , 27 , 28 , 29 , 30 , 33 , 34 , 36 ] parallel RCTs (see Additional file 4 : Appendix D). RoB for each study is presented below.

This section presents the study results and the meta-analyses. Additional file 6 : Appendix F contains additional forest plots. We describe the results narratively without prioritization or hierarchy. We did not contact study authors for missing/additional data. Caution is advised when interpreting the meta-analyses because of studies’ quality/RoB and imprecision.

The considerable statistical heterogeneity (I 2  > 70%) in the data for the two largest comparisons (1 and 2) precluded a meta-analysis of response rates. The studies showed divergent effect estimates, which may be explained by their different outcome measures. There were differences inherent to the study designs with cluster RCTs adjusted for clustering. There were important differences in the survey implementation procedures, including different interfaces, skipped questions, confidentiality measures and different degrees of supervision. Ignoring these considerations would have resulted in pooled analyses prone to misleading inferences.

Comparison 1: paper-and-pencil versus web-based administration mode

Nine studies (ten documents) compared PAPI surveys to web-based surveys [ 14 , 16 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ]. The studies included one cluster RCT with high RoB, three RCTs with high RoB and five RCTs with RoB concerns.

Response rate

Five studies reported response rate [ 16 , 30 , 31 , 32 , 34 , 37 ]. Three studies reported between-group differences [ 30 , 31 , 32 , 34 ], but because of considerable heterogeneity (I 2  > 90%) we present the effect estimates for each study separately (Fig.  2 ). Van de Looij-Jansen [ 16 ] reported a narrative summary rather than outcome data. Trapl [ 37 ] reported a 100% response rate.

Odds ratios for various survey delivery mode comparisons: Adolescents’ response rates (results not pooled)

Denniston [ 31 ], reported a cluster RCT in two documents [ 31 , 32 ] and accounted for clustering in the analyses. Therefore, we did not conduct design effect adjustment [ 41 ]. The odds of response increased by nearly 80% for PAPI compared with a web mode (OR 0.22, 95% CI 0.19 to 0.26; n  = 7747). Participants could skip questions in some of the modes (“with skip patterns”). Treated as an independent intervention arm, the group “on your own” web without skip patterns had the lowest response rate (28%; 559/1997) compared with the other web formats (in-class web without skips and with skips) and markedly lower odds of response relative to PAPI (OR 0.04, 95% CI 0.03 to 0.04). Low odds of response affect the pooled rates among the web survey modes. The pooled response rate for the two web in-class modes (with and without skips) was 90.7%, which was no different to the PAPI response rate (OR 0.94, 95% CI 0.78 to 1.14; n  = 5750).

Mauz [ 30 ] explored three survey modes that we combined into an “overall web mode”. Each mode included varying proportions of participants receiving PAPI surveys or web surveys (see Table 1 ), but separate data for web participants were not reported. The odds of response decreased by nearly 70% when using PAPI compared with a web mode (OR 0.29, 95% CI 0.23 to 0.38; n  = 1195) [ 30 ].

Miech [ 34 ] found evidence of no effect on response rates for PAPI compared with web mode (electronic tablets) (OR 1.03, 95% CI 0.97 to 1.08; n  = 41,514).

Van de Looij-Jansen [ 16 ] reported an overall response rate of 90%, with no difference between PAPI or web modes (data not reported) and Trapl [ 37 ] reported 100% response rate.

Mental health variation by mode of survey delivery

Nine studies (ten documents) reported between-modes variations in point estimates for various mental health and substance use scores at the time of survey completion [ 14 , 16 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ].

Two studies (considerable heterogeneity: I 2  = 82%) of Dutch adolescents from secondary schools in rural and urban areas reported between-modes variations for adolescents’ mental health scores (Fig.  3 ) [ 16 , 35 ]. Raat [ 35 ] reported that for the mental health subscale of the Child Health Questionnaire (CHQ-CF), PAPI mode participants had slightly lower scores compared with web users (MD -1.90, 95% CI -3.84 to 0.04; n  = 933). Conversely, van de Looij-Jansen [ 16 ] reported no between-mode variations in self-reported total scores for the Strength and Difficulties Questionnaire (SDQ). Boys tended to report better mental health scores when completing surveys using PAPI than the web (MD 1.0, 95% CI -0.10 to 2.10; n  = 279).

Mean differences for paper-and-pencil versus web administration survey delivery modes: Adolescents’ self-reported mental health

Two studies estimated between-mode variations for adolescents’ self-reported psychological wellbeing scores [ 16 , 30 ]. Mauz [ 30 ] reported the number of adolescents experiencing favourable psychological wellbeing, expressed as t values, using the KIDSCREEN (the Health-Related Quality of Life Questionnaire for Children and Adolescents aged from 8 to 18 years, Questionnaires—kidscreen.org). The narrative findings indicated that psychological wellbeing was the same for both PAPI and web-based questionnaire modes (PAPI 50.5% vs web 49.3% ( n  = 1194), P  = 0.07 adjusted with Bonferroni correction). Similarly, van de Looij-Jansen [ 16 ] reported no between-mode variations in mean scores of adolescents’ self-reported psychological wellbeing obtained from nine items about feelings and moods from the CHQ-CF (MD pooled for boys and girls -0.97, 95% CI -3.21 to 1.28; n  = 531) (Fig.  4 ).

Mean differences for paper-and-pencil versus web administration survey delivery modes: Adolescents’ psychological wellbeing (nine items about feelings and moods derived from the CHQ-CF)

Denniston [ 31 ] found evidence of no between-mode estimate variations for adolescents’ self-reported sadness (OR 1.02, 95% CI 0.90 to 1.15; n  = 5786) or suicide attempts (OR 1.01, 95% CI 0.83 to 1.24; n  = 5786) measured using the Youth Risk Behavior Surveys [ 31 , 32 ].

Hamann [ 33 ] found evidence of no between-mode estimate variations for adolescents’ self-reported anxiety (MD 1.65, 95% CI -5.18 to 8.48; n  = 56) or depression (MD 0.78, 95% CI -1.54 to 3.10; n  = 56) measured using the Spence Children’s Anxiety Scale (SCAS) and the German version of the Children’s Depression Inventory (CDI) [ 33 ].

Six studies (7 documents) reported adolescents’ self-reported lifetime alcohol use [ 14 , 30 , 31 , 32 , 34 , 36 , 37 ]. Lygidakis [ 14 ] reported on adolescents who said they “ have been drunk ” and therefore we did not pool this study with studies of lifetime use. In Lygidakis [ 14 ], lifetime estimates of self-reported alcohol use were 11% lower in the PAPI group compared with the web survey group (OR 0.89, 95% CI 0.79 to 1.00; n  = 190). A pooled analysis of five studies [ 30 , 31 , 32 , 34 , 36 , 37 ] suggested that the odds of alcohol lifetime use were 13% higher among adolescents completing the web survey compared with those using PAPI (OR 1.13, 95% CI 1.00 to 1.28; n  = 49,554); substantial heterogeneity was observed (I 2  = 59%) (Fig.  5 ).

Odds ratios for paper-and-pencil versus web administration of surveys: Adolescents’ self-reported lifetime alcohol use

A pooled analysis of two studies, Denniston [ 31 ] and Trapl [ 37 ], showed evidence of no between-mode estimate variations for adolescents’ self-reported marijuana use (OR 1.05, 95% CI 0.93 to 1.18; n  = 6,061) (Fig.  6 ).

Pooled estimate variations for paper-and-pencil versus web administration of surveys: Adolescents’ self-reported lifetime marijuana use

Participant variation by mode of survey delivery

Gender was the only participant characteristic for which the included studies reported disaggregated data. We calculated estimate variations by gender within studies rather than between survey mode comparisons.

In Van de Looij-Jansen [ 16 ], boys tended to report better mental health scores than girls for total mental health score, emotional symptoms, and psychological well-being. The largest and more precise difference was for emotional symptoms (pooled MD for both survey modes -1.31, 95% CI -1.64 to -0.98; n  = 531), whereas the mental health total scores reported with the PAPI version of the SDQ proved to be the least precise (MD -0.30, 95% CI -1.54 to 0.94; n  = 261). The absence of statistical heterogeneity in the results for emotional symptoms and psychological well-being suggests that boys reported better scores than girls regardless of the survey mode (Fig.  7 ).

Mean difference by gender for paper-and-pencil and web administration of surveys: Adolescents’ self-reported mental health outcomes

In Raghupathy [ 36 ], the odds of reporting lifetime alcohol use increased by more than one half in girls (OR 1.61, 95% CI 0.99 to 2.62; n  = 339). Less precise estimate variations were observed when using PAPI vs web mode (Fig.  8 ).

Odds ratios for gender variations for paper-and-pencil and web administration of surveys: Adolescents’ self-reported lifetime alcohol use

Comparison 2: telephone interview vs postal questionnaires

Two studies reported outcome data for this comparison ( n  = 2322) [ 28 , 29 ]. Trained interviewers performed the telephone interviews in both studies. Interviewers in Erhart [ 29 ] used computer-assisted telephone interviews whereas in Wettergren [ 28 ] interviewers were trained to read the questions aloud and record participants’ answers. There were concerns for RoB for both studies.

We did not pool the response rates due to considerable heterogeneity (I 2  > 90%); the studies are presented separately [ 28 , 29 ]. The studies reported opposing results (Fig.  2 ). Erhart [ 29 ] reported a 41% completion rate for telephone interviews compared with 46% for postal questionnaires (OR 0.82, 95% CI 0.68 to 1.00; n  = 1,737), whereas Wettergren [ 28 ] reported a response rate of 77% for telephone interviews and 64% for postal questionnaires (OR 1.89, 95% CI 1.32 to 2.72; n  = 585).

The studies evaluated the effect of differences in survey mode on estimate variations of adolescents’ self-reported mental health measured by the SDQ total score [ 29 ] and the mental health component of the RAND 36-Item Short Form Health Survey (SF-36) measure [ 28 ]. We converted the data in Wettergren [ 28 ] to a zero to 10 scale to obtain a more homogenous pooled analysis. In the meta-analysis, adolescents reported 1.06 points better mental health when a telephone interview was used (MD 1.06, 95% CI 0.81 to 1.30; n  = 1,609) (Fig.  9 ).

Pooled mean difference for survey delivery by telephone interview versus postal questionnaires: Adolescents’ self-reported mental health

Wettergren [ 28 ] found evidence of no estimate variation for adolescents’ self-reported anxiety (MD -0.60, 95% CI -1.21 to 0.01; n  = 580) and a small estimate variation for self-reported depression on the Hospital Anxiety and Depression Scale (HADS) favouring telephone interviews relative to postal questionnaires (MD -0.50, 95% CI -0.94 to -0.06; n  = 585).

Wettergren [ 28 ] reported participants’ gender differences in self-reported estimate variations of mental health (SF-36) alongside anxiety and depression (both measured with the HADS). Boys tended to report better mental health (SF-36) and anxiety (HADS) scores than girls, with the largest gender difference in anxiety (MD -1.85, 95% CI -2.42 to -1.28, n  = 585) [ 28 ]. Postal questionnaires seem to result in a larger gender difference in self-reported mental health scores compared with telephone questionnaires (I 2  = 53%). No differences between survey modes were observed for anxiety scores (I 2  = 0%). Boys and girls reported similar depression scores (MD -0.07, 95% CI -0.49 to 0.35; I 2  = 0%) for both survey modes (Fig.  10 ).

Pooled mean differences by gender for survey delivery by post and telephone: Adolescents’ self-reported mental health

Comparison 3: active vs passive parental consent.

One cluster RCT compared schools randomised into groups where adolescents required active parental consent to undertake the survey or where passive parental consent was accepted [ 38 ]. The study had high RoB.

District schools assigned to passive parental consent achieved a response rate of 79% compared to 29% achieved by schools assigned to active consent mode ( p  = 0.001, number of participants per mode not reported) [ 38 ].

Courser [ 38 ] did not report any mental health variation or participant variations by survey mode.

Comparison 4: web first vs in-person first survey versions

One RCT [ 27 ] investigated the order of survey delivery. One group of students was offered an in-person survey, with web follow-up in case of non-response. A second group was asked to complete a web survey first, with in-person survey in case of non-response. There are some concerns over the study’s RoB.

McMorris [ 27 ] found evidence of no difference in response rates between adolescents completing a web survey first or an in-person survey first (OR 0.57, 95% CI 0.24 to 1.31; n  = 386) (Fig.  2 ).

McMorris [ 27 ] found evidence of no difference on adolescents’ self-reported lifetime alcohol use (OR 0.84, 95% CI 0.55 to 1.27; n  = 359) or lifetime marijuana use (OR 0.65, 95% CI 0.41 to 1.01; n  = 359) between the two survey modes. McMorris [ 27 ] did not report on participant variations by survey mode.

Comparison 5: voucher vs no voucher

One RCT [ 26 ] (reported in two documents) investigated whether an unconditional monetary incentive (a supermarket voucher) increases the response rate among vulnerable children and youths receiving a postal questionnaire [ 26 , 39 ]. The study was classified as high RoB.

Pejtersen [ 26 ] found that the monetary incentive yielded a response rate of 76% versus 43% without the incentive (OR 4.11, 95% CI 2.43 to 6.97; n  = 262) (Fig.  2 ).

The study also found that offering a voucher made no difference to adolescents’ self-reported emotional symptoms compared with no voucher (MD -0.70, 95% CI -1.58 to 0.18; n  = 156) measured using the emotional symptoms subscale of the SDQ [ 26 , 39 ]. Pejtersen [ 26 ] did not report on participant variations by survey mode.

Comparison 6: internal versus external supervision

One Swiss cluster-RCT evaluated the effect of external supervision (by a senior student or researcher) compared to internal supervision (by the teacher) when students completed online interviews [ 40 ]. The study was classified as high RoB.

Walser [ 40 ] only reported outcomes relevant to mental health variations, finding evidence of no variations in adolescents’ self-reported lifetime alcohol use according to the survey mode (OR 1.08, 95% CI 0.79 to 1.47; n  = 1,197).

Subgroup and sensitivity analyses

There were too few studies, and no quasi-RCTs, to complete the planned subgroup and sensitivity analyses.

Reporting bias assessment

We could not assess reporting biases, because too few studies were available (i.e., less than 10 studies) for each comparison [ 23 ].

Certainty assessment

We opted not to perform a GRADE assessment due to the limited quantity of studies for each comparison under consideration and the mixed quality of studies.

This review identified fifteen RCTs that investigated six different comparisons among adolescents. Although the included studies were of mixed quality, several effective methods and strategies to improve adolescents’ response rates to mental health surveys were identified. Findings show that response rates varied with survey mode, consent type, and incentives.

Comparisons of web versus PAPI mode yielded discrepant findings that must be interpreted in relation to survey delivery context. One study showed that postal invitations to a web survey was associated with higher response rates compared to PAPI mode [ 30 ], possibly due to the additional effort required to return the completed PAPI survey by post. In contrast, there were no significant differences in response rates for web and PAPI modes conducted in classrooms during school hours [ 16 , 31 , 32 , 34 ]. However, one study showed that inviting adolescents to complete a web survey on their own (at home within 2–3 weeks following the invitation) dramatically decreased response rates compared with completing PAPI or web surveys at school (28% vs. ~ 90%) [ 31 , 32 ]. These findings show that response rates may vary according to both delivery mode and context. A previous meta-analysis showed that web surveys yield lower response rates (on average 12 percentage points) than other modes [ 12 ]. However, this review did not focus specifically on adolescents. More studies are needed to determine whether response rates among adolescents differ between web and PAPI surveys delivered outside school.

Conflicting evidence was found for telephone interview surveys compared to postal PAPI surveys. One study found significantly higher response rates (77% vs 64%) for telephone interview surveys [ 28 ], while another found significantly but marginally (48% vs. 43%) higher response rates for postal PAPI surveys [ 29 ]. The reasons for these opposing findings are unclear, but other contextual factors may play a role such as the age of the studies (conducted before 2010) reflecting potential time related differences in attitudes towards telephone interviews and postal PAPI surveys. One study [ 27 ] found that response rates did not differ significantly when comparing a web survey and follow-up in-person interview for non-responders with in-person interview and follow-up web survey for non-responders. Administering a web survey first is a cost-saving approach which is unlikely to adversely impact adolescents’ response rates.

One study showed that unconditional monetary incentives (i.e., voucher) increased response rates by 33 percentage points [ 26 ], supporting a prior review on postal surveys [ 42 ]. Interestingly, evidence favours monetary incentives unconditional on response compared with similar incentives conditional on response to improve response rates [ 11 , 42 ]. In contrast, a recent meta-analysis [ 12 ] concluded that incentives had no effect on response rates in web surveys. These discrepant findings may indicate that incentives matter less for response rates in web surveys compared to other modes. Our review also identified one study showing that passive parental consent achieved more than double the response rate of active consent (79% vs. 29%) [ 38 ]. A prior meta-analysis of studies found similar evidence in favour of passive parental consent [ 43 ]. If ethical and data protection considerations permit, using passive parental consent may boost response rates substantially.

Survey mode influenced mental health scores in certain comparisons. We found no evidence of effect on self-reported mental health scores (across a range of measures) between PAPI and web surveys [ 16 , 30 , 31 , 32 , 34 , 35 , 36 , 37 ]. However, our pooled analysis of lifetime alcohol use showed 13% higher use when a web mode was used compared to a PAPI mode. This could possibly be attributed to differential response rates, for example if heavy drinkers are less likely to respond to a PAPI compared to web survey. In contrast, two studies indicated that lifetime marijuana use did not differ between web and PAPI survey modes [ 31 , 32 , 37 ]. The reasons for such differences are unclear and should be further researched. Telephone interview compared with postal PAPI surveys was associated with slightly better mental health scores [ 28 , 29 ]. These differences were quite small and probably of limited practical significance [ 28 ]. Nonetheless, survey designers should be aware that adolescents may report fewer mental health problems in telephone interviews. Such findings may be due to differential response rates as already mentioned, for example if those with mental health problems are less likely to respond to telephone surveys compared to PAPI surveys. Another reason may be that adolescents are less willing to report such problems directly to another person. The added anonymity of non-telephone surveys may encourage adolescents to provide more genuine responses to sensitive questions concerning their mental health. A study that compared supervision by either teachers or researchers during an in-class web survey [ 40 ] found no significant differences in mental health scores, which suggests that the choice of supervision personnel does not impact responses.

There was little evidence of differences between gender and survey characteristics on mental health scores. While several studies highlighted that males report better mental health than females [ 16 , 28 ], there was no indication that specific survey modes impacted males’ and females’ mental health differentially (i.e., no interaction effect). Many studies did not report mental health scores separately for males and females.

Our review complements earlier reviews of factors that influence response rates [ 11 , 12 , 42 , 43 , 44 ]. Together, these reviews provide useful information regarding how to design surveys to maximise response rates. The extent to which their findings are generalizable to adolescents in recent decades is unclear. Our own review show that relatively few studies have focused specifically on adolescents. Nevertheless, many of our findings are in line with those outlined in previous reviews. One outstanding question is whether web surveys yield lower response rates than other modes also for adolescents. The studies included in our review highlights the need to consider contextual factors when comparing response rates between surveys. For example, survey mode may have less impact on response rates in class-room settings. Our findings highlight the need for more studies to provide high-quality evidence of methods and strategies to ensure adequate response rates in mental health surveys of adolescents. This is particularly important given the present worldwide focus on adolescent mental health and the decreasing response rates in surveys.

Although we found relevant RCTs, they were of insufficient quality to draw firm conclusions. The studies in some comparisons showed considerable heterogeneity and meta-analysis was not feasible for most comparisons. For several comparisons, only one or two studies were available. In RCTs where one survey mode was superior to another, the results need to be confirmed with better conducted (and/or reported) studies.

The studies had a range of differences that reduce the comparability of studies and the generalisability and strength of our findings. Various questionnaires were used, differing greatly in content, length, and appearance. Questionnaires were managed in different ways, for example some used skips to ensure confidentiality, and some did not permit the questions to be read aloud during interview. Different methods were used to deliver questionnaires: postal, in the classroom, or sent to parents. The studies investigated a mix of outcomes using a range of tools and with study-specific adaptations in some cases.

The median publication year of the studies is 2010. The inclusion of older RCTs may mean that in a world of high internet and smart phone usage, the applicability of the earlier findings may be weakened.

Key strengths of this review include the team’s expertise in synthesis methods, topic area, information retrieval, and machine learning. We identified a substantial number of RCTs in adolescent populations, some with many participants, using an extensive search in databases augmented by forwards and backwards citation searching.

Although it is not usually common practice to search for outcomes in literature searches for reviews of effect of interventions [ 45 ], given the challenges of searching for this review topic, we considered it necessary to reduce the screening burden by including the concept of outcomes in our search. This approach may have lowered the search sensitivity where authors did not mention outcomes of interest in the abstract [ 46 ] and may also have introduced publication bias, because outcomes with positive results might be more likely to reported in the abstract than negative results [ 47 ]. Our citation searches should have mitigated both issues somewhat since they rely on publications citing each other, rather than containing specific words.

The review used machine learning for study selection reducing the study selection workload by 95%. Our experience confirms the widely documented potential of automated and semi-automated methods to improve systematic review efficiency [ 48 , 49 ]. The workload savings enabled us to spend more time in discussions with content experts.

The review results are affected by statistical heterogeneity in the analyses, which may be due to methodological and clinical heterogeneity in the variables, as well as the large variability in the design and conduct of the studies. There were not enough studies to explore heterogeneity using subgroup and sensitivity analyses, nor to test for publication bias. In many instances, results come from a single study, which greatly reduces the applicability of the findings considering none of the studies had low RoB.

We limited eligible studies to those undertaken in high income countries and as a result we cannot generalize our findings to low- or middle-income countries. The body of evidence comes from nationwide surveys in schools in the USA and Europe.

Implications for research

There is a need for more evidence on how best to identify records which report research into modes of data collection.

Some of the analyses showed unexpected results which might merit further research. These include lifetime alcohol use being higher when a web survey was used compared to PAPI, although there was no difference for lifetime marijuana use. Also, the evidence of differences in reported mental health for telephone compared with web surveys merit further investigation. Whether and in what situations web surveys yield poorer response rates compared to other modes in adolescents should also be investigated in future studies.

The absence of research evidence on the impact of survey mode on mental health scores by gender or other demographic characteristics, suggests this area could merit research.

There is a need for research that could better reflect the current situation where adolescents’ use of the internet and smart phones is widespread.

Implications for practice

Survey designers must balance practical concerns against the sampling, non-response and measurement error associated with specific design features. This review, and others, highlight methods and strategies that may improve survey response rates among adolescents with minimal impact on the assessment of mental health status [ 11 , 12 , 42 ]. Based on the poor reporting in the included studies, authors should be encouraged to register their trials and make their protocols publicly available. Authors and journal editors should follow the CONSORT reporting guidelines [ 50 ].

Despite the absence of low RoB studies, few studies for some comparisons and the focus on research undertaken in high income countries, there are methods and strategies that could be considered for improving survey response rates among adolescents being surveyed about mental health and substance use. For example, the use of monetary incentives may lead to higher response rates. Our review show that survey mode has limited impact on response rates in surveys delivered in school settings. Outside school settings, web surveys may be superior to other modes, but more research is needed to determine this. More studies using controlled designs are needed to further identify effective methods and strategies to ensure adequate response rates among adolescents. Some studies indicate that mental health scores may differ between certain survey modes. Finally, there was limited evidence on any differences between gender and survey characteristics on mental health scores.

Availability of data and materials

The templates for data collection, the extracted data and the data used for all of the analyses are available from the main author upon reasonable request.

Abbreviations

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Randomised controlled trial

Standardized mean difference

Grading of Recommendations, Assessment, Development, and Evaluations

Mean difference

Paper-and-pencil

Child Health Questionnaire

Children’s Depression Inventory

Spence Children’s Anxiety Scale

RAND 36-Item Short Form Health Survey

Hospital Anxiety and Depression Scale

World Health Organization. Adolescent mental health [web page] Geneva: World Health Organization; 2021 [updated 17.11.2021. Available from: https://www.who.int/news-room/fact-sheets/detail/adolescent-mental-health .

Vos T, Lim SS, Abbafati C, Abbas KM, Abbasi M, Abbasifard M, et al. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020;396(10258):1204–22.

Article   Google Scholar  

Jackson SL. Research Methods and Statistics: A Critical Approach. 4th ed. Andover: Cengage Learning; 2011.

Google Scholar  

Kalb LG, Cohen C, Lehmann H, Law P. Survey non-response in an internet-mediated, longitudinal autism research study. J Am Med Inform Assoc. 2012;19(4):668–73.

Article   PubMed   PubMed Central   Google Scholar  

Lallukka T, Pietiläinen O, Jäppinen S, Laaksonen M, Lahti J, Rahkonen O. Factors associated with health survey response among young employees: a register-based study using online, mailed and telephone interview data collection methods. BMC Public Health. 2020;20(1):184.

Groves RM, Peytcheva E. The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis. Public Opin Q. 2008;72(2):167–89.

Volken T. Second-stage non-response in the Swiss health survey: determinants and bias in outcomes. BMC Public Health. 2013;13(1):167.

Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. JAMA. 2012;307(17):1805–6.

Article   CAS   PubMed   Google Scholar  

Stedman RC, Connelly NA, Heberlein TA, Decker DJ, Allred SB. The End of the (Research) World As We Know It? Understanding and Coping With Declining Response Rates to Mail Surveys. Soc Nat Resour. 2019;32(10):1139–54.

McQuillan G, Kruszon-Moran D, Di H, Schaar D, Lukacs S, Fakhouri T, et al. Assessing consent for and response to health survey components in an era of falling response rates: National Health and Nutrition Examination Survey, 2011–2018. Survey Research Methods. 2021;15(3):257–68.

PubMed   PubMed Central   Google Scholar  

Fan W, Yan Z. Factors affecting response rates of the web survey: A systematic review. Comput Hum Behav. 2010;26(2):132–9.

Daikeler J, Bošnjak M, Lozar MK. Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates. J Surv Stat Methodol. 2020;8(3):513–39.

Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database of Syst Rev. 2009;2009(3):MR000008.

Lygidakis C, Rigon S, Cambiaso S, Bottoli E, Cuozzo F, Bonetti S, et al. A web-based versus paper questionnaire on alcohol and tobacco in adolescents. Telemed J E Health. 2010;16(9):925–30.

Townsend L, Kobak K, Kearney C, Milham M, Andreotti C, Escalera J, et al. Development of Three Web-Based Computerized Versions of the Kiddie Schedule for Affective Disorders and Schizophrenia Child Psychiatric Diagnostic Interview: Preliminary Validity Data. J Am Acad Child Adolesc Psychiatry. 2020;59(2):309–25.

Article   PubMed   Google Scholar  

Van De Looij-Jansen PM, De Wilde EJ. Comparison of web-based versus paper-and-pencil self-administered questionnaire: effects on health indicators in Dutch adolescents. Health Serv Res. 2008;43(5 Pt 1):1708–21.

Clarke M, Oxman AD, Paulsen E, Higgins JPT, Green S. Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 510 (updated March 2011): Cochrane Collaboration; 2011.

Bidonde MJ, Bang L, Brunborg GS, Hafstad EV, Meneses Echavez JF. Methods, strategies and incentives to increase response to questionnaires and surveys among adolescents. - Protocol for a methodological systematic review - prosjektbeskrivelse. Oslo: Norwegian Institute of Public Health; 2022.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40–6.

Thomas J, Brunton J, Graziosi S. EPPI-Reviewer 4.0: software for research synthesis. London, UK: Social Science Research Unit, Institute of Education, University of London; 2010.

Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley; 2019.

Book   Google Scholar  

Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368:l6890.

Higgins JPT, Deeks JJ. Chapter 7. Selecting studies and collecting data. In: Hggins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 510 (updated March 2011): Cochrane Collaboration; 2011.

Pejtersen JH. The effect of monetary incentive on survey response for vulnerable children and youths: A randomized controlled trial. PLoS One. 2020;15(5):ArtID e0233025.

McMorris BJ, Petrie RS, Catalano RF, Fleming CB, Haggerty KP, Abbott RD. Use of web and in-person survey modes to gather data from young adults on sex and drug use: An evaluation of cost, time, and survey error based on a randomized mixed-mode design. Eval Rev. 2009;33(2):138–58.

Wettergren L, Mattsson E, von Essen L. Mode of administration only has a small effect on data quality and self-reported health status and emotional distress among Swedish adolescents and young adults. J Clin Nurs. 2011;20(11–12):1568–77.

Erhart M, Wetzel RM, Krugel A, Ravens-Sieberer U. Effects of phone versus mail survey methods on the measurement of health-related quality of life and emotional and behavioural problems in adolescents. BMC Public Health. 2009;9:491.

Mauz E, Hoffmann R, Houben R, Krause L, Kamtsiuris P, Goswald A. Mode equivalence of health indicators between data collection modes and mixed-mode survey designs in population-based health interview surveys for children and adolescents: Methodological study. J Med Internet Res. 2018;20(3):e64.

Denniston MM, Brener ND, Kann L, Eaton DK, McManus T, Kyle TM, et al. Comparison of paper-and-pencil versus Web administration of the Youth Risk Behavior Survey (YRBS): Participation, data quality, and perceived privacy and anonymity. Comput Hum Behav. 2010;26(5):1054–60.

Eaton DK, Brener ND, Kann L, Denniston MM, McManus T, Kyle TM, et al. Comparison of paper-and-pencil versus web administration of the youth risk behavior survey (YRBS): Risk behavior prevalence estimates. Eval Rev. 2010;34(2):137–53.

Hamann C, Schultze-Lutter F, Tarokh L. Web-based assessment of mental well-being in early adolescence: A reliability study. J Med Internet Res. 2016;18(6):e138.

Miech RA, Couper MP, Heeringa SG, Patrick ME. The impact of survey mode on US national estimates of adolescent drug prevalence: Results from a randomized controlled study. Addiction. 2021;116(5):1144–51.

Raat H, Mangunkusumo RT, Landgraf JM, Kloek G, Brug J. Feasibility, reliability, and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet administration compared with the standard paper version. Qual Life Res. 2007;16(4):675–85.

Article   PubMed Central   Google Scholar  

Raghupathy S, Hahn-Smith S. The effect of survey mode on high school risk behavior data: A comparison between web and paper-based surveys. Curr Issues in Educ. 2013;16(2):1–11.

Trapl ES. Understanding adolescent survey responses: Impact of mode and other characteristics on data outcomes and quality [Doctoral dissertation, Case Western Reserve University]. OhioLINK Electronic Theses and Dissertations Center2007.

Courser MW, Shamblen SR, Lavrakas PJ, Collins D, Ditterline P. The impact of active consent procedures on nonresponse and nonresponse error in youth survey data: Evidence from a new experiment. Eval Rev. 2009;33(4):370–95.

National Library of Medicine. The effect of monetary incentive on survey response for vulnerable children and youth [NCT01741675] Bethesda, MD: National Library of Medicine; 2014. Available from: https://clinicaltrials.gov/ct2/show/NCT01741675 .

Walser S, Killias M. Who should supervise students during self-report interviews? A controlled experiment on response behavior in online questionnaires. J Exp Criminol. 2012;8(1):17–28.

Higgins JPT, Eldridge S, Li T. Chapter 23: Including variants on randomized trials. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 63 (updated February 2022): Cochrane; 2022.

Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, et al. Increasing response rates to postal questionnaires: systematic review. BMJ. 2002;324(7347):1183.

Liu C, Cox RB Jr, Washburn IJ, Croff JM, Crethar HC. The effects of requiring parental consent for research on adolescents’ risk behaviors: A meta-analysis. J Adolesc Health. 2017;61(1):45–52.

Daikeler J, Silber H, Bošnjak M. A Meta-Analysis of How Country-Level Factors Affect Web Survey Response Rates. Int J Mark Res. 2021;64(3):306–33.

Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, et al. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 63 (updated February 2022): Cochrane; 2022.

Frandsen TF, Bruun Nielsen MF, Lindhardt CL, Eriksen MB. Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements. J Clin Epidemiol. 2020;127:69–75.

Duyx B, Swaen GMH, Urlings MJE, Bouter LM, Zeegers MP. The strong focus on positive results in abstracts may cause bias in systematic reviews: a case study on abstract reporting bias. Syst Rev. 2019;8(1):174.

Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242–53.

van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3(2):125–33.

Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.

Download references

Acknowledgements

The Department of Child and Development and the Division of Health Services within the Norwegian Institute of Public Health funded this project. We thank our colleagues Dr. Simon Lewin and Dr. Chris Ross for their time and Ingvild Kirkehei for reviewing the search strategy.

Open access funding provided by Norwegian Institute of Public Health (FHI) The authors report no funding sources.

Author information

Authors and affiliations.

Division of Health Services, Norwegian Institute of Public Health, Oslo, Norway

Julia Bidonde, Jose F. Meneses-Echavez & Elisabet Hafstad

Facultad de Cultura Física, Deporte, y Recreación, Universidad Santo Tomás, Bogotá, Colombia

Jose F. Meneses-Echavez

Department of Child Health and Development, Norwegian Institute of Public Health, Oslo, Norway

Geir Scott Brunborg & Lasse Bang

Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden

Geir Scott Brunborg

You can also search for this author in PubMed   Google Scholar

Contributions

L.B: Conceptualization (equal); Formal Analysis (equal); Writing – Original Draft Preparation (equal); Writing – Review & Editing (equal). J.B: Conceptualization (lead); Data Curation (lead); Formal Analysis (lead); Investigation (lead); Methodology (lead); Project Administration (lead); Supervision (lead); Validation (lead); Visualization (equal); Writing – Original Draft Preparation (lead); Writing – Review & Editing (lead). G.S.B: Conceptualization (equal); Formal Analysis (equal); Writing – Original Draft Preparation (equal); Writing – Review & Editing (equal). E.H: Conceptualization (equal); Investigation (equal); Methodology (equal); Writing – Original Draft Preparation (equal); Writing – Review & Editing (equal). J.F.M-E: Conceptualization (equal); Data Curation (equal); Formal Analysis (equal); Investigation (equal); Methodology (equal); Validation (equal); Visualization (lead); Writing – Original Draft Preparation (equal); Writing – Review & Editing (equal). All authors read and approved the final manuscript.

Corresponding author

Correspondence to Lasse Bang .

Ethics declarations

Ethics approval and consent to participate.

This study pooled anonymized data from existing studies. No ethical approval was therefore required for the present study. The original studies that collected the data acquired ethical approvals and consents from participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Eligibility criteria and Glossary.

Additional file 2.

Search strategies and lists of excluded studies.

Additional file 3.

Detailed data extraction for the included studies.

Additional file 4.

Risk of bias assessment.

Additional file 5.

PRISMA checklist.

Additional file 6.

Additional Forest plots.

Additional file 7.

Protocol changes.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Cite this article.

Bidonde, J., Meneses-Echavez, J.F., Hafstad, E. et al. Methods, strategies, and incentives to increase response to mental health surveys among adolescents: a systematic review. BMC Med Res Methodol 23 , 270 (2023). https://doi.org/10.1186/s12874-023-02096-z

Download citation

Received : 04 April 2023

Accepted : 06 November 2023

Published : 16 November 2023

DOI : https://doi.org/10.1186/s12874-023-02096-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Adolescents
• Mental health
• Surveys and questionnaires
• Systematic review

BMC Medical Research Methodology

ISSN: 1471-2288

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Online First
• Enhancing title and abstract screening for systematic reviews with GPT-3.5 turbo
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0002-8182-0582 Omid Kohandel Gargari 1 ,
• Mohammad Hossein Mahmoudi 2 ,
• Mahsa Hajisafarali 1 ,
• http://orcid.org/0009-0006-4862-1131 Reza Samiee 3 , 4
• 1 Alborz Artificial Intelligence Association , Alborz University of Medical Sciences , Karaj , Alborz , Iran (the Islamic Republic of)
• 2 Industrial Engineering Department , Sharif University of Technology , Tehran , Iran (the Islamic Republic of)
• 3 NCWEB Association , Tehran University of Medical Sciences , Tehran , Iran (the Islamic Republic of)
• 4 Students’ Scientific Research Center , Tehran University of Medical Sciences , Tehran , Iran (the Islamic Republic of)
• Correspondence to Dr Omid Kohandel Gargari, Alborz Artificial Intelligence Association, Alborz University of Medical Sciences, Karaj, Alborz, Iran (the Islamic Republic of); kohandelgargar{at}gmail.com

http://dx.doi.org/10.1136/bmjebm-2023-112678

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• Systematic Reviews as Topic
• Health Services Research

After conducting a database search, the subsequent phase in the execution of systematic reviews (SRs) involves title and abstract screening. 1 This stage bears significant importance and necessitates the involvement of dedicated and experienced researchers who can exhibit precision and accuracy, particularly when the search yields a substantial number of studies. Besides the qualities of experience and dedication demonstrated by the screeners, several other factors influence the quality of the screening process, such as effective team management, the adoption of a double-screening approach and, notably, the implementation of a well-structured screening design. A screening tool comprises a set of questions that must be addressed by the screeners, and these questions should adhere to the following criteria: (1) they must be objective, (2) they should be single-barrelled and (3) they should encompass questions answerable with ‘yes’, ‘no’ or ‘unsure’ responses. 2

The domain of large language and transformer models has showcased a promising trajectory of advancement, consistently improving day by day. These models are trained on a vast corpora of text and possess the capability to comprehend and generate human-like text. 3 A prominent example within this realm is the Generative Pre-Trained Transformer (GPT) developed by OpenAI, with the latest iteration being GPT-4 at the time of composing this discourse. GPT-4 has exhibited commendable performance across a range of human-related tasks and has surpassed its predecessor, GPT-3.5, in evaluations conducted by the company. 4

This single-case study was conceived to assess the performance of GPT 3.5 in the context of title and abstract screening for SRs. To execute this task, a recently published SR titled ‘Light Therapy in Insomnia Disorder: A Systematic Review and Meta-Analysis’ was selected, and the databases were queried using the keywords stipulated in the original paper. 5 Two key rationales underpinned the selection of this review: first, it yielded a relatively moderate number of studies, and second, its eligibility criteria were somewhat subjective, and challenging to discern during the screening process, making it a suitable testbed to evaluate GPT-3.5’s capabilities. For instance, this study enrolled patients experiencing sleep difficulties but did not specify the particular types of sleep disorders although reviewers did not face much trouble but models had difficulties with studies that included patients with secondary sleeping troubles like patients with cancer. Furthermore, it was unclear which specific light therapy was chosen for inclusion.

The initial search yielded 330 citations, which were subsequently imported into EndNote X20. An RTF file containing titles and abstracts was generated, followed by its conversion into a more compatible TXT format, thus facilitating further data processing. This transformation laid the foundation for our experimental data set, comprising the research paper titles, abstracts and accompanying metadata. The screening team consisted of three researchers: (1) an expert with screening experience from over 20 SRs, (2) a senior researcher with screening experience from 10 SRs and (3) a junior researcher without any prior screening experience. All researchers possessed a strong command of the English language and a thorough understanding of SR principles.

Supplemental material

To compare the performance of eligibility screening of the human screeners with the performance of GPT-3.5, a range of prompts were devised for GPT by two of the authors (OKG and MHM), these prompts were carefully chosen during several discussion sessions. A prompt is a specific input or instruction given to a language model, to generate a desired output or response. The integration of the OpenAI GPT 3.5 Turbo API played a pivotal role in advancing our research. This powerful tool enabled us to initiate requests to evaluate the pertinence of prompts to individual papers. The binary relevancy results of this interaction were recorded in a structured Pandas data frame that had been prepared in advance. The code for this process is available at the provided link: https://github.com/mamishere/Article-Relevancy-Extraction-GPT3.5-Turbo .

GPT evaluated the eligibility of studies based on the provided prompts, resulting in the creation of numpy arrays containing binary outcomes for each prompt response. These numpy arrays, along with the numpy arrays generated by the researchers, were employed to compute sensitivity, specificity, accuracy and the F1 score for both the researchers and the prompts. The labels used as the ‘gold standard’ were the studies included in the selected SR. 5

Prompt 1, which replicated the criteria from the original paper, demonstrated 80% accuracy and 62% sensitivity. In contrast, Prompt 2 broadened the population by focusing on patients with ‘sleep troubles’ instead of providing a specific definition, leading to reduced accuracy, sensitivity, and specificity.

For subsequent prompts, GPT was assigned the role of an ‘Experienced Systematic Researcher’ (Prompt 3), which increased accuracy while decreasing sensitivity. Prompt 4 introduced an ‘inclusivity sentence’ to instruct the model to include studies it was uncertain about, prioritising inclusivity, leading to increased sensitivity and reduced specificity.

Prompt 5, which combined the original criteria with the inclusion of an ‘inclusivity sentence’, demonstrated the highest sensitivity, similar to the junior researcher and surpassing the senior researcher. Prompt 6, which omitted segmented criteria in favour of a more descriptive sentence, resulted in a significant reduction in sensitivity, suggesting that GPT responded better to segmented and classic criteria.

Prompt 7 assessed the impact of phrasing by modifying the language of the previous prompt and providing a more detailed description of the inclusivity phrase. This modification substantially increased the model’s sensitivity.

Lastly, Prompt 8 introduced a screening tool with four questions, requiring the model to include a study if the answer to all four questions was ‘Yes’ or ‘Unsure’. Surprisingly, the model performed poorly in this format, potentially due to the technical structure of GPT.

GPT is a potent tool, and we propose its usage in title and abstract screening for SRs, following the method we have delineated in this case report, alongside other researchers. 6 However, it is imperative to recognise that GPT is not yet fully capable of independently completing this task and should be employed as an assistant to mitigate the risk of overlooking potential studies.

Notably, even the human researchers did not attain exemplary performance, attributable to the inherent challenges posed by the subjective criteria and the absence of clear, objective definitions. We advise researchers wanting to deploy GPT to manually screen a proportion of titles and abstracts, experiment with different prompts and consider the combination of findings, a method unexplored in our study. It is crucial to first establish a clear study objective before designing prompts.

The most formidable challenge encountered in this study pertained to defining the population for the model. For instance, the lack of clarity in the criteria for ‘sleep troubles’ resulted in substantial bias in the model’s performance and significant disparities among researchers. It needs to be acknowledged that the findings of this single-case study are not generalisable, and each study objective necessitates its distinct format. This study serves as an illustrative example and offers guidance for replication with more cases and further research on the topic.

The prompt texts are available in the online supplemental table 1 .

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

• Higgins JPT ,
• Chandler J , et al
• Polanin JR ,
• Pigott TD ,
• Espelage DL , et al
• Vaswani A ,
• Shazeer N ,
• Parmar N , et al
• ↵ GPT-4 technical report ; 2023 . Available : https://ui.adsabs.harvard.edu/abs/2023arXiv230308774O
• Reynaud E ,
• Maruani J , et al
• Deng J , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1
• Data supplement 2

Twitter @Omidkohandelg

Contributors OKG: Designed the research, wrote the manuscript, data analysis, title and abstract screening. MHM: Wrote Python codes. MH and RS: Title and abstract screening.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Provenance and peer review Not commissioned; internally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF: