methodological review of the literature

Which review is that? A guide to review types

• Which review is that?
• Review Comparison Chart
• Decision Tool
• Critical Review
• Integrative Review
• Narrative Review
• State of the Art Review
• Narrative Summary
• Systematic Review
• Meta-analysis
• Comparative Effectiveness Review
• Diagnostic Systematic Review
• Network Meta-analysis
• Prognostic Review
• Psychometric Review
• Review of Economic Evaluations
• Systematic Review of Epidemiology Studies
• Living Systematic Reviews
• Umbrella Review
• Review of Reviews
• Rapid Review
• Rapid Evidence Assessment
• Rapid Realist Review
• Qualitative Evidence Synthesis
• Qualitative Interpretive Meta-synthesis
• Qualitative Meta-synthesis
• Qualitative Research Synthesis
• Framework Synthesis - Best-fit Framework Synthesis
• Meta-aggregation
• Meta-ethnography
• Meta-interpretation
• Meta-narrative Review
• Meta-summary
• Thematic Synthesis
• Mixed Methods Synthesis
• Narrative Synthesis
• Bayesian Meta-analysis
• EPPI-Centre Review
• Critical Interpretive Synthesis
• Realist Synthesis - Realist Review
• Scoping Review
• Mapping Review
• Systematised Review
• Concept Synthesis
• Expert Opinion - Policy Review
• Technology Assessment Review

Methodological Review

• Systematic Search and Review

A methodological review is a type of systematic secondary research (i.e., research synthesis) which focuses on summarising the state-of-the-art methodological practices of research in a substantive field or topic" (Chong et al, 2021).

Methodological reviews "can be performed to examine any methodological issues relating to the design, conduct and review of research studies and also evidence syntheses". Munn et al, 2018)

Further Reading/Resources

Clarke, M., Oxman, A. D., Paulsen, E., Higgins, J. P. T., & Green, S. (2011). Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Cochrane Handbook for systematic reviews of interventions . Full Text PDF

Aguinis, H., Ramani, R. S., & Alabduljader, N. (2023). Best-Practice Recommendations for Producers, Evaluators, and Users of Methodological Literature Reviews. Organizational Research Methods, 26(1), 46-76. https://doi.org/10.1177/1094428120943281 Full Text

Jha, C. K., & Kolekar, M. H. (2021). Electrocardiogram data compression techniques for cardiac healthcare systems: A methodological review. IRBM . Full Text

References Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC medical research methodology , 18 (1), 1-9. Full Text Chong, S. W., & Reinders, H. (2021). A methodological review of qualitative research syntheses in CALL: The state-of-the-art. System , 103 , 102646. Full Text

• << Previous: Technology Assessment Review
• Next: Systematic Search and Review >>
• Last Updated: Jul 4, 2024 8:46 AM
• URL: https://unimelb.libguides.com/whichreview

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved July 8, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Jul 8, 2024 3:13 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

• Privacy Policy

Home » Literature Review – Types Writing Guide and Examples

Literature Review – Types Writing Guide and Examples

Table of Contents

Literature Review

Definition:

A literature review is a comprehensive and critical analysis of the existing literature on a particular topic or research question. It involves identifying, evaluating, and synthesizing relevant literature, including scholarly articles, books, and other sources, to provide a summary and critical assessment of what is known about the topic.

Types of Literature Review

Types of Literature Review are as follows:

• Narrative literature review : This type of review involves a comprehensive summary and critical analysis of the available literature on a particular topic or research question. It is often used as an introductory section of a research paper.
• Systematic literature review: This is a rigorous and structured review that follows a pre-defined protocol to identify, evaluate, and synthesize all relevant studies on a specific research question. It is often used in evidence-based practice and systematic reviews.
• Meta-analysis: This is a quantitative review that uses statistical methods to combine data from multiple studies to derive a summary effect size. It provides a more precise estimate of the overall effect than any individual study.
• Scoping review: This is a preliminary review that aims to map the existing literature on a broad topic area to identify research gaps and areas for further investigation.
• Critical literature review : This type of review evaluates the strengths and weaknesses of the existing literature on a particular topic or research question. It aims to provide a critical analysis of the literature and identify areas where further research is needed.
• Conceptual literature review: This review synthesizes and integrates theories and concepts from multiple sources to provide a new perspective on a particular topic. It aims to provide a theoretical framework for understanding a particular research question.
• Rapid literature review: This is a quick review that provides a snapshot of the current state of knowledge on a specific research question or topic. It is often used when time and resources are limited.
• Thematic literature review : This review identifies and analyzes common themes and patterns across a body of literature on a particular topic. It aims to provide a comprehensive overview of the literature and identify key themes and concepts.
• Realist literature review: This review is often used in social science research and aims to identify how and why certain interventions work in certain contexts. It takes into account the context and complexities of real-world situations.
• State-of-the-art literature review : This type of review provides an overview of the current state of knowledge in a particular field, highlighting the most recent and relevant research. It is often used in fields where knowledge is rapidly evolving, such as technology or medicine.
• Integrative literature review: This type of review synthesizes and integrates findings from multiple studies on a particular topic to identify patterns, themes, and gaps in the literature. It aims to provide a comprehensive understanding of the current state of knowledge on a particular topic.
• Umbrella literature review : This review is used to provide a broad overview of a large and diverse body of literature on a particular topic. It aims to identify common themes and patterns across different areas of research.
• Historical literature review: This type of review examines the historical development of research on a particular topic or research question. It aims to provide a historical context for understanding the current state of knowledge on a particular topic.
• Problem-oriented literature review : This review focuses on a specific problem or issue and examines the literature to identify potential solutions or interventions. It aims to provide practical recommendations for addressing a particular problem or issue.
• Mixed-methods literature review : This type of review combines quantitative and qualitative methods to synthesize and analyze the available literature on a particular topic. It aims to provide a more comprehensive understanding of the research question by combining different types of evidence.

Parts of Literature Review

Parts of a literature review are as follows:

Introduction

The introduction of a literature review typically provides background information on the research topic and why it is important. It outlines the objectives of the review, the research question or hypothesis, and the scope of the review.

Literature Search

This section outlines the search strategy and databases used to identify relevant literature. The search terms used, inclusion and exclusion criteria, and any limitations of the search are described.

Literature Analysis

The literature analysis is the main body of the literature review. This section summarizes and synthesizes the literature that is relevant to the research question or hypothesis. The review should be organized thematically, chronologically, or by methodology, depending on the research objectives.

Critical Evaluation

Critical evaluation involves assessing the quality and validity of the literature. This includes evaluating the reliability and validity of the studies reviewed, the methodology used, and the strength of the evidence.

The conclusion of the literature review should summarize the main findings, identify any gaps in the literature, and suggest areas for future research. It should also reiterate the importance of the research question or hypothesis and the contribution of the literature review to the overall research project.

The references list includes all the sources cited in the literature review, and follows a specific referencing style (e.g., APA, MLA, Harvard).

How to write Literature Review

Here are some steps to follow when writing a literature review:

• Define your research question or topic : Before starting your literature review, it is essential to define your research question or topic. This will help you identify relevant literature and determine the scope of your review.
• Conduct a comprehensive search: Use databases and search engines to find relevant literature. Look for peer-reviewed articles, books, and other academic sources that are relevant to your research question or topic.
• Evaluate the sources: Once you have found potential sources, evaluate them critically to determine their relevance, credibility, and quality. Look for recent publications, reputable authors, and reliable sources of data and evidence.
• Organize your sources: Group the sources by theme, method, or research question. This will help you identify similarities and differences among the literature, and provide a structure for your literature review.
• Analyze and synthesize the literature : Analyze each source in depth, identifying the key findings, methodologies, and conclusions. Then, synthesize the information from the sources, identifying patterns and themes in the literature.
• Write the literature review : Start with an introduction that provides an overview of the topic and the purpose of the literature review. Then, organize the literature according to your chosen structure, and analyze and synthesize the sources. Finally, provide a conclusion that summarizes the key findings of the literature review, identifies gaps in knowledge, and suggests areas for future research.
• Edit and proofread: Once you have written your literature review, edit and proofread it carefully to ensure that it is well-organized, clear, and concise.

Examples of Literature Review

Here’s an example of how a literature review can be conducted for a thesis on the topic of “ The Impact of Social Media on Teenagers’ Mental Health”:

• Start by identifying the key terms related to your research topic. In this case, the key terms are “social media,” “teenagers,” and “mental health.”
• Use academic databases like Google Scholar, JSTOR, or PubMed to search for relevant articles, books, and other publications. Use these keywords in your search to narrow down your results.
• Evaluate the sources you find to determine if they are relevant to your research question. You may want to consider the publication date, author’s credentials, and the journal or book publisher.
• Begin reading and taking notes on each source, paying attention to key findings, methodologies used, and any gaps in the research.
• Organize your findings into themes or categories. For example, you might categorize your sources into those that examine the impact of social media on self-esteem, those that explore the effects of cyberbullying, and those that investigate the relationship between social media use and depression.
• Synthesize your findings by summarizing the key themes and highlighting any gaps or inconsistencies in the research. Identify areas where further research is needed.
• Use your literature review to inform your research questions and hypotheses for your thesis.

For example, after conducting a literature review on the impact of social media on teenagers’ mental health, a thesis might look like this:

“Using a mixed-methods approach, this study aims to investigate the relationship between social media use and mental health outcomes in teenagers. Specifically, the study will examine the effects of cyberbullying, social comparison, and excessive social media use on self-esteem, anxiety, and depression. Through an analysis of survey data and qualitative interviews with teenagers, the study will provide insight into the complex relationship between social media use and mental health outcomes, and identify strategies for promoting positive mental health outcomes in young people.”

Reference: Smith, J., Jones, M., & Lee, S. (2019). The effects of social media use on adolescent mental health: A systematic review. Journal of Adolescent Health, 65(2), 154-165. doi:10.1016/j.jadohealth.2019.03.024

Reference Example: Author, A. A., Author, B. B., & Author, C. C. (Year). Title of article. Title of Journal, volume number(issue number), page range. doi:0000000/000000000000 or URL

Applications of Literature Review

some applications of literature review in different fields:

• Social Sciences: In social sciences, literature reviews are used to identify gaps in existing research, to develop research questions, and to provide a theoretical framework for research. Literature reviews are commonly used in fields such as sociology, psychology, anthropology, and political science.
• Natural Sciences: In natural sciences, literature reviews are used to summarize and evaluate the current state of knowledge in a particular field or subfield. Literature reviews can help researchers identify areas where more research is needed and provide insights into the latest developments in a particular field. Fields such as biology, chemistry, and physics commonly use literature reviews.
• Health Sciences: In health sciences, literature reviews are used to evaluate the effectiveness of treatments, identify best practices, and determine areas where more research is needed. Literature reviews are commonly used in fields such as medicine, nursing, and public health.
• Humanities: In humanities, literature reviews are used to identify gaps in existing knowledge, develop new interpretations of texts or cultural artifacts, and provide a theoretical framework for research. Literature reviews are commonly used in fields such as history, literary studies, and philosophy.

Role of Literature Review in Research

Here are some applications of literature review in research:

• Identifying Research Gaps : Literature review helps researchers identify gaps in existing research and literature related to their research question. This allows them to develop new research questions and hypotheses to fill those gaps.
• Developing Theoretical Framework: Literature review helps researchers develop a theoretical framework for their research. By analyzing and synthesizing existing literature, researchers can identify the key concepts, theories, and models that are relevant to their research.
• Selecting Research Methods : Literature review helps researchers select appropriate research methods and techniques based on previous research. It also helps researchers to identify potential biases or limitations of certain methods and techniques.
• Data Collection and Analysis: Literature review helps researchers in data collection and analysis by providing a foundation for the development of data collection instruments and methods. It also helps researchers to identify relevant data sources and identify potential data analysis techniques.
• Communicating Results: Literature review helps researchers to communicate their results effectively by providing a context for their research. It also helps to justify the significance of their findings in relation to existing research and literature.

Purpose of Literature Review

Some of the specific purposes of a literature review are as follows:

• To provide context: A literature review helps to provide context for your research by situating it within the broader body of literature on the topic.
• To identify gaps and inconsistencies: A literature review helps to identify areas where further research is needed or where there are inconsistencies in the existing literature.
• To synthesize information: A literature review helps to synthesize the information from multiple sources and present a coherent and comprehensive picture of the current state of knowledge on the topic.
• To identify key concepts and theories : A literature review helps to identify key concepts and theories that are relevant to your research question and provide a theoretical framework for your study.
• To inform research design: A literature review can inform the design of your research study by identifying appropriate research methods, data sources, and research questions.

Characteristics of Literature Review

Some Characteristics of Literature Review are as follows:

• Identifying gaps in knowledge: A literature review helps to identify gaps in the existing knowledge and research on a specific topic or research question. By analyzing and synthesizing the literature, you can identify areas where further research is needed and where new insights can be gained.
• Establishing the significance of your research: A literature review helps to establish the significance of your own research by placing it in the context of existing research. By demonstrating the relevance of your research to the existing literature, you can establish its importance and value.
• Informing research design and methodology : A literature review helps to inform research design and methodology by identifying the most appropriate research methods, techniques, and instruments. By reviewing the literature, you can identify the strengths and limitations of different research methods and techniques, and select the most appropriate ones for your own research.
• Supporting arguments and claims: A literature review provides evidence to support arguments and claims made in academic writing. By citing and analyzing the literature, you can provide a solid foundation for your own arguments and claims.
• I dentifying potential collaborators and mentors: A literature review can help identify potential collaborators and mentors by identifying researchers and practitioners who are working on related topics or using similar methods. By building relationships with these individuals, you can gain valuable insights and support for your own research and practice.
• Keeping up-to-date with the latest research : A literature review helps to keep you up-to-date with the latest research on a specific topic or research question. By regularly reviewing the literature, you can stay informed about the latest findings and developments in your field.

Advantages of Literature Review

There are several advantages to conducting a literature review as part of a research project, including:

• Establishing the significance of the research : A literature review helps to establish the significance of the research by demonstrating the gap or problem in the existing literature that the study aims to address.
• Identifying key concepts and theories: A literature review can help to identify key concepts and theories that are relevant to the research question, and provide a theoretical framework for the study.
• Supporting the research methodology : A literature review can inform the research methodology by identifying appropriate research methods, data sources, and research questions.
• Providing a comprehensive overview of the literature : A literature review provides a comprehensive overview of the current state of knowledge on a topic, allowing the researcher to identify key themes, debates, and areas of agreement or disagreement.
• Identifying potential research questions: A literature review can help to identify potential research questions and areas for further investigation.
• Avoiding duplication of research: A literature review can help to avoid duplication of research by identifying what has already been done on a topic, and what remains to be done.
• Enhancing the credibility of the research : A literature review helps to enhance the credibility of the research by demonstrating the researcher’s knowledge of the existing literature and their ability to situate their research within a broader context.

Limitations of Literature Review

Limitations of Literature Review are as follows:

• Limited scope : Literature reviews can only cover the existing literature on a particular topic, which may be limited in scope or depth.
• Publication bias : Literature reviews may be influenced by publication bias, which occurs when researchers are more likely to publish positive results than negative ones. This can lead to an incomplete or biased picture of the literature.
• Quality of sources : The quality of the literature reviewed can vary widely, and not all sources may be reliable or valid.
• Time-limited: Literature reviews can become quickly outdated as new research is published, making it difficult to keep up with the latest developments in a field.
• Subjective interpretation : Literature reviews can be subjective, and the interpretation of the findings can vary depending on the researcher’s perspective or bias.
• Lack of original data : Literature reviews do not generate new data, but rather rely on the analysis of existing studies.
• Risk of plagiarism: It is important to ensure that literature reviews do not inadvertently contain plagiarism, which can occur when researchers use the work of others without proper attribution.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Data Verification – Process, Types and Examples

Research Results Section – Writing Guide and...

Research Gap – Types, Examples and How to...

Table of Contents – Types, Formats, Examples

Ethical Considerations – Types, Examples and...

Research Paper Conclusion – Writing Guide and...

• Open access
• Published: 07 September 2020

A tutorial on methodological studies: the what, when, how and why

• Lawrence Mbuagbaw   ORCID: orcid.org/0000-0001-5855-5461 1 , 2 , 3 ,
• Daeria O. Lawson 1 ,
• Livia Puljak 4 ,
• David B. Allison 5 &
• Lehana Thabane 1 , 2 , 6 , 7 , 8  

BMC Medical Research Methodology volume  20 , Article number:  226 ( 2020 ) Cite this article

40k Accesses

61 Altmetric

Metrics details

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Peer Review reports

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 , 2 , 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 , 7 , 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig.  1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 , 13 , 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

Comparing two groups

Determining a proportion, mean or another quantifier

Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.

Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].

Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]

Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 , 66 , 67 ].

Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].

Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].

Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].

Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n  = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n  = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n  = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

What is the aim?

Methodological studies that investigate bias

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies that investigate quality (or completeness) of reporting

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Methodological studies that investigate the consistency of reporting

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

Methodological studies that investigate factors associated with reporting

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies that investigate methods

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Methodological studies that summarize other methodological studies

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Methodological studies that investigate nomenclature and terminology

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

Other types of methodological studies

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

What is the design?

Methodological studies that are descriptive

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Methodological studies that are analytical

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

What is the sampling strategy?

Methodological studies that include the target population

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n  = 103) [ 30 ].

Methodological studies that include a sample of the target population

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

What is the unit of analysis?

Methodological studies with a research report as the unit of analysis

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Methodological studies with a design, analysis or reporting item as the unit of analysis

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig.  2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Availability of data and materials

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Abbreviations

Consolidated Standards of Reporting Trials

Evidence, Participants, Intervention, Comparison, Outcome, Timeframe

Grading of Recommendations, Assessment, Development and Evaluations

Participants, Intervention, Comparison, Outcome, Timeframe

Preferred Reporting Items of Systematic reviews and Meta-Analyses

Studies Within a Review

Studies Within a Trial

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

PubMed   Google Scholar  

Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.

PubMed   PubMed Central   Google Scholar  

Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj. 2017;358:j4008.

Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Studies. 2020;6(1):13.

Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020.

Abbade LPF, Wang M, Sriganesh K, Jin Y, Mbuagbaw L, Thabane L. The framing of research questions using the PICOT format in randomized controlled trials of venous ulcer disease is suboptimal: a systematic survey. Wound Repair Regen. 2017;25(5):892–900.

Gohari F, Baradaran HR, Tabatabaee M, Anijidani S, Mohammadpour Touserkani F, Atlasi R, Razmgir M. Quality of reporting randomized controlled trials (RCTs) in diabetes in Iran; a systematic review. J Diabetes Metab Disord. 2015;15(1):36.

Wang M, Jin Y, Hu ZJ, Thabane A, Dennis B, Gajic-Veljanoski O, Paul J, Thabane L. The reporting quality of abstracts of stepped wedge randomized trials is suboptimal: a systematic survey of the literature. Contemp Clin Trials Commun. 2017;8:1–10.

Shanthanna H, Kaushal A, Mbuagbaw L, Couban R, Busse J, Thabane L: A cross-sectional study of the reporting quality of pilot or feasibility trials in high-impact anesthesia journals Can J Anaesthesia 2018, 65(11):1180–1195.

Kosa SD, Mbuagbaw L, Borg Debono V, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemporary Clinical Trials. 2018;65:144–50.

Zhang Y, Florez ID, Colunga Lozano LE, Aloweni FAB, Kennedy SA, Li A, Craigie S, Zhang S, Agarwal A, Lopes LC, et al. A systematic survey on reporting and methods for handling missing participant data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:57–66.

CAS   PubMed   Google Scholar  

Hernández AV, Boersma E, Murray GD, Habbema JD, Steyerberg EW. Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading? Am Heart J. 2006;151(2):257–64.

Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, Fruci V, Dennis B, Bawor M, Thabane L. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary-of-findings tables with a new format. J Clin Epidemiol. 2016;74:7–18.

The Northern Ireland Hub for Trials Methodology Research: SWAT/SWAR Information [ https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/ ]. Accessed 31 Aug 2020.

Chick S, Sánchez P, Ferrin D, Morrice D. How to conduct a successful simulation study. In: Proceedings of the 2003 winter simulation conference: 2003; 2003. p. 66–70.

Google Scholar  

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106(3):485–8.

Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med New York. 1996;63(3–4):216–24.

CAS   Google Scholar  

Areia M, Soares M, Dinis-Ribeiro M. Quality reporting of endoscopic diagnostic studies in gastrointestinal journals: where do we stand on the use of the STARD and CONSORT statements? Endoscopy. 2010;42(2):138–47.

Knol M, Groenwold R, Grobbee D. P-values in baseline tables of randomised controlled trials are inappropriate but still common in high impact journals. Eur J Prev Cardiol. 2012;19(2):231–2.

Chen M, Cui J, Zhang AL, Sze DM, Xue CC, May BH. Adherence to CONSORT items in randomized controlled trials of integrative medicine for colorectal Cancer published in Chinese journals. J Altern Complement Med. 2018;24(2):115–24.

Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

The Cochrane Methodology Register Issue 2 2009 [ https://cmr.cochrane.org/help.htm ]. Accessed 31 Aug 2020.

Mbuagbaw L, Kredo T, Welch V, Mursleen S, Ross S, Zani B, Motaze NV, Quinlan L. Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis. J Clin Epidemiol. 2016;74:66–72.

Barton S, Peckitt C, Sclafani F, Cunningham D, Chau I. The influence of industry sponsorship on the reporting of subgroup analyses within phase III randomised controlled trials in gastrointestinal oncology. Eur J Cancer. 2015;51(18):2732–9.

Setia MS. Methodology series module 5: sampling strategies. Indian J Dermatol. 2016;61(5):505–9.

Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;3(4):337–46.

Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, Neumann I, Li L, Lopes LC, Briel M, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, et al. Is this clinical trial fully registered?: a statement from the International Committee of Medical Journal Editors*. Ann Intern Med. 2005;143(2):146–8.

Ohtake PJ, Childs JD. Why publish study protocols? Phys Ther. 2014;94(9):1208–9.

Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.

Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002;359(9302):248–52.

Porta M (ed.): A dictionary of epidemiology, 5th edn. Oxford: Oxford University Press, Inc.; 2008.

El Dib R, Tikkinen KAO, Akl EA, Gomaa HA, Mustafa RA, Agarwal A, Carpenter CR, Zhang Y, Jorge EC, Almeida R, et al. Systematic survey of randomized trials evaluating the impact of alternative diagnostic strategies on patient-important outcomes. J Clin Epidemiol. 2017;84:61–9.

Helzer JE, Robins LN, Taibleson M, Woodruff RA Jr, Reich T, Wish ED. Reliability of psychiatric diagnosis. I. a methodological review. Arch Gen Psychiatry. 1977;34(2):129–33.

Chung ST, Chacko SK, Sunehag AL, Haymond MW. Measurements of gluconeogenesis and Glycogenolysis: a methodological review. Diabetes. 2015;64(12):3996–4010.

CAS   PubMed   PubMed Central   Google Scholar  

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002;21(11):1513–24.

Moen EL, Fricano-Kugler CJ, Luikart BW, O’Malley AJ. Analyzing clustered data: why and how to account for multiple observations nested within a study participant? PLoS One. 2016;11(1):e0146721.

Zyzanski SJ, Flocke SA, Dickinson LM. On the nature and analysis of clustered data. Ann Fam Med. 2004;2(3):199–200.

Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152.

Bui DDA, Del Fiol G, Hurdle JF, Jonnalagadda S. Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform. 2016;64:265–72.

Bui DD, Del Fiol G, Jonnalagadda S. PDF text classification to leverage information extraction from publication reports. J Biomed Inform. 2016;61:141–8.

Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19(1):32.

Speich B. Blinding in surgical randomized clinical trials in 2015. Ann Surg. 2017;266(1):21–2.

Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, et al. A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews. J Clin Epidemiol. 2017;84:37–46.

Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, Min J, Wang G, Fu J, Mao B. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: A survey of reports indexed in the Science Citation Index Expanded. Eur J Integrative Med. 2011;3(4):e309–16.

Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007;50(4):266–77.

Oltean H, Gagnier JJ. Use of clustering analysis in randomized controlled trials in orthopaedic surgery. BMC Med Res Methodol. 2015;15:17.

Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS One. 2014;9(5):e96407.

Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg. 2006;244(5):663–7.

de Vries TW, van Roon EN. Low quality of reporting adverse drug reactions in paediatric randomised controlled trials. Arch Dis Child. 2010;95(12):1023–6.

Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13.

Kaiser KA, Cofield SS, Fontaine KR, Glasser SP, Thabane L, Chu R, Ambrale S, Dwary AD, Kumar A, Nayyar G, et al. Is funding source related to study reporting quality in obesity or nutrition randomized control trials in top-tier medical journals? Int J Obes. 2012;36(7):977–81.

Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes. 2008;32(10):1531–6.

Khan NR, Saad H, Oravec CS, Rossi N, Nguyen V, Venable GT, Lillard JC, Patel P, Taylor DR, Vaughn BN, et al. A review of industry funding in randomized controlled trials published in the neurosurgical literature-the elephant in the room. Neurosurgery. 2018;83(5):890–7.

Hansen C, Lundh A, Rasmussen K, Hrobjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev. 2019;8:Mr000047.

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280–5.

Liu LQ, Morris PJ, Pengel LH. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.

Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, Vandvik PO, Malaga G, Johnston BC, Dahm P, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95.

Lee SY, Teoh PJ, Camm CF, Agha RA. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–72.

Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494–500.

Alvarez F, Meyer N, Gourraud PA, Paul C. CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals. Br J Dermatol. 2009;161(5):1159–65.

Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemporary Clin trials. 2014;38(2):245–50.

Thabane L, Chu R, Cuddy K, Douketis J. What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes. 2007;31(10):1554–9.

Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence Based Med. 2017;22(4):139.

METRIC - MEthodological sTudy ReportIng Checklist: guidelines for reporting methodological studies in health research [ http://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#METRIC ]. Accessed 31 Aug 2020.

Jager KJ, Zoccali C, MacLeod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int. 2008;73(3):256–60.

Parker SG, Halligan S, Erotocritou M, Wood CPJ, Boulton RW, Plumb AAO, Windsor ACJ, Mallett S. A systematic methodological review of non-randomised interventional studies of elective ventral hernia repair: clear definitions and a standardised minimum dataset are needed. Hernia. 2019.

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.

Schiller P, Burchardi N, Niestroj M, Kieser M. Quality of reporting of clinical non-inferiority and equivalence randomised trials--update and extension. Trials. 2012;13:214.

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Thabut G, Estellat C, Boutron I, Samama CM, Ravaud P. Methodological issues in trials assessing primary prophylaxis of venous thrombo-embolism. Eur Heart J. 2005;27(2):227–36.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64.

Ritchie A, Seubert L, Clifford R, Perry D, Bond C. Do randomised controlled trials relevant to pharmacy meet best practice standards for quality conduct and reporting? A systematic review. Int J Pharm Pract. 2019.

Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, Pericic TP, Pieper D, Puljak L. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2019.

Tan A, Porcher R, Crequit P, Ravaud P, Dechartres A. Differences in treatment effect size between overall survival and progression-free survival in immunotherapy trials: a Meta-epidemiologic study of trials with results posted at ClinicalTrials.gov. J Clin Oncol. 2017;35(15):1686–94.

Croitoru D, Huang Y, Kurdina A, Chan AW, Drucker AM. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182(6):1469–76.

Khan MS, Ochani RK, Shaikh A, Vaduganathan M, Khan SU, Fatima K, Yamani N, Mandrola J, Doukky R, Krasuski RA: Assessing the Quality of Reporting of Harms in Randomized Controlled Trials Published in High Impact Cardiovascular Journals. Eur Heart J Qual Care Clin Outcomes 2019.

Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME. From conference abstract to full paper: differences between data presented in conferences and journals. FASEB J. 2005;19(7):673–80.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, Scott P. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, et al. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol. 2017;17(1):181.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

Analytical study [ https://medical-dictionary.thefreedictionary.com/analytical+study ]. Accessed 31 Aug 2020.

Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6 e381.

Schalken N, Rietbergen C. The reporting quality of systematic reviews and Meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol. 2017;8:1395.

Ranker LR, Petersen JM, Fox MP. Awareness of and potential for dependent error in the observational epidemiologic literature: A review. Ann Epidemiol. 2019;36:15–9 e12.

Paquette M, Alotaibi AM, Nieuwlaat R, Santesso N, Mbuagbaw L. A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation. Syst Rev. 2019;8(1):241.

Download references

Acknowledgements

This work did not receive any dedicated funding.

Author information

Authors and affiliations.

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Lawrence Mbuagbaw, Daeria O. Lawson & Lehana Thabane

Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario, L8N 4A6, Canada

Lawrence Mbuagbaw & Lehana Thabane

Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Lawrence Mbuagbaw

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN, 47405, USA

David B. Allison

Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

Lehana Thabane

Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Lawrence Mbuagbaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mbuagbaw, L., Lawson, D.O., Puljak, L. et al. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 20 , 226 (2020). https://doi.org/10.1186/s12874-020-01107-7

Download citation

Received : 27 May 2020

Accepted : 27 August 2020

Published : 07 September 2020

DOI : https://doi.org/10.1186/s12874-020-01107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodological study
• Meta-epidemiology
• Research methods
• Research-on-research

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

QALY maximisation and people's preferences: a methodological review of the literature

Affiliation.

• 1 Sheffield Health Economics Group, School of Health and Related Research, University of Sheffield, UK. [email protected]
• PMID: 15386656
• DOI: 10.1002/hec.924

In cost-utility analysis, the numbers of quality-adjusted life years (QALYs) gained are aggregated according to the sum-ranking (or QALY maximisation) rule. This requires that the social value from health improvements is a simple product of gains in quality of life, length of life and the number of persons treated. The results from a systematic review of the literature suggest that QALY maximisation is descriptively flawed. Rather than being linear in quality and length of life, it would seem that social value diminishes in marginal increments of both. And rather than being neutral to the characteristics of people other than their propensity to generate QALYs, the social value of a health improvement seems to be higher if the person has worse lifetime health prospects and higher if that person has dependents. In addition, there is a desire to reduce inequalities in health. However, there are some uncertainties surrounding the results, particularly in relation to what might be affecting the responses, and there is the need for more studies of the general public that attempt to highlight the relative importance of various key factors.

Copyright (c) 2004 John Wiley & Sons, Ltd.

PubMed Disclaimer

Similar articles

• Societal Preferences for Interventions with the Same Efficiency: Assessment and Application to Decision Making. Shiroiwa T, Saito S, Shimozuma K, Kodama S, Noto S, Fukuda T. Shiroiwa T, et al. Appl Health Econ Health Policy. 2016 Jun;14(3):375-85. doi: 10.1007/s40258-016-0236-3. Appl Health Econ Health Policy. 2016. PMID: 26940671
• Public preferences for responsibility versus public preferences for reducing inequalities. Edlin R, Tsuchiya A, Dolan P. Edlin R, et al. Health Econ. 2012 Dec;21(12):1416-26. doi: 10.1002/hec.1799. Epub 2011 Nov 10. Health Econ. 2012. PMID: 22072569
• The QALY model and individual preferences for health states and health profiles over time: a systematic review of the literature. Tsuchiya A, Dolan P. Tsuchiya A, et al. Med Decis Making. 2005 Jul-Aug;25(4):460-7. doi: 10.1177/0272989X05276854. Med Decis Making. 2005. PMID: 16061898 Review.
• QALY-maximisation and public preferences: results from a general population survey. Bryan S, Roberts T, Heginbotham C, McCallum A. Bryan S, et al. Health Econ. 2002 Dec;11(8):679-93. doi: 10.1002/hec.695. Health Econ. 2002. PMID: 12457369
• Utilitarianism and the measurement and aggregation of quality--adjusted life years. Dolan P. Dolan P. Health Care Anal. 2001;9(1):65-76. doi: 10.1023/A:1011387524579. Health Care Anal. 2001. PMID: 11372576 Review.
• Preferences as fairness judgments: a critical review of normative frameworks of preference elicitation and development of an alternative based on constitutional economics. Rogowski W, John J. Rogowski W, et al. Cost Eff Resour Alloc. 2024 Jan 30;22(1):10. doi: 10.1186/s12962-024-00510-x. Cost Eff Resour Alloc. 2024. PMID: 38291472 Free PMC article. Review.
• Explicit discrimination and ingroup favoritism, but no implicit biases in hypothetical triage decisions during COVID-19. Gradwohl N, Neth H, Giese H, Gaissmaier W. Gradwohl N, et al. Sci Rep. 2024 Jan 12;14(1):1213. doi: 10.1038/s41598-023-50385-w. Sci Rep. 2024. PMID: 38216652 Free PMC article.
• QALY-type preference and willingness-to-pay among end-of-life patients with cancer treatments: a pilot study using discrete choice experiment. Yin Y, Peng Q, Ma L, Dong Y, Sun Y, Xu S, Ding N, Liu X, Zhao M, Tang Y, Mei Z, Shao H, Yan D, Tang W. Yin Y, et al. Qual Life Res. 2024 Mar;33(3):753-765. doi: 10.1007/s11136-023-03562-3. Epub 2023 Dec 11. Qual Life Res. 2024. PMID: 38079024
• Real-world disparities and ethical considerations with access to CFTR modulator drugs: Mind the gap! Zampoli M, Morrow BM, Paul G. Zampoli M, et al. Front Pharmacol. 2023 Mar 27;14:1163391. doi: 10.3389/fphar.2023.1163391. eCollection 2023. Front Pharmacol. 2023. PMID: 37050905 Free PMC article. Review.
• Reference-dependent age weighting of quality-adjusted life years. Attema AE, Brouwer WBF, Pinto-Prades JL. Attema AE, et al. Health Econ. 2022 Dec;31(12):2515-2536. doi: 10.1002/hec.4593. Epub 2022 Sep 4. Health Econ. 2022. PMID: 36057854 Free PMC article.

Publication types

• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Methodological Overlap in Early Childhood Education Self-Regulation Research: An Interdisciplinary Systematic Review

• Open access
• Published: 09 July 2024

Cite this article

You have full access to this open access article

• Robbie A. Ross   ORCID: orcid.org/0000-0002-2615-9621 1 &
• Kate E. Ascetta 1  

Young children’s self-regulation (SR) skills are linked to many important outcomes across the lifespan and school stakeholders widely agree that these skills should be prioritized in schools. Despite broad agreement about the importance of these skills, the diverse field of SR research is rife with a lack of clarity in both conceptual definitions and measurement with many different terms and definitions used interchangeably. With a systematic review of early childhood SR research in schools, we aimed to add to these efforts by cataloguing which SR variables researchers measure and the specific methodological choices made to capture such variables. Coders identified 319 peer-reviewed articles published between 2010–2020 that included a school-based measurement of self-regulation related constructs among children between 3- and 8-years. This revealed 11 distinct variable terms used widely in the literature to refer to SR related skills or characteristics and behavior problems was overwhelmingly the most frequent. Studies of many of these variables relied heavily on adult reports, usually teachers, and the use of multiple informants to measure variables was uncommon. Findings further highlight the conceptual clutter in SR research, point to possible points of empirical weaknesses in the measurement of these skills, and identify some methodological overlap in how variables are being operationalized in the literature.

Avoid common mistakes on your manuscript.

Introduction

Self-regulation (SR) skills—a broad and sometimes poorly defined constellation of abilities at the heart of children’s development of many basic life skills—is a popular area of research. In recent years, the field has seen significant interest not only from developmental and education researchers, but also from practitioners, educators, and education policy makers. And for good reason – SR skills in early life have been linked to a wide array of important academic, social, and behavioral outcomes across the lifespan (e.g., Moffitt et al., 2011 ; Woodward et al., 2017 ). Specific terms and definitions vary depending on the research field (e.g., self-management, self-control, willpower, effortful control, hot and cool executive functioning, etc.), but nearly all refer to a set of skills that enable individuals to monitor and regulate prepotent or automatic behaviors, emotions, and attention to pursue a goal (Burman et al., 2015 ). Over the last several decades of expansion and growth in this area of research, the field has manifested considerable variability in how SR related skills are conceptually defined and operationally measured (e.g., Jones et al., 2019 ). For example, single terms like “self-control” are commonly used to refer to multiple, distinct concepts depending on the context (e.g., delay of gratification, inhibitory control; Jones et al., 2016a ; Thorndike, 1904 ). Similarly, multiple distinct terms are often used to describe concepts that are likely to share considerable conceptual overlap (Jones et al., 2019 ; Kelley, 1927 ). The academic field of SR research is plagued by conceptual clutter and definitional discrepancies which have given rise to substantial challenges in the measurement of such skills and constructs for both academic research and practical applications in schools (e.g., Jones et al., 2016c , 2019 ; McKown, 2019 ). This lack of consensus and poorly defined taxonomy of self-regulation and related skills in the academic literature presents significant challenges to effective dissemination of findings across disciplines at the academic level and may lead to wasteful spending of public education dollars on ineffective programs adopted to target students’ self-regulation skill development in schools. With this systematic review of school-based self-regulation research, we aim to take a first step towards clarifying these conceptual definitions by cataloguing and analyzing how scholars are measuring self-regulation related skills and concepts in their research.

Conceptual Clutter in Self-Regulation Research

The definitional vagueness and conceptual overlap among self-regulation related variables and constructs have been described many times by several scholars in the field (e.g., Booth et al., 2018 ; Duckworth & Kern, 2011 ; Eisenberg et al., 2010 ; Hofmann et al., 2012 ; Morrison & Grammer, 2016 ; Nigg, 2017 ; Zhou et al., 2012 ). Cognitive neuroscience researchers focus on what is termed cognitive control which is defined generally as one’s ability to flexibly adapt behaviors and cognitive information processing in service of an internal goal (e.g., Hutchison & Morton, 2016 ). Social psychologists and education researchers view self-regulation and self-control , terms often used interchangeably within the field, as the ability to modify and regulate emotions and behaviors in service of goals (e.g., Blair, 2002 ; Hagger et al., 2010 ). However, others suggest that self-control and self-regulation refer to related, but distinct concepts (e.g., Hofmann et al., 2012 ). Although the monitoring and regulation of emotions and one’s expressions of them is included in many conceptual definitions of self-regulation related concepts, some scholars also draw definitional distinctions between emotion-related self-regulation and emotion-regulation (e.g., Eisenberg et al., 2010 ). Cognitive and developmental psychologists have focused on executive function skills which are considered to be a set of multidimensional cognitive abilities used to control thoughts, emotions, and actions (e.g., Diamond, 2014 ). Although executive function research itself is rife with conceptual clutter and definitional overlap (Jones et al., 2016b ; Morrison & Grammer, 2016 ), these cognitive abilities are often thought to subserve the behavioral manifestations of self-regulation and self-control typically studied in social psychology (e.g., Heatherton, 2011 ; Hofmann et al., 2012 ). Other scholars suggest that most conceptual differences between executive function and effortful control —another construct often associated with self-regulation—reflect differences in measurement traditions across the two siloed fields rather than a substantive difference between the two constructs (Zhou et al., 2012 ).

Despite differing opinions about the nature of these constructs and their relationships with one another, these skills are all approached from a strengths-building perspective, and the importance of these skills for future success, regardless of what they are called, is well-documented. Across each of these research subfields, a reliable pattern exists such that stronger performance on tasks measuring self-regulation constructs is associated with a wide range of positive academic, social, and health outcomes; poor performance on these tasks is associated with negative outcomes, and has been linked to some pathological developments in childhood and/or adulthood (e.g., Jones et al., 2015 ; Moffitt et al., 2011 ; Robson et al., 2020 ). Some constructs have been more closely linked with specific outcomes than others. For example, executive function skills in preschool have been linked with performance in mathematics in later childhood (e.g., Verdine et al., 2014 ). Conversely, some areas of research approach similar phenomena from a deficit perspective, focusing on the measurement of behavior problems , usually framed as symptoms of clinical or subclinical psychopathology, as evidence or manifestation of a lack of self-regulatory functioning (e.g., Lonigan et al., 2017 ). For example, effortful control and impulsivity in early elementary grades have been linked with symptoms of emotional and behavioral disorders in adolescence (e.g., Wang et al., 2015 ). Self-regulation skills, measured in a variety of ways, have been consistently and inversely linked to certain externalizing and internalizing behavior problems, to varying extents, across development in early life in both clinical and non-clinical populations (e.g., Eisenberg et al., 2010 ; Lonigan et al., 2017 ; Willoughby et al., 2011 ). Although these conceptualizations tend to frame behavior problems as a correlate of self-regulation skills rather than a component of them, we include this variable in our review due to the frequency with which behavior problems are measured in educational research.

Justification for this Review

Within the field of self-regulation research, scholars have worked to align and clarify conceptual definitions across related subdomains that are often considered to be part and parcel of self-regulation. Most often, these reviews of the literature and calls for integration across models are based on analyses of construct definitions and patterns of findings across subfields (e.g., Booth et al., 2018 ; Hofmann et al., 2012 ; Zhou et al., 2012 ). While these approaches are critical to untangling the web of terminology that clutters the field, a catalogue of the measurement tools and measurement approaches used to capture these variables is needed. The specific methodologies selected to operationalize variables, however consistently defined, ultimately drive the conclusions drawn from the patterns discovered in the data. Efforts that scholars have made on this front have already begun to underscore the challenges inherent to a field that is characterized by vague conceptual definitions and many measurement tools to choose from. For example, a meta-analysis of a wide range of self-control tasks and questionnaires found only modest convergence across them, highlighting that even when definitions align across studies, the specific measurement tool selected to capture the variable might contribute significantly to the study outcome and scholars’ interpretations of the relations among these constructs (Duckworth & Kern, 2011 ). Similarly, another meta-analysis of 150 studies investigating the associations between early childhood self-regulation and various outcomes including academics showed that the informant used in the measurement of self-regulation (e.g., parent-report, teacher-report, direct assessment) was a significant moderator of pooled mean effects (Robson et al., 2020 ).

With this systematic review of recent school-based research on children’s self-regulation skills, we contribute to the efforts to declutter this field by answering the following research question: How are researchers measuring self-regulation skills among young children in educational contexts? To address this, we aimed to develop a catalog of the self-regulation related variables researchers study, the specific measurement tools used to measure these variables, and the informants recruited to provide data. To our knowledge this effort represents the first attempt to systematically review the recent educational literature regarding the specific measurements used to capture self-regulation related skills among young children.

A systematic literature review was conducted utilizing the PRISMA protocol (Page et al., 2021 ); see Fig.  1 for PRISMA protocol flow chart.

PRISMA flow chart

Search Strategy

The first and second author conducted independent searches of PsycINFO and Educational Resources Information Center (ERIC) EBSCO for empirical English language articles published in peer-reviewed journals from 2010 through 2020. We chose this specific publication time period in light of the widespread disruptions to daily school functioning and in-school data collection resulting from the COVID-19 pandemic. As school-based data collection methods were dramatically altered or paused altogether in 2020 and 2021, we limited our search to focus exclusively on research that was conducted and published prior to these disruptions as it is likely to reflect researchers’ data collection procedures in the future as schools return to pre-pandemic functioning. Our aim was to capture all empirical articles published in this period that included a measurement of a self-regulation related variable (SR variable) that was captured within an educational setting among young children. For the initial database searches, the following search term combinations were used: (a) “emotion* regulation” AND “early childhood” + school OR preschool* + school; (b) “challenging behavior*” AND “early childhood” + school OR preschool* + school; (c) “behavior problem*”AND “early childhood” + school OR preschool* + school, (d) “self-regulation” AND “early childhood” + school OR preschool* + school; (e) “self-management” AND “early childhood” + school OR preschool* + school; (f) “executive function” AND “early childhood” + school OR preschool* + school. Many of these variables overlap in their content and definitions, such as behavior problems and challenging behaviors, however scholars often use terms interchangeably and inconsistently across studies. Our aim was to capture as many articles as possible that include variables related to self-regulation to build a catalog of studies.

Eligibility Criteria

Studies were included based on several criteria. First, studies measured at least one of the SR variables listed in our search terms. This was determined based on an article authors’ descriptions of variables and measurement tools included in the methods sections of the paper. In cases where variable labels were vague (e.g., social-emotional skills), coders determined eligibility based on descriptions of these variables and included them when they were characterized as being similar in nature to one of the search terms (e.g., regulating emotions, controlling behavior, aggression). Second, studies included child participants who were 3- to 8-years-old at the time of data collection. In cases of longitudinal designs that included data collection at ages outside of this range, studies were included if the child participants were in the 3- to 8-year-old range at the time that at least one relevant variable was measured. Third, data collection for the relevant SR variable took place in a school setting (e.g., Head Start, childcare center, public 4 k, early intervention classroom, second grade classroom). Finally, to allow comparisons across similar educational settings, only studies with a U.S. sample were included in the review. For the sake of clarity, throughout this literature review we are using the term variable to refer to the constructs that article authors describe measuring (e.g., executive function), and the term measurement tool to refer to the specific assessment, questionnaire, or task authors used to measure that variable.

Data Extraction and Coding

The first and second authors’ searches of the PsycINFO and ERIC databases yielded 5,225 studies. After removing 1,659 duplicates, the remaining 3,566 articles were screened in two phases. First, using a Qualtrics survey, the titles and abstracts of the articles were screened. If an article was coded as “exclude” the coder(s) selected an option that indicated the reason for exclusion from the drop-down menu in the Qualtrics survey. Please see Fig.  1 for specific reasons articles were removed during screening. The initial screening process yielded 583 included articles. Following this initial screening, inter-rater agreement, or the extent to which both coders agreed an article should be included or excluded, was calculated across 30% ( n  = 1070) of the screened articles. Percent agreement for inclusion and exclusion decisions across coders was 89%. In the second phase of screening, eligibility for inclusion was determined by conducting a full read of 583 articles, yielding 319 included articles. Percent agreement across coders in this phase was 90%.

Descriptive Coding

Descriptive data were extracted from the remaining 319 articles. An excel data sheet was created by the coders for data collection purposes. For training, 30 included articles were randomly selected for each coder to fully code and disagreements were discussed until reliability was obtained. First, coders recorded the specific measurement tool used to capture the relevant variable. It was common for multiple relevant SR variables to be included within an eligible article and for multiple measurement tools to be used to measure the same variable in a study.

Next, each coder recorded the reported informant(s) for each measurement tool. Informants were coded as either teacher, parent, other school official, researcher, peer, or child. Due to the young ages of child participants in this literature review, none of the included articles featured child self-report measures. Measurements that featured a direct child assessment that included having the child complete a specific task or test in real time were coded as having a child informant. When a child’s behaviors in a naturalistic setting (i.e., classroom, playground) were observed or video-recorded and then coded by a researcher, these measurements were coded as direct observations with a researcher informant. Thus, measurements with either a researcher or child informant represent what we consider direct child-level measurements. These are contrasted with adult-report measures, such as parent- or teacher-completed questionnaires about a child’s characteristics. In the case of an adult-reported questionnaire, the informant was considered the person who completed it. Some measurement tools feature direct assessments of children as well as an assessor’s report of the child’s overall behavior patterns during the tasks irrespective of their actual scores on the assessment (e.g., Preschool Self-Regulation Assessment, PSRA; Smith-Donald et al., 2007 ). When children’s direct assessment scores were considered separately from the assessor’s report, these were coded as distinct measurements. The direct assessment score was considered a direct child assessment with a child informant, while the assessor’s report was considered an observation with a researcher informant.

Next, coders recorded the SR variable (i.e., emotion regulation , challenging behavior , behavior problems , self-regulation , self-management , executive function , other , and unclear ) that the specific measurement tool captured, according to the article author(s). Given that a large portion of the variables (44.5%) were initially coded as other or unclear, the coders conferred and decided to revise the coding procedures to better categorize the variables that our search yielded. A new category was then determined if the variable was reportedly measured more than five times across all included articles (e.g., effortful control, approaches to learning). Therefore, if a variable was measured fewer than five times (i.e., 0.7% of all measurements or less), it was categorized as “other” (e.g., proneness to fear, engagement, mental flexibility). This secondary coding resulted in the final SR variable categories that will be presented in the results: (a) emotion regulation , (b) behavior problems , (c) self-regulation , (d) executive function , (e) approaches to learning , (f) attention , (g) effortful control , (h) behavior/social/emotional competence , (i) behavioral regulation , (j) social-emotional skills , and (k) other .

Inter-rater agreement (IRA), or the extent to which both coders distinguished between data extracted from eligible articles, was calculated for 30% of all included articles via percent agreement; IRA was 87.8%. Disagreements between the coders were resolved by discussion. In addition, Cohen’s kappa statistic was run to assess the agreement between the two coders ( k  = 0.74; McHugh, 2012 ).

Our findings are based on data extracted directly from the methods sections of 319 articles. To address our research questions, our analyses focused primarily on the specific measurement tools researchers used to capture various SR related variables in their studies. Across these included articles, authors reported 701 unique measurements of 11 different SR related variables (see Table  1 ). By far, the most frequently measured variable was behavior problems (38% of all measurements) followed by self-regulation (14%) and executive function (12%). Secondly, authors used 74 different measurement tools to assess their reported SR variables (see Table  2 for the most frequently used measurement tools). Lastly, in addition to analyzing the SR variables and measurement tools used across articles, we also catalogued the informants who provided information for each measurement tool. Other school officials and peers each only served as informants once among the included variables and are thus excluded from most of the analyses regarding informants. The remaining measurement tools used in the included articles were either questionnaires that adults (parents or teachers) completed on behalf of a target child or direct measurements of the target child via behavioral observation in a specific context (e.g., classroom, playground), or via direct assessment of the child. Table 3 provides a breakdown of the 11 SR variables and associated frequency for each type of informant. Adult-reports represented a considerable majority, 69%, of the measurements of all SR variables and teachers comprised a substantial portion (75%) of these, making teachers the most frequently used informant among the studies included in this analysis.

Overview of Measurement Tools

Across the 319 included articles, 74 distinct measurement tools were catalogued that captured 11 different SR variables; however, the 14 most frequently used measurement tools—those that were reportedly used 10 times or more (see Table  2 )—captured 564 (80%) of the 701 total instances of an SR variable measurement. In cases where revised or modified versions of similar measurement tools were reported, these were collapsed and grouped together. For example, uses of the Teacher Observation of Classroom Adaptation—Revised (TOCA-R; Werthamer-Larsson et al., 1991 ), and the Teacher Observation of Classroom Adaptation—Checklist (TOCA-C; Leaf et al., 2002 ) are grouped together as the same measurement tool (TOCA).

In addition to calculating the number of different variables each measurement tool was used to capture, we also considered the consistency with which each tool was used to capture the same variable. Table 2 shows a breakdown of the 14 most frequently used measurement tools and the frequency with which each tool was used to measure each of the included SR variables. In the following section, we report our findings regarding the top 5 most frequently used measurement tools. Although direct assessments were the most commonly used individual measurement tool, the overwhelming majority of measurements of children’s skills came from adults’ reports.

Direct Child Assessments

The most commonly used measurement tool was a direct child assessment. We coded any measures that required the target child to complete a specific task or test in real time which was then scored by a researcher as a direct child assessment. This resulted in a measurement tool category that includes several different specific assessments or tasks such as the PSRA (Smith-Donald et al., 2007 ), HTKS (McClelland et al., 2014 ), and Dimensional Change Card Sort (DCCS; Zelazo, 2006 ). This kind of tool was used 164 times, representing nearly 23% of all measurements included in the review. While direct child assessments include a variety of different behavioral and cognitive tasks and assessments, this methodological approach was most often used to measure executive function ( n  = 78) and self-regulation ( n  = 50). It was also used to measure behavioral regulation ( n  = 9), effortful control ( n  = 8), attention ( n  = 7), other variables (e.g., learning engagement, n  = 4), emotion regulation ( n  = 3), and approaches to learning ( n  = 2). By far, the most frequently used tool within this category was the PSRA ( n  = 69), which includes a battery of ten behavioral tasks (e.g., pencil tap, snack delay, balance beam), or one or more individual tasks selected from the battery or tasks very similar to these (i.e., gift delay; Kochanska et al., 1996 ). According to the developers, the PSRA is designed to capture young children’s behavioral, emotional and attentional self-regulation (Smith-Donald et al., 2007 ). Among the articles included in this review, researchers reported using the PSRA or similar behavioral tasks primarily to measure executive function ( n  = 29) or self-regulation ( n  = 24). For example, McLear et al. ( 2016 ) used four tasks from the PSRA battery – the balance beam, pencil tap, toy sort, and gift wrap tasks – to measure self-regulation in kindergarteners in their study investigating how parents’ representations of attachment styles and children’s self-regulation skills predict academic achievement. The HTKS, which developers describe as a measure of behavioral self-regulation and executive function, was also used 23 times to capture self-regulation ( n  = 12), behavioral regulation, ( n  = 6), and executive function ( n  = 5), (McClelland et al., 2014 ).

Social Skills Rating System

The Social Skills Rating System (SSRS; Gresham & Elliott, 1990 ) and the subsequent revisions—Social Skills Improvement System (SSIS; Gresham & Elliott, 2008 ) and the Social Rating Scale (SRS; Meisels & Atkins-Burnett, 1999 ), were collapsed and coded as a single measurement tool group (i.e., SSRS). SSRS is described as an assessment tool used to measure child and adolescent social domains (i.e., social skills, problem behaviors, and academic competence). SSRS is a multi-informant tool that includes rating forms for teachers, parents and children. We found that the SSRS tool group was selected 82 times. In their methods sections, study authors reported using the SSRS tool group to measure seven distinct SR variables: (a) behavior problems ( n  = 33); (b) social emotional skills ( n  = 15); (c) approaches to learning ( n  = 10); (d) other (e.g., adjustment problems, learning related problems; n  = 8); (e) behavioral, social, and emotional competence ( n  = 7); (f) self-regulation ( n  = 7); (g) behavioral regulation ( n  = 2). For example, Lin et al. ( 2016 ) examined peer-relations in a rural U.S. setting. The authors included a measure of problem behaviors, SSRS, to examine the impact of children’s externalizing and internalizing behaviors as a possible mediating factor on their learning-related behavior. Lin et al. ( 2016 ) found that children with higher SSRS scores in the fall had lower learning-related behavior scores; subsequently mediating their relationships with peers.

Child Behavior Checklist

The Child Behavior Checklist (CBCL; Achenbach, 1999 ) is described by the developers as an assessment tool for use by teachers and parents to screen for behavioral, social, and emotional problems. The CBCL covers eight domain categories: aggressive behavior, attention problems, anxious/depressed, rule-breaking behavior, social problems, somatic complaints, thought problems, and withdrawn/depressed. We coded 78 unique uses of the CBCL across the 319 included articles. Authors used the CBCL to measure seven SR variables with 50 teacher reports and 28 parent reports. Sixty seven of these 78 measurements (86%) of the CBCL were to measure behavior problems. For example, Bagner et al. ( 2010 ) examined the psychometric properties of an assessment tool (e.g., Revised Edition of the School Observation Coding System; REDSOCS) with 64 children diagnosed with Oppositional Defiance Disorder (ODD). The authors reported in their methods section that they used the CBCL, specifically the Aggressive Behavior scale, as part of their diagnostic screening process to identify children ages 4 to 6 years old with ODD. Parents completed the CBCL as part of the recruitment and inclusion process for the study. The CBCL was also infrequently used to measure attention, social-emotional skills, and self-, emotion, and behavioral regulation (see Table  2 ).

Direct Observation

Researchers’ observations of children’s real time behaviors in natural settings were used to measure relevant variables 66 times across the included articles. According to authors’ descriptions, this measurement tool was most frequently used to capture behavior problems ( n  = 27, 41%). For example, in their investigation of a teacher implemented classroom intervention, Conroy et al. ( 2014 ) used trained researchers to observe preschool students identified as being at-risk for developing emotional or behavioral disorders (EBD) and their teachers during typical classroom activities. The observers coded teacher behaviors, student–teacher interactions, and child behaviors including disruption, aggression, and defiance. The authors found that these observed behavior problems decreased following the implementation of the intervention (Conroy et al., 2014 ). Our catalogue of the use of this measurement tool indicates that researchers also used direct observation to measure a wide range of other variables, including self-regulation ( n  = 12), attention ( n  = 6), emotion regulation ( n  = 5), approaches to learning ( n  = 5), other variables (e.g., academic engagement, n  = 4), executive function ( n  = 3), effortful control ( n  = 3), and behavioral regulation ( n  = 1).

Children’s Behavior Questionnaire

The Children’s Behavior Questionnaire (CBQ; Rothbart et al., 2001 ) was developed to be completed by caregivers to understand the temperament of children ages 3 to 7 years. Rothbart et al. ( 2001 ) designed the CBQ to capture individual differences in temperament related to 15 characteristics (i.e., Activity Level, Anger/Frustration, Attentional Focusing, Discomfort, Fear, High Intensity Pleasure, Impulsivity, Inhibitory Control, Low Intensity Pleasure, Perceptual Sensitivity, Positive Anticipation, Sadness, Shyness, Smiling/Laughter, and Soothability) and factor analysis indicated three broad temperament dimensions: Extraversion/Surgency, Negative Affectivity, and Effortful Control. Based on our coding process, the CBQ was used 42 times across seven SR variables. The CBQ was used most often to measure effortful control ( n  = 18; 43%). For example, Valiente et al. ( 2011 ) examined the relation between child effortful control and academic achievement over a six-year period using a multi-measure and multi-reporter approach to capturing effortful control. Effortful control, using the CBQ, was reported by parents and teachers at pretest and observations of children in a laboratory. Valiente et al. ( 2011 ) found that children who had high levels of effortful control at Time 1 and persisted at difficult tasks achieved higher academic success years later. Additionally, it was used to capture self-, emotion, and behavioral regulation variables, executive function, and attention (see Table  2 ).

As a first step towards decluttering the field of self-regulation research, our aim for this project was to document how self-regulation research is being conducted in educational contexts by cataloguing the most frequently used measurement tools, the variables that researchers used them to measure, and the informants who provided the data. We searched the psychology and educational research literatures for studies that included measurements of variables related to the monitoring and regulation of behaviors, cognitions, and/or emotions.

Tools and Informants Used to Measure SR Variables

An important first step in rectifying the problem of conceptual confusion and clutter in self-regulation research is to take stock of precisely how researchers are measuring the myriad variables that are often included in the constellation of terms related to self-regulation. Our findings revealed that some measurement tools were used almost exclusively to measure a specific variable, while others were used more diversely to capture multiple different variables. For example, most of the times that researchers used the CBCL to capture an SR variable, it was for the measurement of behavior problems. The same pattern was true for several other tools including the Behavioral Assessment Scales for Children (Reynolds & Kamphaus, 1992 ), Strengths and Difficulties Questionnaire (Goodman, 2006 ), Adjustment Scales for Preschool Intervention (Lutz et al., 2002 ), TOCA (Werthamer-Larsson et al., 1991 ), and MacArthur Health and Behavior Questionnaire (Armstrong & Goldstein, 2003 ). Each of these measurement tools were used to measure several different SR variables, but usage was highly concentrated in behavior problems, suggesting a fair amount of agreement across scholars regarding how best to measure this variable.

Direct child assessments were the most frequently used measurements across all variables. Forty-one percent and 31% of all direct child assessments were used to measure executive function and self-regulation, respectively. Among the direct child assessments, the PSRA or similar tasks were used most frequently, and nearly all these uses were to measure self-regulation or executive function, further highlighting the conceptual and measurement overlap across these two variables. While our catalogue shows that the PSRA battery and similar tasks were the most common specific tasks within this measurement tool category, this review did not further investigate which specific tasks (e.g., pencil tap, snack delay) were used across each of these variables. Analyzing these differences will be important to pursue to further clarify the ways in which researchers are measuring self-regulation and executive function when assessed directly from children to determine the extent of empirical overlap across these two variables (Jones et al., 2016a ; Morrison & Grammer, 2016 ). However, it is clear that the literature on self-regulation and executive function captured in this review is almost exclusively based on direct child assessments of these skills.

This analysis also revealed several measurement tools that seem to be used quite diversely across the literature. For example, while 42% of CBQ uses were documented to capture effortful control, it was also used broadly to measure emotion regulation, self-regulation, executive function, attention, behavioral regulation, and other variables. Each of these tools contains several subscales and factors that can be dissociated from the larger scales which might contribute to their broad uses (Gresham & Elliott, 1990 , 2008 ; Meisels & Atkins-Burnett, 1999 ; Putnam & Rothbart, 2006 ; Rothbart et al., 2001 ). The coding protocol for this review did not consider the uses of subscales or factor scores separately from a measurement’s full scale, so we cannot conclude whether the wide range of variables that these measures are used to capture reflect uses of the full scale or of component factors or subscales. Future work will attempt to clarify any subscales used and the variables they were used to measure.

The methodological choices that researchers make to measure variables for data collection extend beyond the selection of measurement tools to the selection of the informants who will provide the data. Informants offer differential pictures of a target child’s behaviors depending on a number of factors including whether the data is collected via observation of a child in a naturalistic context, through a direct assessment with a researcher, by questionnaire completed by a teacher or parent, or children’s self-reports (e.g., Achenbach et al., 1987 ; Duckworth & Kern, 2011 ). Across all articles included in this review, children’s SR variables were primarily measured by having adults complete questionnaires about a target child’s skills and behaviors and 75% of these adult-reports were completed by teachers. However, specific methodologies and informant use varied across SR variables. For example, behavior problems were most frequently measured by teacher-report and self-regulation and executive function were much more likely to be measured via direct assessments of the child. While the use of adult-reports to measure child characteristics has considerable practical and some empirical advantages over child-level measurements—including the time, effort, and money needed to administer them and the relative convergent validity across them—such questionnaires are also limited by threats to validity through bias and differential reference contexts (Duckworth & Yeager, 2015 ). Additionally, correspondence across parent- and teacher-reports of children’s social, emotional, and behavioral functioning is consistently low to moderate (Achenbach et al., 1987 ; Duckworth & Kern, 2011 ; Meyer et al., 2001 ), most likely reflecting variance associated with a combination of differential bias for parents and teachers and real differences in behavior patterns across settings (Gresham et al., 2018 ). Relying exclusively on teachers to serve as informants may be especially practical for data collection in school settings which may contribute to the overrepresentation of teachers and underrepresentation of parents as informants across all variables included in this review. However, failing to gather reports from parents in addition to teachers systematically excludes a valuable source of contextualized information about children’s behaviors and may lead to broad misunderstandings about the nature of such variables (Achenbach et al., 1987 ; De Los Reyes et al., 2015 ).

Direct measures of child performance and behavior, although considerably more resource intensive, have certain advantages over adult-reported questionnaires (e.g., reduced effects of bias, increased ecological validity); however, meta-analyses have suggested that children’s performances on behavioral tasks tend to show even weaker convergent validity across measures than among adult-reports and may require a lengthy battery of multiple tasks to approach adequate validity (Duckworth & Kern, 2011 ; Duckworth & Yeager, 2015 ; Meyer et al., 2001 ). Without a single gold standard measurement available to capture aspects of children’s regulatory functioning and behaviors, the use of a multi-informant approach is generally agreed upon as the best strategy for improving overall reliability and validity estimates (e.g., Achenbach, 2006 ; De Los Reyes et al., 2015 ; Duckworth & Yeager, 2015 ; Kraemer et al., 2003 ), with some scholars specifically advising the use of adult-reports in conjunction with direct assessments or observations of children’s performance (Duckworth & Kern, 2011 ; Duckworth & Yeager, 2015 ). We found differences across variable categories in the uses of multiple informants to measure SR related skills and behaviors as well as the practice of collecting adult-reports in addition to child-level measurements. While 27% of studies measuring behavior problems used multiple informant methods, only 1% included a combination of both child-level measurements and adult reports. Both self-regulation and executive function were commonly measured using multiple informants and used the inclusion of both adult-reports and child-level measures frequently. These were also the variables most likely to include child-level measures at all. Together, these patterns suggest that there may be a lack of agreement across scholars regarding the best practices for measuring them and may point to vague definitions in the literature and overall poor understanding of the nature of these variables or how they might meaningfully differ from one another.

Implications for Research and Practice

Our review of the past decade of research on children’s self-regulation skills in educational settings catalogued many specific variable terms that are used frequently in academic literature to describe such skills. Our analysis of the measurement methods used to capture these variables led to several specific recommendations for future research efforts in these areas. First, our review revealed that studies of behavior problems rely heavily on teachers’ reports of students’ behavior patterns, only occasionally included additional informants in these measurements, and very rarely measured children directly. Teachers are well-positioned to provide informed and detailed accounts of student behavior patterns. However, investigations of problematic and disproportional rates of students of color referred for behavioral problems or subjected to exclusionary discipline practices (Department of Education, 2018 ) have suggested that factors related to teachers’ interpretations of behaviors, such as teacher bias (e.g., Gilliam et al., 2016 ) and teachers’ own social-emotional competence (e.g., Jennings & Greenberg, 2009 ) might play a role in these referral rates (Ura & d’Abreu, 2022 ). To balance potential such issues with teacher reports of student behaviors, scholars studying behavior problems could benefit from using a multi-informant approach, as is the currently agreed upon best practice for such measurements (e.g., Kraemer et al., 2003 ). Although collecting data from multiple informants—including the child—is especially important in cases where high stakes decisions are being made about individual students (i.e., screening for diagnoses, classroom placement, treatment plans), utilizing such practices will also offer nuanced information for researchers studying the nature and etiology of behavior problems (De Los Reyes et al., 2015 ). Additionally, including direct measures of self-regulation or related variables along with measures of behavior problems will also contribute to the understanding of the specific role that regulatory functioning plays in children’s behavior problems and can begin to address the question of whether behavior problems represent a deficit of regulatory skills (Campbell, 2016 ). These methods are resource intensive – particularly direct child assessments – and may not be feasible or appropriate for some study designs, but investing in a multi-informant approach where possible might return a deeper understanding of how behavior problems emerge and best practices for prevention and support.

Self-regulation, executive function, and behavioral regulation were most often measured with child-level measurements and effortful control was also frequently measured this way, suggesting that a large portion of our recent empirical understanding of these variables is primarily determined by observations of children in natural settings or by administering a specific task for them to complete with an experimenter. Our understanding of these variables might also benefit from a multi-informant approach that includes teacher- or parent-reports of such skills. While these variables were the most likely of all the variables included in the review to be measured with multiple informants contributing data and for those to include both child-level measurements and adult-reports, these were still in the minority. Considering the resources needed to collect behavioral data from individual children via observations or direct assessments, the addition of an adult-report from either parents or teachers in such circumstances might not be prohibitively costly and could considerably improve the validity of overall measurements of these variables (Duckworth & Kern, 2011 ; Duckworth & Yeager, 2015 ).

The remaining variables included in this review were all primarily measured using teacher-reports, with very few instances of multiple informants providing data, and very little consistency in the measurement tools used to capture each variable. Although this review did not compile specific variable term definitions used in the literature, this lack of measurement consensus may point to an overall lack of clarity or precision at the definition level for some variables (i.e., behavior/social/emotional competence, social-emotional skills, other), which will naturally give rise to a hodgepodge of strategies to operationalize them. However, complete and concise definitions for these variables do exist in their respective literatures (e.g., Darling-Churchill & Lippman, 2016 ) and scholars have carefully considered best practices for measuring them (Jones et al., 2016c ).

Limitations and Future Directions

Although we aimed to conduct an exhaustive search of the literature on self-regulation related concepts in educational settings, this precluded the inclusion of all research featuring the measurement of self-regulation. This topic is of interest to scholars across many different academic domains that extend well beyond what we covered in this review and rich bodies of literature exist for each of them (e.g., Booth et al., 2018 ). While including all of these domains in the search would not have been feasible, the theories, definitions, and measurement traditions within each of them may meaningfully influence the research being conducted in school contexts and consideration for each of these domains should be included in future attempts to declutter and clarify the field. Additionally, our search of the literature was limited to the use of ERIC and PsycINFO databases which are high quality sources of educational and psychological peer-reviewed literature. The use of other comprehensive databases such as SCOPUS or Web of Science might have introduced additional disciplines into our search results and reduced the possibility of selection bias in our included list of articles. Future catalogues of the SR literature should include searches of these databases.

We found that the most commonly used measurement tool across all SR variables included in this review was a direct assessment of child performance. Our data extraction procedures did not include the documentation of the specific tasks researchers used in these assessments or which variables they were used to measure, so we are limited in the conclusions we can draw regarding the use of direct assessments beyond the broad category. Similarly, direct observations of child behavior were also commonly used across many variables in the review, but we did not specify precisely which observation tools, coding schemes, or observation contexts were used. Future analyses of the literature, particularly efforts aimed at disentangling the measurement methods for self-regulation and executive function, will include a comparison of the specific direct assessment tasks or observation protocols used for each of these variables to identify any empirical overlap.

Many scholars have put forth substantial efforts towards clarifying the specific cognitive and behavioral skills involved in the regulation of the self in service of goals by aligning definitions and conceptual frameworks across subfields (e.g., Booth et al., 2018 ; Hofmann et al., 2012 ; Jones et al., 2016a ; Zhou et al., 2012 ), and these efforts should continue. This review aimed to contribute to this body of work by providing a catalog of measurement patterns for multiple SR variables that have been studied in educational contexts in recent years. Although we aligned measurement tools and practices with the specific SR variables that study authors used them to measure, we did not document how authors were specifically defining their variables of interest or broader conceptual frameworks in which their research may be situated. This is an important aspect of the operationalization of variables and should be included in the efforts to clarify the field. A logical next step might be to connect conceptual frameworks and variable definitions that researchers are working from to the specific measurement tools and methodological practices they use to measure those variables. This work would provide a comprehensive view of what scholars think various SR variable terms represent and how they go about measuring them and would bridge the existing work on conceptual definitions to the current work on measurement practices.

Our aim for this review was to provide a catalog and analysis of how scholars have been approaching the study of the skills and behaviors related to young children’s self-regulatory functioning in educational settings and to identify patterns among the variables described and the measurement methods used to capture these variables. Our review revealed a broad range of different variable terms that are included in this literature and the patterns of measurement of these variables suggest there may be more clarity and empirical agreement across the field for some variables than for others. More work on this front is needed to ensure that the academic work that drives educational and policy initiatives around supporting these skills in schools is accurate and methodologically sound. Honing the operational definitions and measurement tools most appropriate for this concept will contribute to more effective and efficient communication across researchers and education stakeholders. As early childhood research becomes more interdisciplinary, it is critical that researchers share a common conceptual understanding allowing us to be more targeted in our work.

Achenbach, T. M. (1999). The child behavior checklist and related instruments. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcomes assessment (2nd ed., pp. 429–466). Lawrence Erlbaum Associates Publishers.

Achenbach, T. M. (2006). As others see us: Clinical and research implications of cross-informant correlations for psychopathology. Current Directions in Psychological Science, 15 (2), 94–98. https://doi.org/10.1111/j.0963-7214.2006.00414.x

Article   Google Scholar  

Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101 (2), 213–232. https://doi.org/10.1037/0033-2909.101.2.213

Armstrong, J. M., & Goldstein, L. H. (2003). Manual for the MacArthur Health and Behavior Questionnaire (HBQ 1.0) . MacArthur Foundation Research Network on Psychopathology and Development, University of Pittsburgh

Bagner, D. M., Boggs, S. R., & Eyberg, S. M. (2010). Evidence-based school behavior assessment of externalizing behavior in young children. Education & Treatment of Children, 33 (1), 65. https://doi.org/10.1353/etc.0.0084

Blair, C. (2002). School readiness: Integrating cognition and emotion in a neurobiological conceptualization of children’s functioning at school entry. American Psychologist, 57 (2), 111–127. https://doi.org/10.1037/0003-066X.57.2.111

Booth, A., Hennessy, E., & Doyle, O. (2018). Self-regulation: Learning across disciplines. Journal of Child and Family Studies, 27 (12), 3767–3781. https://doi.org/10.1007/s10826-018-1202-5

Burman, J. T., Green, C. D., & Shanker, S. (2015). On the meanings of self-regulation: Digital Humanities in service of conceptual clarity. Child Development, 86 (5), 1507–1521. https://doi.org/10.1111/cdev.12395

Campbell, S. (2016). Measuring social and emotional development in early childhood: Should behavior problems be included? [Peer commentary on “Review of measures of social and emotional development.” Journal of Applied Developmental Psychology, 45 , 19–41. https://doi.org/10.1016/j.appdev.2016.01.008

Conroy, M. A., Sutherland, K. S., Vo, A. K., Carr, S., & Ogston, P. L. (2014). Early childhood teachers’ use of effective instructional practices and the collateral effects on young children’s behavior. Journal of Positive Behavior Interventions, 16 (2), 81–92. https://doi.org/10.1177/1098300713478

Darling-Churchill, K. E., & Lippman, L. (2016). Early childhood social and emotional development: Advancing the field of measurement. Journal of Applied Developmental Psychology, 45 , 1–7. https://doi.org/10.1016/j.appdev.2016.02.002

De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A., Burgers, D. E., & Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141 (4), 858–900. https://doi.org/10.1037/a0038498

Diamond, A. (2014). Executive functions: Insights into ways to help more children thrive. Zero to Three, 35 (2), 9–17.

Google Scholar  

Duckworth, A. L., & Kern, M. L. (2011). A meta-analysis of the convergent validity of self-control measures. Journal of Research in Personality, 45 (3), 259–268. https://doi.org/10.1016/j.jrp.2011.02.004

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44 (4), 237–251. https://doi.org/10.3102/0013189X15584327

Eisenberg, N., Spinrad, T. L., & Eggum, N. D. (2010). Emotion-related self-regulation and its relation to children's maladjustment. Annual Review of Clinical Psychology , 6 , 495–525. https://doi.org/10.1146/annurev.clinpsy.121208.131208

Goodman, R. (2006). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38 (5), 581–586. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x

Gresham, F. M., & Elliott, S. N. (1990). The social skills rating system . American Guidance Service.

Gresham, F. M., & Elliott, S. N. (2008). Social Skills Improvement System: Rating Scales . Pearson Assessments.

Gresham, F. M., Elliott, S. N., Metallo, S., Byrd, S., Wilson, E., & Cassidy, K. (2018). Cross-informant agreement of children’s social-emotional skills: An investigation of ratings by teachers, parents, and students from a nationally representative sample. Psychology in the Schools, 55 (2), 208–223. https://doi.org/10.1002/pits.22101

Gilliam, W. S., Maupin, A. N., Reyes, C. R., Accavitti, M., & Shic, F. (2016). Do early educators’ implicit biases regarding sex and race relate to behavior expectations and recommendations of preschool expulsions and suspensions? [Research study brief] . Yale University Child Study Center, 9, 1–18. https:// marylandfamiliesengage.org/wp-content/uploads/ 2019/07/Preschool-Implicit-Bias-Policy-Brief.pdf

Hagger, M. S., Wood, C. W., Stiff, C., & Chatzisarantis, N. L. (2010). Self-regulation and self-control in exercise: The strength-energy model. International Review of Sport and Exercise Psychology, 3 (1), 62–86. https://doi.org/10.1080/17509840903322815

Heatherton, T. F. (2011). Neuroscience of self and self-regulation. Annual Review of Psychology, 62 , 363–390. https://doi.org/10.1146/annurev.psych.121208.131616

Hofmann, W., Schmeichel, B. J., & Baddeley, A. D. (2012). Executive functions and self-regulation. Trends in Cognitive Sciences, 16 (3), 174–180. https://doi.org/10.1016/j.tics.2012.01.006

Hutchison, R. M., & Morton, J. B. (2016). It’s a matter of time: Reframing the development of cognitive control as a modification of the brain’s temporal dynamics. Developmental Cognitive Neuroscience, 18 , 70–77. https://doi.org/10.1016/j.dcn.2015.08.006

Jennings, P. A., & Greenberg, M. T. (2009). The prosocial classroom: Teacher social and emotional competence in relation to student and classroom outcomes. Review of Educational Research , 79, 491–525. https://doi.org/10.3102/0034654308325693

Jones, S. M., Bailey, R., Barnes, S. P., & Partee, A., (2016a). Executive function mapping project executive summary: Untangling the terms and skills related to executive function and self-regulation in early childhood. OPRE Report # 2016–88. Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/sites/default/files/documents/opre/ef_mapping_snapshot_101416_final_508.pdf

Jones, S., Bailey, R., Brush, K., Nelson, B., & Barnes, S. (2016b). What is the Same and What is Different?. Making sense of the “non-cognitive” domain: Helping educators translate research into practice . Harvard University Graduate School of Education EASEL Lab. https://easel.gse.harvard.edu/files/gse-easel-lab/files/words_matter_paper.pdf

Jones, D. E., Greenberg, M., & Crowley, M. (2015). Early social-emotional functioning and public health: The relationship between kindergarten social competence and future wellness. American Journal of Public Health, 105 (11), 2283–2290. https://doi.org/10.2105/AJPH.2015.302630

Jones, S. M., McGarrah, M. W., & Kahn, J. (2019). Social and emotional learning: A principled science of human development in context. Educational Psychologist, 54 (3), 129–143. https://doi.org/10.1080/00461520.2019.1625776

Jones, S. M., Zaslow, M., Darling-Churchill, K. E., & Halle, T. G. (2016c). Assessing early childhood social and emotional development: Key conceptual and measurement issues. Journal of Applied Developmental Psychology, 45 , 42–48. https://doi.org/10.1016/j.appdev.2016.02.008

Kelley, T. L. (1927). Interpretation of educational measurements . World Book Company. pp. 62–65.

Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160 (9), 1566–1577. https://doi.org/10.1176/appi.ajp.160.9.1566

Kochanska, G., Murray, K., Jacques, T. Y., Koenig, A. L., & Vandegeest, K. A. (1996). Inhibitory control in young children and its role in emerging internalization. Child Development, 67 (2), 490–507. https://doi.org/10.1037/0012-1649.36.2.220

Leaf, P. J., Schultz, D., Keys, S., & Ialongo, N. (2002). The Teacher Observation of Classroom Adaptation-Checklist (TOCA-C) . Johns Hopkins Center for the Prevention of Youth Violence.

Lin, T. J., Justice, L. M., Paul, N., & Mashburn, A. J. (2016). Peer interaction in rural preschool classrooms: Contributions of children’s learning-related behaviors, language and literacy skills, and problem behaviors. Early Childhood Research Quarterly, 37 , 106–117.

Lonigan, C. J., Spiegel, J. A., Goodrich, J. M., Morris, B. M., Osborne, C. M., Lerner, M. D., & Phillips, B. M. (2017). Does preschool self-regulation predict later behavior problems in general or specific problem behaviors? Journal of Abnormal Child Psychology, 45 (8), 1491–1502. https://doi.org/10.1007/s10802-016-0260-7

Lutz, M. N., Fantuzzo, J. F., & McDermott, P. (2002). Contextually relevant assessment of the emotional and behavioral adjustment of low-income preschool children. Early Childhood Research Quarterly, 17 (3), 338–355. https://doi.org/10.1016/S0885-2006(02)00168-0

McClelland, M. M., Cameron, C. E., Duncan, R., Bowles, R. P., Acock, A. C., Miao, A., & Pratt, M. E. (2014). Predictors of early growth in academic achievement: The head-toes-knees-shoulders task. Frontiers in Psychology, 5 , 1–14. https://doi.org/10.3389/fpsyg.2014.00599

McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica , 22 (3), 276–282. https://hrcak.srce.hr/89395

McKown, C. (2019). Challenges and opportunities in the applied assessment of studentsocial and emotional learning. Educational Psychologist, 54 (3), 205–221. https://doi.org/10.1080/00461520.2019.1614446

McLear, C., Trentacosta, C. J., & Smith-Darden, J. (2016). Child self-regulation, parental secure base scripts, and at-risk kindergartners’ academic achievement. Early Education and Development, 27 (4), 440–456. https://doi.org/10.1080/10409289.2016.1091972

Meisels, S. J., & Atkins-Burnett, S. (1999). Social Skills Rating System field trial analysis report and recommendations . Final project report prepared for National Opinion Research Center.

Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001). Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist, 56 (2), 128–165. https://doi.org/10.1037/0003-066X.56.2.128

Moffitt, T. E., Arseneault, L., Belsky, D., Dickson, N., Hancox, R. J., Harrington, H., Houts, R., Poulton, R., Roberts, B. W., Ross, S., Sears, M. R., Thompson, W. M., & Caspi, A. (2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the National Academy of Sciences, 108 (7), 2693–2698. https://doi.org/10.1073/pnas.1010076108

Morrison, F. J., & Grammer, J. K. (2016). Conceptual clutter and measurement mayhem: Proposals for cross-disciplinary integration in conceptualizing and measuring executive function. In J. A. Griffin, P. McCardle, & L. S. Freund (Eds.), Executive function in preschool-age children: Integrating measurement, neurodevelopment, and translational research (pp. 327–348). American Psychological Association. https://doi.org/10.1037/14797-015

Nigg, J. T. (2017). Annual research review: On the relations among self-regulation, self-control, executive functioning, effortful control, cognitive control, impulsivity, risk-taking, and inhibition for developmental psychopathology. Journal of Child Psychology and Psychiatry, 58 (4), 361–383. https://doi.org/10.1111/jcpp.12675

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J.M., Akl, E.A., Brennan, S.E., Chou, R., Glanville, J., Grimshaw, J.M., Hróbjartsson, A., Lalu, M.M., Li, T., Loder, E.W., Mayo-Wilson, E., McDonald, S., ... & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372 (71). https://doi.org/10.1136/bmj.n71

Putnam, S. P., & Rothbart, M. K. (2006). Development of short and very short forms of the children’s behavior questionnaire. Journal of Personality Assessment, 87 (1), 102–112. https://doi.org/10.1207/s15327752jpa8701_09

Reynolds, C. R., & Kamphaus, R. W. (1992). Behavior assessment system for children . American Guidance Service.

Robson, D. A., Allen, M. S., & Howard, S. J. (2020). Self-regulation in childhood as a predictor of future outcomes: A meta-analytic review. Psychological Bulletin, 146 (4), 324–354. https://doi.org/10.1037/bul0000227

Rothbart, M. K., Ahadi, S. A., Hershey, K. L., & Fisher, P. (2001). Investigations of temperament at three to seven years: The children’s behavior questionnaire. Child Development, 72 (5), 1394–1408. https://doi.org/10.1111/1467-8624.00355

Smith-Donald, R., Raver, C. C., Hayes, T., & Richardson, B. (2007). Preliminary construct and concurrent validity of the Preschool Self-regulation Assessment (PSRA) for field-based research. Early Childhood Research Quarterly, 22 (2), 173–187. https://doi.org/10.1016/j.ecresq.2007.01.002

Thorndike, E. L. (1904). An Introduction to the Theory of Mental and SocialMeasurements . New York, NY: The Science Press.

Ura, S. K., & d’Abreu, A. (2022). Racial bias, social–emotional competence, and teachers’ evaluation of student behavior. Children & Schools, 44 (1), 17–26. https://doi.org/10.1093/cs/cdab028

U.S. Department of Education, Office for Civil Rights. (2018). School climate and safety, 2015–2016 civil rights data collection. https://www2.ed.gov/about/offices/list/ocr/docs/school-climate-and-safety.pdf

Valiente, C., Eisenberg, N., Haugen, R., Spinrad, T. L., Hofer, C., Liew, J., & Kupfer, A. (2011). Children’s effortful control and academic achievement: Mediation through social functioning. Early Education and Development, 22 (3), 411–433. https://doi.org/10.1080/10409289.2010.505259

Verdine, B. N., Irwin, C. M., Golinkoff, R. M., & Hirsh-Pasek, K. (2014). Contributions of executive function and spatial skills to preschool mathematics achievement. Journal of Experimental Child Psychology, 126 , 37–51. https://doi.org/10.1016/j.jecp.2014.02.012

Wang, F. L., Chassin, L., Eisenberg, N., & Spinrad, T. L. (2015). Effortful control predicts adolescent antisocial-aggressive behaviors and depressive symptoms: Co-occurrence and moderation by impulsivity. Child Development, 86 (6), 1812–1829. https://doi.org/10.1111/cdev.12406

Werthamer-Larsson, L., Kellam, S. G., & Wheeler, L. (1991). Effect of first-grade classroom environment on shy behavior, aggressive behavior, and concentration problems. American Journal of Community Psychology, 19 (4), 585–602. https://doi.org/10.1007/BF00937993

Willoughby, M., Kupersmidt, J., Voegler-Lee, M., & Bryant, D. (2011). Contributions of hot and cool self-regulation to preschool disruptive behavior and academic achievement. Developmental Neuropsychology, 36 (2), 162–180. https://doi.org/10.1080/87565641.2010.549980

Woodward, L. J., Lu, Z., Morris, A. R., & Healey, D. M. (2017). Preschool self-regulation predicts later mental health and educational achievement in very preterm and typically developing children. The Clinical Neuropsychologist, 31 (2), 404–422. https://doi.org/10.1080/13854046.2016.1251614

Zelazo, P. D. (2006). The Dimensional Change Card Sort (DCCS): A method of assessing executive function in children. Nature Protocols, 1 (1), 297–301. https://doi.org/10.1038/nprot.2006.46

Zhou, Q., Chen, S. H., & Main, A. (2012). Commonalities and differences in the research on children’s effortful control and executive function: A call for an integrated model of self-regulation. Child Development Perspectives, 6 (2), 112–121. https://doi.org/10.1111/j.1750-8606.2011.00176.x

Download references

Open access funding provided by the Carolinas Consortium. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

Department of Educational and Developmental Science, University of South Carolina, 820 Main Street, Columbia, SC, 29208, USA

Robbie A. Ross & Kate E. Ascetta

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Robbie A. Ross .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Ross, R.A., Ascetta, K.E. Methodological Overlap in Early Childhood Education Self-Regulation Research: An Interdisciplinary Systematic Review. Early Childhood Educ J (2024). https://doi.org/10.1007/s10643-024-01715-8

Download citation

Accepted : 24 June 2024

Published : 09 July 2024

DOI : https://doi.org/10.1007/s10643-024-01715-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Self-regulation
• Executive function
• Behavior problems
• Measurement
• Social-emotional learning
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 03 July 2024

The impact of evidence-based nursing leadership in healthcare settings: a mixed methods systematic review

• Maritta Välimäki 1 , 2 ,
• Shuang Hu 3 ,
• Tella Lantta 1 ,
• Kirsi Hipp 1 , 4 ,
• Jaakko Varpula 1 ,
• Jiarui Chen 3 ,
• Gaoming Liu 5 ,
• Yao Tang 3 ,
• Wenjun Chen 3 &
• Xianhong Li 3  

BMC Nursing volume  23 , Article number:  452 ( 2024 ) Cite this article

269 Accesses

Metrics details

The central component in impactful healthcare decisions is evidence. Understanding how nurse leaders use evidence in their own managerial decision making is still limited. This mixed methods systematic review aimed to examine how evidence is used to solve leadership problems and to describe the measured and perceived effects of evidence-based leadership on nurse leaders and their performance, organizational, and clinical outcomes.

We included articles using any type of research design. We referred nurses, nurse managers or other nursing staff working in a healthcare context when they attempt to influence the behavior of individuals or a group in an organization using an evidence-based approach. Seven databases were searched until 11 November 2021. JBI Critical Appraisal Checklist for Quasi-experimental studies, JBI Critical Appraisal Checklist for Case Series, Mixed Methods Appraisal Tool were used to evaluate the Risk of bias in quasi-experimental studies, case series, mixed methods studies, respectively. The JBI approach to mixed methods systematic reviews was followed, and a parallel-results convergent approach to synthesis and integration was adopted.

Thirty-one publications were eligible for the analysis: case series ( n  = 27), mixed methods studies ( n  = 3) and quasi-experimental studies ( n  = 1). All studies were included regardless of methodological quality. Leadership problems were related to the implementation of knowledge into practice, the quality of nursing care and the resource availability. Organizational data was used in 27 studies to understand leadership problems, scientific evidence from literature was sought in 26 studies, and stakeholders’ views were explored in 24 studies. Perceived and measured effects of evidence-based leadership focused on nurses’ performance, organizational outcomes, and clinical outcomes. Economic data were not available.

Conclusions

This is the first systematic review to examine how evidence is used to solve leadership problems and to describe its measured and perceived effects from different sites. Although a variety of perceptions and effects were identified on nurses’ performance as well as on organizational and clinical outcomes, available knowledge concerning evidence-based leadership is currently insufficient. Therefore, more high-quality research and clinical trial designs are still needed.

Trail registration

The study was registered (PROSPERO CRD42021259624).

Peer Review reports

Global health demands have set new roles for nurse leaders [ 1 ].Nurse leaders are referred to as nurses, nurse managers, or other nursing staff working in a healthcare context who attempt to influence the behavior of individuals or a group based on goals that are congruent with organizational goals [ 2 ]. They are seen as professionals “armed with data and evidence, and a commitment to mentorship and education”, and as a group in which “leaders innovate, transform, and achieve quality outcomes for patients, health care professionals, organizations, and communities” [ 3 ]. Effective leadership occurs when team members critically follow leaders and are motivated by a leader’s decisions based on the organization’s requests and targets [ 4 ]. On the other hand, problems caused by poor leadership may also occur, regarding staff relations, stress, sickness, or retention [ 5 ]. Therefore, leadership requires an understanding of different problems to be solved using synthesizing evidence from research, clinical expertise, and stakeholders’ preferences [ 6 , 7 ]. If based on evidence, leadership decisions, also referred as leadership decision making [ 8 ], could ensure adequate staffing [ 7 , 9 ] and to produce sufficient and cost-effective care [ 10 ]. However, nurse leaders still rely on their decision making on their personal [ 11 ] and professional experience [ 10 ] over research evidence, which can lead to deficiencies in the quality and safety of care delivery [ 12 , 13 , 14 ]. As all nurses should demonstrate leadership in their profession, their leadership competencies should be strengthened [ 15 ].

Evidence-informed decision-making, referred to as evidence appraisal and application, and evaluation of decisions [ 16 ], has been recognized as one of the core competencies for leaders [ 17 , 18 ]. The role of evidence in nurse leaders’ managerial decision making has been promoted by public authorities [ 19 , 20 , 21 ]. Evidence-based management, another concept related to evidence-based leadership, has been used as the potential to improve healthcare services [ 22 ]. It can guide nursing leaders, in developing working conditions, staff retention, implementation practices, strategic planning, patient care, and success of leadership [ 13 ]. Collins and Holton [ 23 ] in their systematic review and meta-analysis examined 83 studies regarding leadership development interventions. They found that leadership training can result in significant improvement in participants’ skills, especially in knowledge level, although the training effects varied across studies. Cummings et al. [ 24 ] reviewed 100 papers (93 studies) and concluded that participation in leadership interventions had a positive impact on the development of a variety of leadership styles. Clavijo-Chamorro et al. [ 25 ] in their review of 11 studies focused on leadership-related factors that facilitate evidence implementation: teamwork, organizational structures, and transformational leadership. The role of nurse managers was to facilitate evidence-based practices by transforming contexts to motivate the staff and move toward a shared vision of change.

As far as we are aware, however, only a few systematic reviews have focused on evidence-based leadership or related concepts in the healthcare context aiming to analyse how nurse leaders themselves uses evidence in the decision-making process. Young [ 26 ] targeted definitions and acceptance of evidence-based management (EBMgt) in healthcare while Hasanpoor et al. [ 22 ] identified facilitators and barriers, sources of evidence used, and the role of evidence in the process of decision making. Both these reviews concluded that EBMgt was of great importance but used limitedly in healthcare settings due to a lack of time, a lack of research management activities, and policy constraints. A review by Williams [ 27 ] showed that the usage of evidence to support management in decision making is marginal due to a shortage of relevant evidence. Fraser [ 28 ] in their review further indicated that the potential evidence-based knowledge is not used in decision making by leaders as effectively as it could be. Non-use of evidence occurs and leaders base their decisions mainly on single studies, real-world evidence, and experts’ opinions [ 29 ]. Systematic reviews and meta-analyses rarely provide evidence of management-related interventions [ 30 ]. Tate et al. [ 31 ] concluded based on their systematic review and meta-analysis that the ability of nurse leaders to use and critically appraise research evidence may influence the way policy is enacted and how resources and staff are used to meet certain objectives set by policy. This can further influence staff and workforce outcomes. It is therefore important that nurse leaders have the capacity and motivation to use the strongest evidence available to effect change and guide their decision making [ 27 ].

Despite of a growing body of evidence, we found only one review focusing on the impact of evidence-based knowledge. Geert et al. [ 32 ] reviewed literature from 2007 to 2016 to understand the elements of design, delivery, and evaluation of leadership development interventions that are the most reliably linked to outcomes at the level of the individual and the organization, and that are of most benefit to patients. The authors concluded that it is possible to improve individual-level outcomes among leaders, such as knowledge, motivation, skills, and behavior change using evidence-based approaches. Some of the most effective interventions included, for example, interactive workshops, coaching, action learning, and mentoring. However, these authors found limited research evidence describing how nurse leaders themselves use evidence to support their managerial decisions in nursing and what the outcomes are.

To fill the knowledge gap and compliment to existing knowledgebase, in this mixed methods review we aimed to (1) examine what leadership problems nurse leaders solve using an evidence-based approach and (2) how they use evidence to solve these problems. We also explored (3) the measured and (4) perceived effects of the evidence-based leadership approach in healthcare settings. Both qualitative and quantitative components of the effects of evidence-based leadership were examined to provide greater insights into the available literature [ 33 ]. Together with the evidence-based leadership approach, and its impact on nursing [ 34 , 35 ], this knowledge gained in this review can be used to inform clinical policy or organizational decisions [ 33 ]. The study is registered (PROSPERO CRD42021259624). The methods used in this review were specified in advance and documented in a priori in a published protocol [ 36 ]. Key terms of the review and the search terms are defined in Table  1 (population, intervention, comparison, outcomes, context, other).

In this review, we used a mixed methods approach [ 37 ]. A mixed methods systematic review was selected as this approach has the potential to produce direct relevance to policy makers and practitioners [ 38 ]. Johnson and Onwuegbuzie [ 39 ] have defined mixed methods research as “the class of research in which the researcher mixes or combines quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study.” Therefore, we combined quantitative and narrative analysis to appraise and synthesize empirical evidence, and we held them as equally important in informing clinical policy or organizational decisions [ 34 ]. In this review, a comprehensive synthesis of quantitative and qualitative data was performed first and then discussed in discussion part (parallel-results convergent design) [ 40 ]. We hoped that different type of analysis approaches could complement each other and deeper picture of the topic in line with our research questions could be gained [ 34 ].

Inclusion and exclusion criteria

Inclusion and exclusion criteria of the study are described in Table  1 .

Search strategy

A three-step search strategy was utilized. First, an initial limited search with #MEDLINE was undertaken, followed by analysis of the words used in the title, abstract, and the article’s key index terms. Second, the search strategy, including identified keywords and index terms, was adapted for each included data base and a second search was undertaken on 11 November 2021. The full search strategy for each database is described in Additional file 1 . Third, the reference list of all studies included in the review were screened for additional studies. No year limits or language restrictions were used.

Information sources

The database search included the following: CINAHL (EBSCO), Cochrane Library (academic database for medicine and health science and nursing), Embase (Elsevier), PsycINFO (EBSCO), PubMed (MEDLINE), Scopus (Elsevier) and Web of Science (academic database across all scientific and technical disciplines, ranging from medicine and social sciences to arts and humanities). These databases were selected as they represent typical databases in health care context. Subject headings from each of the databases were included in the search strategies. Boolean operators ‘AND’ and ‘OR’ were used to combine the search terms. An information specialist from the University of Turku Library was consulted in the formation of the search strategies.

Study selection

All identified citations were collated and uploaded into Covidence software (Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia www.covidence.org ), and duplicates were removed by the software. Titles and abstracts were screened and assessed against the inclusion criteria independently by two reviewers out of four, and any discrepancies were resolved by the third reviewer (MV, KH, TL, WC). Studies meeting the inclusion criteria were retrieved in full and archived in Covidence. Access to one full-text article was lacking: the authors for one study were contacted about the missing full text, but no full text was received. All remaining hits of the included studies were retrieved and assessed independently against the inclusion criteria by two independent reviewers of four (MV, KH, TL, WC). Studies that did not meet the inclusion criteria were excluded, and the reasons for exclusion were recorded in Covidence. Any disagreements that arose between the reviewers were resolved through discussions with XL.

Assessment of methodological quality

Eligible studies were critically appraised by two independent reviewers (YT, SH). Standardized critical appraisal instruments based on the study design were used. First, quasi-experimental studies were assessed using the JBI Critical Appraisal Checklist for Quasi-experimental studies [ 44 ]. Second, case series were assessed using the JBI Critical Appraisal Checklist for Case Series [ 45 ]. Third, mixed methods studies were appraised using the Mixed Methods Appraisal Tool [ 46 ].

To increase inter-reviewer reliability, the review agreement was calculated (SH) [ 47 ]. A kappa greater than 0.8 was considered to represent a high level of agreement (0–0.1). In our data, the agreement was 0.75. Discrepancies raised between two reviewers were resolved through discussion and modifications and confirmed by XL. As an outcome, studies that met the inclusion criteria were proceeded to critical appraisal and assessed as suitable for inclusion in the review. The scores for each item and overall critical appraisal scores were presented.

Data extraction

For data extraction, specific tables were created. First, study characteristics (author(s), year, country, design, number of participants, setting) were extracted by two authors independently (JC, MV) and reviewed by TL. Second, descriptions of the interventions were extracted by two reviewers (JV, JC) using the structure of the TIDIeR (Template for Intervention Description and Replication) checklist (brief name, the goal of the intervention, material and procedure, models of delivery and location, dose, modification, adherence and fidelity) [ 48 ]. The extractions were confirmed (MV).

Third, due to a lack of effectiveness data and a wide heterogeneity between study designs and presentation of outcomes, no attempt was made to pool the quantitative data statistically; the findings of the quantitative data were presented in narrative form only [ 44 ]. The separate data extraction tables for each research question were designed specifically for this study. For both qualitative (and a qualitative component of mixed-method studies) and quantitative studies, the data were extracted and tabulated into text format according to preplanned research questions [ 36 ]. To test the quality of the tables and the data extraction process, three authors independently extracted the data from the first five studies (in alphabetical order). After that, the authors came together to share and determine whether their approaches of the data extraction were consistent with each other’s output and whether the content of each table was in line with research question. No reason was found to modify the data extraction tables or planned process. After a consensus of the data extraction process was reached, the data were extracted in pairs by independent reviewers (WC, TY, SH, GL). Any disagreements that arose between the reviewers were resolved through discussion and with a third reviewer (MV).

Data analysis

We were not able to conduct a meta-analysis due to a lack of effectiveness data based on clinical trials. Instead, we used inductive thematic analysis with constant comparison to answer the research question [ 46 , 49 ] using tabulated primary data from qualitative and quantitative studies as reported by the original authors in narrative form only [ 47 ]. In addition, the qualitizing process was used to transform quantitative data to qualitative data; this helped us to convert the whole data into themes and categories. After that we used the thematic analysis for the narrative data as follows. First, the text was carefully read, line by line, to reveal topics answering each specific review question (MV). Second, the data coding was conducted, and the themes in the data were formed by data categorization. The process of deriving the themes was inductive based on constant comparison [ 49 ]. The results of thematic analysis and data categorization was first described in narrative format and then the total number of studies was calculated where the specific category was identified (%).

Stakeholder involvement

The method of reporting stakeholders’ involvement follows the key components by [ 50 ]: (1) people involved, (2) geographical location, (3) how people were recruited, (4) format of involvement, (5) amount of involvement, (6) ethical approval, (7) financial compensation, and (8) methods for reporting involvement.

In our review, stakeholder involvement targeted nurses and nurse leader in China. Nurse Directors of two hospitals recommended potential participants who received a personal invitation letter from researchers to participate in a discussion meeting. Stakeholders’ participation was based on their own free will. Due to COVID-19, one online meeting (1 h) was organized (25 May 2022). Eleven participants joined the meeting. Ethical approval was not applied and no financial compensation was offered. At the end of the meeting, experiences of stakeholders’ involvement were explored.

The meeting started with an introductory presentation with power points. The rationale, methods, and preliminary review results were shared with the participants [ 51 ].The meeting continued with general questions for the participants: (1) Are you aware of the concepts of evidence-based practice or evidence-based leadership?; (2) How important is it to use evidence to support decisions among nurse leaders?; (3) How is the evidence-based approach used in hospital settings?; and (4) What type of evidence is currently used to support nurse leaders’ decision making (e.g. scientific literature, organizational data, stakeholder views)?

Two people took notes on the course and content of the conversation. The notes were later transcripted in verbatim, and the key points of the discussions were summarised. Although answers offered by the stakeholders were very short, the information was useful to validate the preliminary content of the results, add the rigorousness of the review, and obtain additional perspectives. A recommendation of the stakeholders was combined in the Discussion part of this review increasing the applicability of the review in the real world [ 50 ]. At the end of the discussion, the value of stakeholders’ involvement was asked. Participants shared that the experience of participating was unique and the topic of discussion was challenging. Two authors of the review group further represented stakeholders by working together with the research team throughout the review study.

Search results

From seven different electronic databases, 6053 citations were identified as being potentially relevant to the review. Then, 3133 duplicates were removed by an automation tool (Covidence: www.covidence.org ), and one was removed manually. The titles and abstracts of 3040 of citations were reviewed, and a total of 110 full texts were included (one extra citation was found on the reference list but later excluded). Based on the eligibility criteria, 31 studies (32 hits) were critically appraised and deemed suitable for inclusion in the review. The search results and selection process are presented in the PRISMA [ 52 ] flow diagram Fig.  1 . The full list of references for included studies can be find in Additional file 2 . To avoid confusion between articles of the reference list and studies included in the analysis, the studies included in the review are referred inside the article using the reference number of each study (e.g. ref 1, ref 2).

Search results and study selection and inclusion process [ 52 ]

Characteristics of included studies

The studies had multiple purposes, aiming to develop practice, implement a new approach, improve quality, or to develop a model. The 31 studies (across 32 hits) were case series studies ( n  = 27), mixed methods studies ( n  = 3) and a quasi-experimental study ( n  = 1). All studies were published between the years 2004 and 2021. The highest number of papers was published in year 2020.

Table  2 describes the characteristics of included studies and Additional file 3 offers a narrative description of the studies.

Methodological quality assessment

Quasi-experimental studies.

We had one quasi-experimental study (ref 31). All questions in the critical appraisal tool were applicable. The total score of the study was 8 (out of a possible 9). Only one response of the tool was ‘no’ because no control group was used in the study (see Additional file 4 for the critical appraisal of included studies).

Case series studies . A case series study is typically defined as a collection of subjects with common characteristics. The studies do not include a comparison group and are often based on prevalent cases and on a sample of convenience [ 53 ]. Munn et al. [ 45 ] further claim that case series are best described as observational studies, lacking experimental and randomized characteristics, being descriptive studies, without a control or comparator group. Out of 27 case series studies included in our review, the critical appraisal scores varied from 1 to 9. Five references were conference abstracts with empirical study results, which were scored from 1 to 3. Full reports of these studies were searched in electronic databases but not found. Critical appraisal scores for the remaining 22 studies ranged from 1 to 9 out of a possible score of 10. One question (Q3) was not applicable to 13 studies: “Were valid methods used for identification of the condition for all participants included in the case series?” Only two studies had clearly reported the demographic of the participants in the study (Q6). Twenty studies met Criteria 8 (“Were the outcomes or follow-up results of cases clearly reported?”) and 18 studies met Criteria 7 (“Q7: Was there clear reporting of clinical information of the participants?”) (see Additional file 4 for the critical appraisal of included studies).

Mixed-methods studies

Mixed-methods studies involve a combination of qualitative and quantitative methods. This is a common design and includes convergent design, sequential explanatory design, and sequential exploratory design [ 46 ]. There were three mixed-methods studies. The critical appraisal scores for the three studies ranged from 60 to 100% out of a possible 100%. Two studies met all the criteria, while one study fulfilled 60% of the scored criteria due to a lack of information to understand the relevance of the sampling strategy well enough to address the research question (Q4.1) or to determine whether the risk of nonresponse bias was low (Q4.4) (see Additional file 4 for the critical appraisal of included studies).

Intervention or program components

The intervention of program components were categorized and described using the TiDier checklist: name and goal, theory or background, material, procedure, provider, models of delivery, location, dose, modification, and adherence and fidelity [ 48 ]. A description of intervention in each study is described in Additional file 5 and a narrative description in Additional file 6 .

Leadership problems

In line with the inclusion criteria, data for the leadership problems were categorized in all 31 included studies (see Additional file 7 for leadership problems). Three types of leadership problems were identified: implementation of knowledge into practice, the quality of clinical care, and resources in nursing care. A narrative summary of the results is reported below.

Implementing knowledge into practice

Eleven studies (35%) aimed to solve leadership problems related to implementation of knowledge into practice. Studies showed how to support nurses in evidence-based implementation (EBP) (ref 3, ref 5), how to engage nurses in using evidence in practice (ref 4), how to convey the importance of EBP (ref 22) or how to change practice (ref 4). Other problems were how to facilitate nurses to use guideline recommendations (ref 7) and how nurses can make evidence-informed decisions (ref 8). General concerns also included the linkage between theory and practice (ref 1) as well as how to implement the EBP model in practice (ref 6). In addition, studies were motivated by the need for revisions or updates of protocols to improve clinical practice (ref 10) as well as the need to standardize nursing activities (ref 11, ref 14).

The quality of the care

Thirteen (42%) focused on solving problems related to the quality of clinical care. In these studies, a high number of catheter infections led a lack of achievement of organizational goals (ref 2, ref 9). A need to reduce patient symptoms in stem cell transplant patients undergoing high-dose chemotherapy (ref 24) was also one of the problems to be solved. In addition, the projects focused on how to prevent pressure ulcers (ref 26, ref 29), how to enhance the quality of cancer treatment (ref 25) and how to reduce the need for invasive constipation treatment (ref 30). Concerns about patient safety (ref 15), high fall rates (ref 16, ref 19), dissatisfaction of patients (ref 16, ref 18) and nurses (ref 16, ref 30) were also problems that had initiated the projects. Studies addressed concerns about how to promote good contingency care in residential aged care homes (ref 20) and about how to increase recognition of human trafficking problems in healthcare (ref 21).

Resources in nursing care

Nurse leaders identified problems in their resources, especially in staffing problems. These problems were identified in seven studies (23%), which involved concerns about how to prevent nurses from leaving the job (ref 31), how to ensure appropriate recruitment, staffing and retaining of nurses (ref 13) and how to decrease nurses’ burden and time spent on nursing activities (ref 12). Leadership turnover was also reported as a source of dissatisfaction (ref 17); studies addressed a lack of structured transition and training programs, which led to turnover (ref 23), as well as how to improve intershift handoff among nurses (ref 28). Optimal design for new hospitals was also examined (ref 27).

Main features of evidence-based leadership

Out of 31 studies, 17 (55%) included all four domains of an evidence-based leadership approach, and four studies (13%) included evidence of critical appraisal of the results (see Additional file 8 for the main features of evidence-based Leadership) (ref 11, ref 14, ref 23, ref 27).

Organizational evidence

Twenty-seven studies (87%) reported how organizational evidence was collected and used to solve leadership problems (ref 2). Retrospective chart reviews (ref 5), a review of the extent of specific incidents (ref 19), and chart auditing (ref 7, ref 25) were conducted. A gap between guideline recommendations and actual care was identified using organizational data (ref 7) while the percentage of nurses’ working time spent on patient care was analyzed using an electronic charting system (ref 12). Internal data (ref 22), institutional data, and programming metrics were also analyzed to understand the development of the nurse workforce (ref 13).

Surveys (ref 3, ref 25), interviews (ref 3, ref 25) and group reviews (ref 18) were used to better understand the leadership problem to be solved. Employee opinion surveys on leadership (ref 17), a nurse satisfaction survey (ref 30) and a variety of reporting templates were used for the data collection (ref 28) reported. Sometimes, leadership problems were identified by evidence facilitators or a PI’s team who worked with staff members (ref 15, ref 17). Problems in clinical practice were also identified by the Nursing Professional Council (ref 14), managers (ref 26) or nurses themselves (ref 24). Current practices were reviewed (ref 29) and a gap analysis was conducted (ref 4, ref 16, ref 23) together with SWOT analysis (ref 16). In addition, hospital mission and vision statements, research culture established and the proportion of nursing alumni with formal EBP training were analyzed (ref 5). On the other hand, it was stated that no systematic hospital-specific sources of data regarding job satisfaction or organizational commitment were used (ref 31). In addition, statements of organizational analysis were used on a general level only (ref 1).

Scientific evidence identified

Twenty-six studies (84%) reported the use of scientific evidence in their evidence-based leadership processes. A literature search was conducted (ref 21) and questions, PICO, and keywords were identified (ref 4) in collaboration with a librarian. Electronic databases, including PubMed (ref 14, ref 31), Cochrane, and EMBASE (ref 31) were searched. Galiano (ref 6) used Wiley Online Library, Elsevier, CINAHL, Health Source: Nursing/Academic Edition, PubMed, and the Cochrane Library while Hoke (ref 11) conducted an electronic search using CINAHL and PubMed to retrieve articles.

Identified journals were reviewed manually (ref 31). The findings were summarized using ‘elevator speech’ (ref 4). In a study by Gifford et al. (ref 9) evidence facilitators worked with participants to access, appraise, and adapt the research evidence to the organizational context. Ostaszkiewicz (ref 20) conducted a scoping review of literature and identified and reviewed frameworks and policy documents about the topic and the quality standards. Further, a team of nursing administrators, directors, staff nurses, and a patient representative reviewed the literature and made recommendations for practice changes.

Clinical practice guidelines were also used to offer scientific evidence (ref 7, ref 19). Evidence was further retrieved from a combination of nursing policies, guidelines, journal articles, and textbooks (ref 12) as well as from published guidelines and literature (ref 13). Internal evidence, professional practice knowledge, relevant theories and models were synthesized (ref 24) while other study (ref 25) reviewed individual studies, synthesized with systematic reviews or clinical practice guidelines. The team reviewed the research evidence (ref 3, ref 15) or conducted a literature review (ref 22, ref 28, ref 29), a literature search (ref 27), a systematic review (ref 23), a review of the literature (ref 30) or ‘the scholarly literature was reviewed’ (ref 18). In addition, ‘an extensive literature review of evidence-based best practices was carried out’ (ref 10). However, detailed description how the review was conducted was lacking.

Views of stakeholders

A total of 24 studies (77%) reported methods for how the views of stakeholders, i.e., professionals or experts, were considered. Support to run this study was received from nursing leadership and multidisciplinary teams (ref 29). Experts and stakeholders joined the study team in some cases (ref 25, ref 30), and in other studies, their opinions were sought to facilitate project success (ref 3). Sometimes a steering committee was formed by a Chief Nursing Officer and Clinical Practice Specialists (ref 2). More specifically, stakeholders’ views were considered using interviews, workshops and follow-up teleconferences (ref 7). The literature review was discussed with colleagues (ref 11), and feedback and support from physicians as well as the consensus of staff were sought (ref 16).

A summary of the project findings and suggestions for the studies were discussed at 90-minute weekly meetings by 11 charge nurses. Nurse executive directors were consulted over a 10-week period (ref 31). An implementation team (nurse, dietician, physiotherapist, occupational therapist) was formed to support the implementation of evidence-based prevention measures (ref 26). Stakeholders volunteered to join in the pilot implementation (ref 28) or a stakeholder team met to determine the best strategy for change management, shortcomings in evidence-based criteria were discussed, and strategies to address those areas were planned (ref 5). Nursing leaders, staff members (ref 22), ‘process owners (ref 18) and program team members (ref 18, ref 19, ref 24) met regularly to discuss the problems. Critical input was sought from clinical educators, physicians, nutritionists, pharmacists, and nurse managers (ref 24). The unit director and senior nursing staff reviewed the contents of the product, and the final version of clinical pathways were reviewed and approved by the Quality Control Commission of the Nursing Department (ref 12). In addition, two co-design workshops with 18 residential aged care stakeholders were organized to explore their perspectives about factors to include in a model prototype (ref 20). Further, an agreement of stakeholders in implementing continuous quality services within an open relationship was conducted (ref 1).

Critical appraisal

In five studies (16%), a critical appraisal targeting the literature search was carried out. The appraisals were conducted by interns and teams who critiqued the evidence (ref 4). In Hoke’s study, four areas that had emerged in the literature were critically reviewed (ref 11). Other methods were to ‘critically appraise the search results’ (ref 14). Journal club team meetings (ref 23) were organized to grade the level and quality of evidence and the team ‘critically appraised relevant evidence’ (ref 27). On the other hand, the studies lacked details of how the appraisals were done in each study.

The perceived effects of evidence-based leadership

Perceived effects of evidence-based leadership on nurses’ performance.

Eleven studies (35%) described perceived effects of evidence-based leadership on nurses’ performance (see Additional file 9 for perceived effects of evidence-based leadership), which were categorized in four groups: awareness and knowledge, competence, ability to understand patients’ needs, and engagement. First, regarding ‘awareness and knowledge’, different projects provided nurses with new learning opportunities (ref 3). Staff’s knowledge (ref 20, ref 28), skills, and education levels improved (ref 20), as did nurses’ knowledge comprehension (ref 21). Second, interventions and approaches focusing on management and leadership positively influenced participants’ competence level to improve the quality of services. Their confidence level (ref 1) and motivation to change practice increased, self-esteem improved, and they were more positive and enthusiastic in their work (ref 22). Third, some nurses were relieved that they had learned to better handle patients’ needs (ref 25). For example, a systematic work approach increased nurses’ awareness of the patients who were at risk of developing health problems (ref 26). And last, nurse leaders were more engaged with staff, encouraging them to adopt the new practices and recognizing their efforts to change (ref 8).

Perceived effects on organizational outcomes

Nine studies (29%) described the perceived effects of evidence-based leadership on organizational outcomes (see Additional file 9 for perceived effects of evidence-based leadership). These were categorized into three groups: use of resources, staff commitment, and team effort. First, more appropriate use of resources was reported (ref 15, ref 20), and working time was more efficiently used (ref 16). In generally, a structured approach made implementing change more manageable (ref 1). On the other hand, in the beginning of the change process, the feedback from nurses was unfavorable, and they experienced discomfort in the new work style (ref 29). New approaches were also perceived as time consuming (ref 3). Second, nurse leaders believed that fewer nursing staff than expected left the organization over the course of the study (ref 31). Third, the project helped staff in their efforts to make changes, and it validated the importance of working as a team (ref 7). Collaboration and support between the nurses increased (ref 26). On the other hand, new work style caused challenges in teamwork (ref 3).

Perceived effects on clinical outcomes

Five studies (16%) reported the perceived effects of evidence-based leadership on clinical outcomes (see Additional file 9 for perceived effects of evidence-based leadership), which were categorized in two groups: general patient outcomes and specific clinical outcomes. First, in general, the project assisted in connecting the guideline recommendations and patient outcomes (ref 7). The project was good for the patients in general, and especially to improve patient safety (ref 16). On the other hand, some nurses thought that the new working style did not work at all for patients (ref 28). Second, the new approach used assisted in optimizing patients’ clinical problems and person-centered care (ref 20). Bowel management, for example, received very good feedback (ref 30).

The measured effects of evidence-based leadership

The measured effects on nurses’ performance.

Data were obtained from 20 studies (65%) (see Additional file 10 for measured effects of evidence-based leadership) and categorized nurse performance outcomes for three groups: awareness and knowledge, engagement, and satisfaction. First, six studies (19%) measured the awareness and knowledge levels of participants. Internship for staff nurses was beneficial to help participants to understand the process for using evidence-based practice and to grow professionally, to stimulate for innovative thinking, to give knowledge needed to use evidence-based practice to answer clinical questions, and to make possible to complete an evidence-based practice project (ref 3). Regarding implementation program of evidence-based practice, those with formal EBP training showed an improvement in knowledge, attitude, confidence, awareness and application after intervention (ref 3, ref 11, ref 20, ref 23, ref 25). On the contrary, in other study, attitude towards EBP remained stable ( p  = 0.543). and those who applied EBP decreased although no significant differences over the years ( p  = 0.879) (ref 6).

Second, 10 studies (35%) described nurses’ engagement to new practices (ref 5, ref 6, ref 7, ref 10, ref 16, ref 17, ref 18, ref 21, ref 25, ref 27). 9 studies (29%) studies reported that there was an improvement of compliance level of participants (ref 6, ref 7, ref 10, ref 16, ref 17, ref 18, ref 21, ref 25, ref 27). On the contrary, in DeLeskey’s (ref 5) study, although improvement was found in post-operative nausea and vomiting’s (PONV) risk factors documented’ (2.5–63%), and ’risk factors communicated among anaesthesia and surgical staff’ (0–62%), the improvement did not achieve the goal. The reason was a limited improvement was analysed. It was noted that only those patients who had been seen by the pre-admission testing nurse had risk assessments completed. Appropriate treatment/prophylaxis increased from 69 to 77%, and from 30 to 49%; routine assessment for PONV/rescue treatment 97% and 100% was both at 100% following the project. The results were discussed with staff but further reasons for a lack of engagement in nursing care was not reported.

And third, six studies (19%) reported nurses’ satisfaction with project outcomes. The study results showed that using evidence in managerial decisions improved nurses’ satisfaction and attitudes toward their organization ( P  < 0.05) (ref 31). Nurses’ overall job satisfaction improved as well (ref 17). Nurses’ satisfaction with usability of the electronic charting system significantly improved after introduction of the intervention (ref 12). In handoff project in seven hospitals, improvement was reported in all satisfaction indicators used in the study although improvement level varied in different units (ref 28). In addition, positive changes were reported in nurses’ ability to autonomously perform their job (“How satisfied are you with the tools and resources available for you treat and prevent patient constipation?” (54%, n  = 17 vs. 92%, n  = 35, p  < 0.001) (ref 30).

The measured effects on organizational outcomes

Thirteen studies (42%) described the effects of a project on organizational outcomes (see Additional file 10 for measured effects of evidence-based leadership), which were categorized in two groups: staff compliance, and changes in practices. First, studies reported improved organizational outcomes due to staff better compliance in care (ref 4, ref 13, ref 17, ref 23, ref 27, ref 31). Second, changes in organization practices were also described (ref 11) like changes in patient documentation (ref 12, ref 21). Van Orne (ref 30) found a statistically significant reduction in the average rate of invasive medication administration between pre-intervention and post-intervention ( p  = 0.01). Salvador (ref 24) also reported an improvement in a proactive approach to mucositis prevention with an evidence-based oral care guide. On the contrary, concerns were also raised such as not enough time for new bedside report (ref 16) or a lack of improvement of assessment of diabetic ulcer (ref 8).

The measured effects on clinical outcomes

A variety of improvements in clinical outcomes were reported (see Additional file 10 for measured effects of evidence-based leadership): improvement in patient clinical status and satisfaction level. First, a variety of improvement in patient clinical status was reported. improvement in Incidence of CAUTI decreased 27.8% between 2015 and 2019 (ref 2) while a patient-centered quality improvement project reduced CAUTI rates to 0 (ref 10). A significant decrease in transmission rate of MRSA transmission was also reported (ref 27) and in other study incidences of CLABSIs dropped following of CHG bathing (ref 14). Further, it was possible to decrease patient nausea from 18 to 5% and vomiting to 0% (ref 5) while the percentage of patients who left the hospital without being seen was below 2% after the project (ref 17). In addition, a significant reduction in the prevalence of pressure ulcers was found (ref 26, ref 29) and a significant reduction of mucositis severity/distress was achieved (ref 24). Patient falls rate decreased (ref 15, ref 16, ref 19, ref 27).

Second, patient satisfaction level after project implementation improved (ref 28). The scale assessing healthcare providers by consumers showed improvement, but the changes were not statistically significant. Improvement in an emergency department leadership model and in methods of communication with patients improved patient satisfaction scores by 600% (ref 17). In addition, new evidence-based unit improved patient experiences about the unit although not all items improved significantly (ref 18).

Stakeholder involvement in the mixed-method review

To ensure stakeholders’ involvement in the review, the real-world relevance of our research [ 53 ], achieve a higher level of meaning in our review results, and gain new perspectives on our preliminary findings [ 50 ], a meeting with 11 stakeholders was organized. First, we asked if participants were aware of the concepts of evidence-based practice or evidence-based leadership. Responses revealed that participants were familiar with the concept of evidence-based practice, but the topic of evidence-based leadership was totally new. Examples of nurses and nurse leaders’ responses are as follows: “I have heard a concept of evidence-based practice but never a concept of evidence-based leadership.” Another participant described: “I have heard it [evidence-based leadership] but I do not understand what it means.”

Second, as stakeholder involvement is beneficial to the relevance and impact of health research [ 54 ], we asked how important evidence is to them in supporting decisions in health care services. One participant described as follows: “Using evidence in decisions is crucial to the wards and also to the entire hospital.” Third, we asked how the evidence-based approach is used in hospital settings. Participants expressed that literature is commonly used to solve clinical problems in patient care but not to solve leadership problems. “In [patient] medication and care, clinical guidelines are regularly used. However, I am aware only a few cases where evidence has been sought to solve leadership problems.”

And last, we asked what type of evidence is currently used to support nurse leaders’ decision making (e.g. scientific literature, organizational data, stakeholder views)? The participants were aware that different types of information were collected in their organization on a daily basis (e.g. patient satisfaction surveys). However, the information was seldom used to support decision making because nurse leaders did not know how to access this information. Even so, the participants agreed that the use of evidence from different sources was important in approaching any leadership or managerial problems in the organization. Participants also suggested that all nurse leaders should receive systematic training related to the topic; this could support the daily use of the evidence-based approach.

To our knowledge, this article represents the first mixed-methods systematic review to examine leadership problems, how evidence is used to solve these problems and what the perceived and measured effects of evidence-based leadership are on nurse leaders and their performance, organizational, and clinical outcomes. This review has two key findings. First, the available research data suggests that evidence-based leadership has potential in the healthcare context, not only to improve knowledge and skills among nurses, but also to improve organizational outcomes and the quality of patient care. Second, remarkably little published research was found to explore the effects of evidence-based leadership with an efficient trial design. We validated the preliminary results with nurse stakeholders, and confirmed that nursing staff, especially nurse leaders, were not familiar with the concept of evidence-based leadership, nor were they used to implementing evidence into their leadership decisions. Our data was based on many databases, and we screened a large number of studies. We also checked existing registers and databases and found no registered or ongoing similar reviews being conducted. Therefore, our results may not change in the near future.

We found that after identifying the leadership problems, 26 (84%) studies out of 31 used organizational data, 25 (81%) studies used scientific evidence from the literature, and 21 (68%) studies considered the views of stakeholders in attempting to understand specific leadership problems more deeply. However, only four studies critically appraised any of these findings. Considering previous critical statements of nurse leaders’ use of evidence in their decision making [ 14 , 30 , 31 , 34 , 55 ], our results are still quite promising.

Our results support a previous systematic review by Geert et al. [ 32 ], which concluded that it is possible to improve leaders’ individual-level outcomes, such as knowledge, motivation, skills, and behavior change using evidence-based approaches. Collins and Holton [ 23 ] particularly found that leadership training resulted in significant knowledge and skill improvements, although the effects varied widely across studies. In our study, evidence-based leadership was seen to enable changes in clinical practice, especially in patient care. On the other hand, we understand that not all efforts to changes were successful [ 56 , 57 , 58 ]. An evidence-based approach causes negative attitudes and feelings. Negative emotions in participants have also been reported due to changes, such as discomfort with a new working style [ 59 ]. Another study reported inconvenience in using a new intervention and its potential risks for patient confidentiality. Sometimes making changes is more time consuming than continuing with current practice [ 60 ]. These findings may partially explain why new interventions or program do not always fully achieve their goals. On the other hand, Dubose et al. [ 61 ] state that, if prepared with knowledge of resistance, nurse leaders could minimize the potential negative consequences and capitalize on a powerful impact of change adaptation.

We found that only six studies used a specific model or theory to understand the mechanism of change that could guide leadership practices. Participants’ reactions to new approaches may be an important factor in predicting how a new intervention will be implemented into clinical practice. Therefore, stronger effort should be put to better understanding the use of evidence, how participants’ reactions and emotions or practice changes could be predicted or supported using appropriate models or theories, and how using these models are linked with leadership outcomes. In this task, nurse leaders have an important role. At the same time, more responsibilities in developing health services have been put on the shoulders of nurse leaders who may already be suffering under pressure and increased burden at work. Working in a leadership position may also lead to role conflict. A study by Lalleman et al. [ 62 ] found that nurses were used to helping other people, often in ad hoc situations. The helping attitude of nurses combined with structured managerial role may cause dilemmas, which may lead to stress. Many nurse leaders opt to leave their positions less than 5 years [ 63 ].To better fulfill the requirements of health services in the future, the role of nurse leaders in evidence-based leadership needs to be developed further to avoid ethical and practical dilemmas in their leadership practices.

It is worth noting that the perceived and measured effects did not offer strong support to each other but rather opened a new venue to understand the evidence-based leadership. Specifically, the perceived effects did not support to measured effects (competence, ability to understand patients’ needs, use of resources, team effort, and specific clinical outcomes) while the measured effects could not support to perceived effects (nurse’s performance satisfaction, changes in practices, and clinical outcomes satisfaction). These findings may indicate that different outcomes appear if the effects of evidence-based leadership are looked at using different methodological approach. Future study is encouraged using well-designed study method including mixed-method study to examine the consistency between perceived and measured effects of evidence-based leadership in health care.

There is a potential in nursing to support change by demonstrating conceptual and operational commitment to research-based practices [ 64 ]. Nurse leaders are well positioned to influence and lead professional governance, quality improvement, service transformation, change and shared governance [ 65 ]. In this task, evidence-based leadership could be a key in solving deficiencies in the quality, safety of care [ 14 ] and inefficiencies in healthcare delivery [ 12 , 13 ]. As WHO has revealed, there are about 28 million nurses worldwide, and the demand of nurses will put nurse resources into the specific spotlight [ 1 ]. Indeed, evidence could be used to find solutions for how to solve economic deficits or other problems using leadership skills. This is important as, when nurses are able to show leadership and control in their own work, they are less likely to leave their jobs [ 66 ]. On the other hand, based on our discussions with stakeholders, nurse leaders are not used to using evidence in their own work. Further, evidence-based leadership is not possible if nurse leaders do not have access to a relevant, robust body of evidence, adequate funding, resources, and organizational support, and evidence-informed decision making may only offer short-term solutions [ 55 ]. We still believe that implementing evidence-based strategies into the work of nurse leaders may create opportunities to protect this critical workforce from burnout or leaving the field [ 67 ]. However, the role of the evidence-based approach for nurse leaders in solving these problems is still a key question.

Limitations

This study aimed to use a broad search strategy to ensure a comprehensive review but, nevertheless, limitations exist: we may have missed studies not included in the major international databases. To keep search results manageable, we did not use specific databases to systematically search grey literature although it is a rich source of evidence used in systematic reviews and meta-analysis [ 68 ]. We still included published conference abstract/proceedings, which appeared in our scientific databases. It has been stated that conference abstracts and proceedings with empirical study results make up a great part of studies cited in systematic reviews [ 69 ]. At the same time, a limited space reserved for published conference publications can lead to methodological issues reducing the validity of the review results [ 68 ]. We also found that the great number of studies were carried out in western countries, restricting the generalizability of the results outside of English language countries. The study interventions and outcomes were too different across studies to be meaningfully pooled using statistical methods. Thus, our narrative synthesis could hypothetically be biased. To increase transparency of the data and all decisions made, the data, its categorization and conclusions are based on original studies and presented in separate tables and can be found in Additional files. Regarding a methodological approach [ 34 ], we used a mixed methods systematic review, with the core intention of combining quantitative and qualitative data from primary studies. The aim was to create a breadth and depth of understanding that could confirm to or dispute evidence and ultimately answer the review question posed [ 34 , 70 ]. Although the method is gaining traction due to its usefulness and practicality, guidance in combining quantitative and qualitative data in mixed methods systematic reviews is still limited at the theoretical stage [ 40 ]. As an outcome, it could be argued that other methodologies, for example, an integrative review, could have been used in our review to combine diverse methodologies [ 71 ]. We still believe that the results of this mixed method review may have an added value when compared with previous systematic reviews concerning leadership and an evidence-based approach.

Our mixed methods review fills the gap regarding how nurse leaders themselves use evidence to guide their leadership role and what the measured and perceived impact of evidence-based leadership is in nursing. Although the scarcity of controlled studies on this topic is concerning, the available research data suggest that evidence-based leadership intervention can improve nurse performance, organizational outcomes, and patient outcomes. Leadership problems are also well recognized in healthcare settings. More knowledge and a deeper understanding of the role of nurse leaders, and how they can use evidence in their own managerial leadership decisions, is still needed. Despite the limited number of studies, we assume that this narrative synthesis can provide a good foundation for how to develop evidence-based leadership in the future.

Implications

Based on our review results, several implications can be recommended. First, the future of nursing success depends on knowledgeable, capable, and strong leaders. Therefore, nurse leaders worldwide need to be educated about the best ways to manage challenging situations in healthcare contexts using an evidence-based approach in their decisions. This recommendation was also proposed by nurses and nurse leaders during our discussion meeting with stakeholders.

Second, curriculums in educational organizations and on-the-job training for nurse leaders should be updated to support general understanding how to use evidence in leadership decisions. And third, patients and family members should be more involved in the evidence-based approach. It is therefore important that nurse leaders learn how patients’ and family members’ views as stakeholders are better considered as part of the evidence-based leadership approach.

Future studies should be prioritized as follows: establishment of clear parameters for what constitutes and measures evidence-based leadership; use of theories or models in research to inform mechanisms how to effectively change the practice; conducting robust effectiveness studies using trial designs to evaluate the impact of evidence-based leadership; studying the role of patient and family members in improving the quality of clinical care; and investigating the financial impact of the use of evidence-based leadership approach within respective healthcare systems.

Data availability

The authors obtained all data for this review from published manuscripts.

World Health Organization. State of the world’s nursing 2020: investing in education, jobs and leadership. 2020. https://www.who.int/publications/i/item/9789240003279 . Accessed 29 June 2024.

Hersey P, Campbell R. Leadership: a behavioral science approach. The Center for; 2004.

Cline D, Crenshaw JT, Woods S. Nurse leader: a definition for the 21st century. Nurse Lead. 2022;20(4):381–4. https://doi.org/10.1016/j.mnl.2021.12.017 .

Article   Google Scholar  

Chen SS. Leadership styles and organization structural configurations. J Hum Resource Adult Learn. 2006;2(2):39–46.

Google Scholar  

McKibben L. Conflict management: importance and implications. Br J Nurs. 2017;26(2):100–3.

Article   PubMed   Google Scholar  

Haghgoshayie E, Hasanpoor E. Evidence-based nursing management: basing Organizational practices on the best available evidence. Creat Nurs. 2021;27(2):94–7. https://doi.org/10.1891/CRNR-D-19-00080 .

Majers JS, Warshawsky N. Evidence-based decision-making for nurse leaders. Nurse Lead. 2020;18(5):471–5.

Tichy NM, Bennis WG. Making judgment calls. Harvard Business Rev. 2007;85(10):94.

Sousa MJ, Pesqueira AM, Lemos C, Sousa M, Rocha Á. Decision-making based on big data analytics for people management in healthcare organizations. J Med Syst. 2019;43(9):1–10.

Guo R, Berkshire SD, Fulton LV, Hermanson PM. %J L in HS. Use of evidence-based management in healthcare administration decision-making. 2017;30(3): 330–42.

Liang Z, Howard P, Rasa J. Evidence-informed managerial decision-making: what evidence counts?(part one). Asia Pac J Health Manage. 2011;6(1):23–9.

Hasanpoor E, Janati A, Arab-Zozani M, Haghgoshayie E. Using the evidence-based medicine and evidence-based management to minimise overuse and maximise quality in healthcare: a hybrid perspective. BMJ evidence-based Med. 2020;25(1):3–5.

Shingler NA, Gonzalez JZ. Ebm: a pathway to evidence-based nursing management. Nurs 2022. 2017;47(2):43–6.

Farokhzadian J, Nayeri ND, Borhani F, Zare MR. Nurse leaders’ attitudes, self-efficacy and training needs for implementing evidence-based practice: is it time for a change toward safe care? Br J Med Med Res. 2015;7(8):662.

Article   PubMed   PubMed Central   Google Scholar  

American Nurses Association. ANA leadership competency model. Silver Spring, MD; 2018.

Royal College of Nursing. Leadership skills. 2022. https://www.rcn.org.uk/professional-development/your-career/nurse/leadership-skills . Accessed 29 June 2024.

Kakemam E, Liang Z, Janati A, Arab-Zozani M, Mohaghegh B, Gholizadeh M. Leadership and management competencies for hospital managers: a systematic review and best-fit framework synthesis. J Healthc Leadersh. 2020;12:59.

Liang Z, Howard PF, Leggat S, Bartram T. Development and validation of health service management competencies. J Health Organ Manag. 2018;32(2):157–75.

World Health Organization. Global Strategic Directions for Nursing and Midwifery. 2021. https://apps.who.int/iris/bitstream/handle/10665/344562/9789240033863-eng.pdf . Accessed 29 June 2024.

NHS Leadership Academy. The nine leadership dimensions. 2022. https://www.leadershipacademy.nhs.uk/resources/healthcare-leadership-model/nine-leadership-dimensions/ . Accessed 29 June 2024.

Canadian Nurses Association. Evidence-informed decision-making and nursing practice: Position statement. 2018. https://hl-prod-ca-oc-download.s3-ca-central-1.amazonaws.com/CNA/2f975e7e-4a40-45ca-863c-5ebf0a138d5e/UploadedImages/documents/Evidence_informed_Decision_making_and_Nursing_Practice_position_statement_Dec_2018.pdf . Accessed 29 June 2024.

Hasanpoor E, Hajebrahimi S, Janati A, Abedini Z, Haghgoshayie E. Barriers, facilitators, process and sources of evidence for evidence-based management among health care managers: a qualitative systematic review. Ethiop J Health Sci. 2018;28(5):665–80.

PubMed   PubMed Central   Google Scholar  

Collins DB, Holton EF III. The effectiveness of managerial leadership development programs: a meta-analysis of studies from 1982 to 2001. Hum Res Dev Q. 2004;15(2):217–48.

Cummings GG, Lee S, Tate K, Penconek T, Micaroni SP, Paananen T, et al. The essentials of nursing leadership: a systematic review of factors and educational interventions influencing nursing leadership. Int J Nurs Stud. 2021;115:103842.

Clavijo-Chamorro MZ, Romero-Zarallo G, Gómez-Luque A, López-Espuela F, Sanz-Martos S, López-Medina IM. Leadership as a facilitator of evidence implementation by nurse managers: a metasynthesis. West J Nurs Res. 2022;44(6):567–81.

Young SK. Evidence-based management: a literature review. J Nurs Adm Manag. 2002;10(3):145–51.

Williams LL. What goes around comes around: evidence-based management. Nurs Adm Q. 2006;30(3):243–51.

Fraser I. Organizational research with impact: working backwards. Worldviews Evidence-Based Nurs. 2004;1:S52–9.

Roshanghalb A, Lettieri E, Aloini D, Cannavacciuolo L, Gitto S, Visintin F. What evidence on evidence-based management in healthcare? Manag Decis. 2018;56(10):2069–84.

Jaana M, Vartak S, Ward MM. Evidence-based health care management: what is the research evidence available for health care managers? Eval Health Prof. 2014;37(3):314–34.

Tate K, Hewko S, McLane P, Baxter P, Perry K, Armijo-Olivo S, et al. Learning to lead: a review and synthesis of literature examining health care managers’ use of knowledge. J Health Serv Res Policy. 2019;24(1):57–70.

Geerts JM, Goodall AH, Agius S, %J SS. Medicine. Evidence-based leadership development for physicians: a systematic literature review. 2020;246: 112709.

Barends E, Rousseau DM, Briner RB. Evidence-based management: The basic principles. Amsterdam; 2014. https://research.vu.nl/ws/portalfiles/portal/42141986/complete+dissertation.pdf#page=203 . Accessed 29 June 2024.

Stern C, Lizarondo L, Carrier J, Godfrey C, Rieger K, Salmond S, et al. Methodological guidance for the conduct of mixed methods systematic reviews. JBI Evid Synthesis. 2020;18(10):2108–18. https://doi.org/10.11124/JBISRIR-D-19-00169 .

Lancet T. 2020: unleashing the full potential of nursing. Lancet (London, England). 2019. p. 1879.

Välimäki MA, Lantta T, Hipp K, Varpula J, Liu G, Tang Y, et al. Measured and perceived impacts of evidence-based leadership in nursing: a mixed-methods systematic review protocol. BMJ Open. 2021;11(10):e055356. https://doi.org/10.1136/bmjopen-2021-055356 .

The Joanna Briggs Institute. Joanna Briggs Institute reviewers’ manual: 2014 edition. Joanna Briggs Inst. 2014; 88–91.

Pearson A, White H, Bath-Hextall F, Salmond S, Apostolo J, Kirkpatrick P. A mixed-methods approach to systematic reviews. JBI Evid Implement. 2015;13(3):121–31.

Johnson RB, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educational Researcher. 2004;33(7):14–26.

Hong, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Reviews. 2017;6(1):61. https://doi.org/10.1186/s13643-017-0454-2 .

Ramis MA, Chang A, Conway A, Lim D, Munday J, Nissen L. Theory-based strategies for teaching evidence-based practice to undergraduate health students: a systematic review. BMC Med Educ. 2019;19(1):1–13.

Sackett DL, Rosenberg WM, Gray JM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. Bmj. British Medical Journal Publishing Group; 1996. pp. 71–2.

Goodman JS, Gary MS, Wood RE. Bibliographic search training for evidence-based management education: a review of relevant literatures. Acad Manage Learn Educ. 2014;13(3):322–53.

Aromataris E, Munn Z. Chapter 3: Systematic reviews of effectiveness. JBI Manual for Evidence Synthesis. 2020; https://synthesismanual.jbi.global .

Munn Z, Barker TH, Moola S, Tufanaru C, Stern C, McArthur A et al. Methodological quality of case series studies: an introduction to the JBI critical appraisal tool. 2020;18(10): 2127–33.

Hong Q, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed methods Appraisal Tool (MMAT) Version 2018: user guide. Montreal: McGill University; 2018.

McKenna J, Jeske D. Ethical leadership and decision authority effects on nurses’ engagement, exhaustion, and turnover intention. J Adv Nurs. 2021;77(1):198–206.

Maxwell M, Hibberd C, Aitchison P, Calveley E, Pratt R, Dougall N, et al. The TIDieR (template for intervention description and replication) checklist. The patient Centred Assessment Method for improving nurse-led biopsychosocial assessment of patients with long-term conditions: a feasibility RCT. NIHR Journals Library; 2018.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77–101.

Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, et al. Stakeholder involvement in systematic reviews: a scoping review. Syst Reviews. 2018;7:1–26.

Braye S, Preston-Shoot M. Emerging from out of the shadows? Service user and carer involvement in systematic reviews. Evid Policy. 2005;1(2):173–93.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Reviews. 2021;10(1):1–11.

Porta M. Pilot investigation, study. A dictionary of epidemiology. Oxford University Press Oxford; 2014. p. 215.

Kreis J, Puhan MA, Schünemann HJ, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expect. 2013;16(4):323–37.

Joseph ML, Nelson-Brantley HV, Caramanica L, Lyman B, Frank B, Hand MW, et al. Building the science to guide nursing administration and leadership decision making. JONA: J Nurs Adm. 2022;52(1):19–26.

Gifford W, Davies BL, Graham ID, Tourangeau A, Woodend AK, Lefebre N. Developing Leadership Capacity for Guideline Use: a pilot cluster Randomized Control Trial: Leadership Pilot Study. Worldviews Evidence-Based Nurs. 2013;10(1):51–65. https://doi.org/10.1111/j.1741-6787.2012.00254.x .

Hsieh HY, Henker R, Ren D, Chien WY, Chang JP, Chen L, et al. Improving effectiveness and satisfaction of an electronic charting system in Taiwan. Clin Nurse Specialist. 2016;30(6):E1–6. https://doi.org/10.1097/NUR.0000000000000250 .

McAllen E, Stephens K, Swanson-Biearman B, Kerr K, Whiteman K. Moving Shift Report to the Bedside: an evidence-based Quality Improvement Project. OJIN: Online J Issues Nurs. 2018;23(2). https://doi.org/10.3912/OJIN.Vol23No02PPT22 .

Thomas M, Autencio K, Cesario K. Positive outcomes of an evidence-based pressure injury prevention program. J Wound Ostomy Cont Nurs. 2020;47:S24.

Cullen L, Titler MG. Promoting evidence-based practice: an internship for Staff nurses. Worldviews Evidence-Based Nurs. 2004;1(4):215–23. https://doi.org/10.1111/j.1524-475X.2004.04027.x .

DuBose BM, Mayo AM. Resistance to change: a concept analysis. Nursing forum. Wiley Online Library; 2020. pp. 631–6.

Lalleman PCB, Smid GAC, Lagerwey MD, Shortridge-Baggett LM, Schuurmans MJ. Curbing the urge to care: a bourdieusian analysis of the effect of the caring disposition on nurse middle managers’ clinical leadership in patient safety practices. Int J Nurs Stud. 2016;63:179–88.

Article   CAS   PubMed   Google Scholar  

Martin E, Warshawsky N. Guiding principles for creating value and meaning for the next generation of nurse leaders. JONA: J Nurs Adm. 2017;47(9):418–20.

Griffiths P, Recio-Saucedo A, Dall’Ora C, Briggs J, Maruotti A, Meredith P, et al. The association between nurse staffing and omissions in nursing care: a systematic review. J Adv Nurs. 2018;74(7):1474–87. https://doi.org/10.1111/jan.13564 .

Lúanaigh PÓ, Hughes F. The nurse executive role in quality and high performing health services. J Nurs Adm Manag. 2016;24(1):132–6.

de Kok E, Weggelaar-Jansen AM, Schoonhoven L, Lalleman P. A scoping review of rebel nurse leadership: descriptions, competences and stimulating/hindering factors. J Clin Nurs. 2021;30(17–18):2563–83.

Warshawsky NE. Building nurse manager well-being by reducing healthcare system demands. JONA: J Nurs Adm. 2022;52(4):189–91.

Paez A. Gray literature: an important resource in systematic reviews. J Evidence-Based Med. 2017;10(3):233–40.

McAuley L, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000;356(9237):1228–31.

Sarah S. Introduction to mixed methods systematic reviews. https://jbi-global-wiki.refined.site/space/MANUAL/4689215/8.1+Introduction+to+mixed+methods+systematic+reviews . Accessed 29 June 2024.

Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546–53.

Download references

Acknowledgements

We want to thank the funding bodies, the Finnish National Agency of Education, Asia Programme, the Department of Nursing Science at the University of Turku, and Xiangya School of Nursing at the Central South University. We also would like to thank the nurses and nurse leaders for their valuable opinions on the topic.

The work was supported by the Finnish National Agency of Education, Asia Programme (grant number 26/270/2020) and the University of Turku (internal fund 26003424). The funders had no role in the study design and will not have any role during its execution, analysis, interpretation of the data, decision to publish, or preparation of the manuscript.

Author information

Authors and affiliations.

Department of Nursing Science, University of Turku, Turku, FI-20014, Finland

Maritta Välimäki, Tella Lantta, Kirsi Hipp & Jaakko Varpula

School of Public Health, University of Helsinki, Helsinki, FI-00014, Finland

Maritta Välimäki

Xiangya Nursing, School of Central South University, Changsha, 410013, China

Shuang Hu, Jiarui Chen, Yao Tang, Wenjun Chen & Xianhong Li

School of Health and Social Services, Häme University of Applied Sciences, Hämeenlinna, Finland

Hunan Cancer Hospital, Changsha, 410008, China

Gaoming Liu

You can also search for this author in PubMed   Google Scholar

Contributions

Study design: MV, XL. Literature search and study selection: MV, KH, TL, WC, XL. Quality assessment: YT, SH, XL. Data extraction: JC, MV, JV, WC, YT, SH, GL. Analysis and interpretation: MV, SH. Manuscript writing: MV. Critical revisions for important intellectual content: MV, XL. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Xianhong Li .

Ethics declarations

Ethics approval and consent to participate.

No ethical approval was required for this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Differences between the original protocol

We modified criteria for the included studies: we included published conference abstracts/proceedings, which form a relatively broad knowledge base in scientific knowledge. We originally planned to conduct a survey with open-ended questions followed by a face-to-face meeting to discuss the preliminary results of the review. However, to avoid extra burden in nurses due to COVID-19, we decided to limit the validation process to the online discussion only.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, supplementary material 6, supplementary material 7, supplementary material 8, supplementary material 9, supplementary material 10, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Välimäki, M., Hu, S., Lantta, T. et al. The impact of evidence-based nursing leadership in healthcare settings: a mixed methods systematic review. BMC Nurs 23 , 452 (2024). https://doi.org/10.1186/s12912-024-02096-4

Download citation

Received : 28 April 2023

Accepted : 13 June 2024

Published : 03 July 2024

DOI : https://doi.org/10.1186/s12912-024-02096-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Evidence-based leadership
• Health services administration
• Organizational development
• Quality in healthcare

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

CASE REPORT article

Invasive fusarium solani infection diagnosed by traditional microbial detection methods and metagenomic next-generation sequencing in a pediatric patient: a case report and literature review.

• 1 Department of Laboratory Medicine, West China Second University Hospital, Sichuan University, Chengdu, China
• 2 Key Laboratory of Birth Defects and Related Diseases of Women and Children (Sichuan University), Ministry of Education, Chengdu, China

Fusarium solani , as an opportunistic pathogen, can infect individuals with immunosuppression, neutropenia, hematopoietic stem cell transplantation (HSCT), or other high-risk factors, leading to invasive or localized infections. Particularly in patients following allogeneic HSCT, Fusarium solani is more likely to cause invasive or disseminated infections. This study focuses on a pediatric patient who underwent HSCT for severe aplastic anemia. Although initial blood cultures were negative, an abnormality was detected in the 1,3-β-D-glucan test (G test) post-transplantation. To determine the causative agent, blood samples were subjected to metagenomic next-generation sequencing (mNGS) and blood cultures simultaneously. Surprisingly, the results of matrix-assisted laser desorption/ionization time-of-flight mass spectrometry (MALDI-TOF MS) and mNGS differed slightly, with mNGS identifying Nectria haematonectria , while MALDI-TOF MS based on culture showed Fusarium solani . To clarify the results, Sanger sequencing was performed for further detection, and the results were consistent with those of MALDI-TOF MS. Since the accuracy of Sanger sequencing is higher than that of mNGS, the diagnosis was revised to invasive Fusarium solani infection. With advancements in technology, various detection methods for invasive fungi have been developed in recent years, such as mNGS, which has high sensitivity. While traditional methods may be time-consuming, they are important due to their high specificity. Therefore, in clinical practice, it is essential to utilize both traditional and novel detection methods in a complementary manner to enhance the diagnosis of invasive fungal infections.

Introduction

Fusarium is a common saprophytic fungus in soil and is a conditional pathogen capable of causing invasive or localized infections ( 18 ). As an opportunistic pathogen, Fusarium has been increasingly reported to cause invasive or localized infections in humans in recent years, primarily reported in the United States, France, Italy, and others. However, there are limited relevant reports in China ( 22 ). In patients with normal immune function, Fusarium often causes superficial or localized lesions, such as onychomycosis and keratitis ( 12 ). A favorable prognosis can be achieved after active antifungal treatment in most cases ( 9 ). However, in patients undergoing organ transplantation, hematological malignancies, or allogeneic HSCT, Fusarium can easily cause invasive or even disseminated infections, typically affecting the lungs, blood vessels, and sinuses, as well as manifest as cranial-cerebral infections ( 20 ).

As an opportunistic pathogen, Fusarium infection is not as common as Aspergillus infection. Some Fusarium species are opportunistic pathogens that mainly cause local infections in individuals with normal immune systems ( 10 ). However, Fusarium can lead to invasive infections in patients with malignant hematological diseases, aplastic anemia, organ transplantation, or those undergoing chemotherapy ( 18 , 20 ). Disseminated Fusarium disease almost only occurs in immunocompromised individuals, especially in cancer patients with neutropenia caused by cytotoxic drug therapy or bone marrow transplantation ( 13 , 21 ). Granulocytes and macrophages play an important role in the immune defense against Fusarium ( 33 ). Fusarium can cause disseminated infections in patients with severely compromised immunity. In some research centers, Fusarium is the second most common pathogen after Aspergillus infection in high-risk populations such as leukemia patients, solid organ transplant recipients, and allogeneic bone marrow or stem cell transplant recipients ( 1 ). The prognosis of disseminated Fusarium disease, which can be life-threatening, is largely influenced by immune status ( 13 , 21 , 30 ).

Fusarium mainly works by producing Fusarium toxins; some of them produce A-type and B-type trichothecene mycotoxins ( 25 ). The A-type trichothecene includes various toxins such as T-2 toxin and HT-2 toxin. Of these, T-2 toxin is the most important toxin in this group because it inhibits eukaryotic protein synthesis and is highly toxic to white blood cells, often leading to immune suppression. B-type trichothecene mycotoxins include nivalenol, deoxynivalenol, 3-/15 acetyl nivalenol, and Fusarenon-X. Scientific research has shown that these toxins affect the immune system of experimental animals, leading to their increased susceptibility to various pathogens ( 7 , 16 ).

In the past, invasive fungal infections were mainly diagnosed through host factors, clinical features, microbiological examination, and histopathology. With the rapid progress of society, novel technologies for diagnosing invasive fungal infections have emerged, such as polymerase chain reaction-based methods and metagenomic next-generation sequencing. In this study, we diagnosed an invasive Fusarium solani infection with the collaboration of traditional microbial detection methods and metagenomic next-generation sequencing in a pediatric patient.

Case presentation

A 3-year-old girl was diagnosed with severe aplastic anemia in another hospital 9 months ago. After admission to the Pediatric Hematology Department of our hospital, HSCT was scheduled according to an individualized treatment plan. Before transplantation, a chemotherapy pretreatment regimen was administered according to standard protocol. Cefoperazone sodium and tazobactam sodium were used to prevent bacterial infections, micafungin was used to prevent fungal infections, ganciclovir was used to prevent viral infections, and the combination of propranolol and rituximab was used to prevent graft-versus-host disease (GVHD). Twelve days after admission, the patient underwent HLA 7/12-compatible haploid allogeneic HSCT and HLA 5/10-compatible non-hematopoietic stem cell transplantation. Hematologic blood pressure increased during infusion, and no other abnormalities were observed.

Seventeen days after HSCT, the patient showed GVHD-related reactions, which gradually worsened. At the same time, liver function-related enzymes increased, and treatment was performed using plasma exchange. Then, the child developed intermittent fevers. Considering that the infection was worsening and C-reactive protein (CRP) levels were continuously elevated, imipenem and cilastatin sodium were administered to fight bacterial infections, and micafungin was continued to prevent fungal infections.

Forty-nine days after HSCT, the patient had a repeated fever with a thermal peak of 39.2°C, accompanied by increased CRP. The G test was positive at 150 pg/mL (normal value <60 pg/mL), the galactomannan test (GM test) was negative, and the blood culture was negative. A chest computed tomography (CT) examination showed multiple nodules and patchy blurred shadows in both lungs. The lower lobes of both lungs were characterized by lamellar consolidation with obvious interstitial changes ( Figure 1A ). Thus, voriconazole was then added to control the fungal infection.

Figure 1 . Radiological examination of the patient. (A,B) Chest CT on 49 days after HSCT and 81 days after HSCT. (C–F) Craniocerebral enhanced MRI and brain CT on 49 days after HSCT and 81 days after HSCT.

The same treatment regimen was continued after the initial HSCT, with consecutive negative blood culture results observed during this period. Eighty-one days after HSCT, the child developed convulsions, respiratory failure with central mechanism, and electrolyte disturbance. Compared to the previous chest CT results, the lesions and interstitial changes in the lungs were aggravated ( Figure 1B ). Compared to 49 days after HSCT, the craniocerebral MRI with contrast and brain CT showed abnormal signals in the cortex and subcortical regions of both cerebral hemispheres, as well as in the anterior and posterior corners of both cerebral ventricles, notably in the bilateral frontal lobe and the right temporal lobe. A nodular abnormal signal was noted in the right frontal white matter area, the cerebral sulci and fissures were widened and deepened, and the bilateral lateral ventricles were widened, indicating brain atrophy changes ( Figures 1C – E ).

The patient’s inflammatory markers were significantly elevated (CRP 370 mg/L, PCT 1.29 ng/mL), the G test result was positive (218 pg/mL), the GM test was negative, and the blood culture remained negative. Therefore, the antifungal treatment regimen was adjusted to the combination of micafungin and amphotericin B. The patient’s condition was complex and unstable, posing a threat to life at any time. Ninety-four days after HSCT, the patient was transferred to the PICU, and the G test turned negative after treatment. However, the patient was in a coma state, and blood tests indicated significantly reduced levels of red blood cells, white blood cells, and platelets, which led to diffuse intravascular coagulation abnormalities. Continuous infusion of blood products was used to support treatment.

One hundred and sixteen days after HSCT, the G test was positive again (254 pg/mL), but the GM test was still negative. The child developed herpes lesions around her lip and body, containing bloody substances, which gradually ruptured and formed necrotic scabs. In order to identify the cause as soon as possible, the medical team performed mNGS of the pathogen in addition to the routine double-vial blood cultures. Two days later, the mNGS revealed a rare fungus, Nectria haematonectria . Thus, according to the guidelines, the team used voriconazole combined with micafungin antifungal therapy and abandoned the use of vancomycin and ceftazidime.

The blood cultures showed a positive result after 4 days of culture, and oval fungal spores were observed under a microscope after Gram staining ( Figure 2A ), which supported our decision to initiate antifungal treatment. They were inoculated on Sabouraud agar (SDA) after 96 h of culture. The colony morphology on SDA is shown in Figure 2B ; the colony exhibited dense, smooth, white mycelium, and the reverse color was reddish in the center and yellow in peripheral areas. The microstructure of the colony was observed by a KOH fungal smear test, Gram staining, lactic acid phenol cotton blue staining, and fungal fluorescent staining. Microscopic examination revealed septate, hyaline hyphae that were 4–5 microns in diameter, branching at acute angles, with numerous microconidia and sparse macroconidia. The microconidia were hyaline, unicellular to bicellular, and cylindrical to oval and formed long lateral phialides. The macroconidia were fusiform, cylindrical, moderately curved, and three or four septate ( Figures 2C – F ).

Figure 2 . Morphological characteristics of Fusarium solani . (A) After 4 days of blood cultivation, oval fungal spores could be observed by microscope after Gram staining (magnification 10 × 100). (B) The morphology of Fusarium solani on SDA after 96 h of culture. (C–F) The morphologies of Fusarium solani were observed using a KOH fungal smear test (magnification 10 × 100), Gram staining (magnification 10 × 100), lactic acid phenol cotton blue staining (magnification 10 × 40), and fungal fluorescent staining (magnification 10 × 100).

In addition, the colony was identified as Fusarium solani with 99.9% confidence using MALDI-TOF MS. However, the results did not match those obtained from mNGS. Therefore, Sanger sequencing was adopted to confirm the identity of the fungus. Prior to sequencing, genomic DNA was extracted, and agarose gel electrophoresis was performed. The Sanger sequencing results confirmed the presence of Fusarium solani , which was consistent with the MALDI-TOF MS findings.

To explore the minimal inhibitory concentrations (MICs) of Fusarium solani , an antifungal susceptibility test was performed using TDR-YEAST, and the values are summarized in Table 1 . MIC values 1 and 2 represent the MICs of Fusarium solani isolated from aerobic and anaerobic culture bottles, respectively.

Table 1 . Antifungal susceptibility profile of Fusarium solani.

Combined with multiple pieces of evidence, the patient developed recurrent fever after HSCT, indicative of a weakened immune system. Pulmonary CT scans indicated a fungal infection, with the G test showing positive results multiple times and the GM test remaining negative ( Figure 3 ). Tests for respiratory pathogens, Epstein–Barr virus (EBV), and cytomegalovirus (CMV) were all negative. Although there were slight discrepancies between the results of MALDI-TOF MS and mNGS, Sanger sequencing was conducted for further detection and yielded consistent results with MALDI-TOF MS. Due to the higher accuracy of Sanger sequencing compared to mNGS, the diagnosis was revised to an invasive Fusarium solani infection. Following guidelines and MICs for Fusarium solani , we adjusted the anti-infective treatment regimen to include a combination of voriconazole and amphotericin B liposomes for the treatment of invasive Fusarium solani infection. We continued to monitor the antifungal efficacy of the treatment. Unfortunately, after more than 5 months of antifungal treatment, the child’s condition continued to deteriorate, and eventually, her parents had to face the difficult decision of discontinuing the treatment.

Figure 3 . Dynamic monitoring of G and GM tests on 3, 7, 38, 49, 81, 94, and 116 days after HSCT.

Although the efficacy of invasive Fusarium solani infection was not satisfactory in this study, experience can be accumulated for the management of similar cases in the future. In this study, after HSCT, successive blood cultures were negative in the early stage, but there was an abnormality in the G test. To identify the causative pathogen, blood samples were simultaneously sent for mNGS and blood culture. Interestingly, the results of MALDI-TOF MS and mNGS were different. The mNGS result was Nectria haematonectria , while culture-based MALDI-TOF MS showed Fusarium solani . To clarify the results, Sanger sequencing was performed for further detection and the results displayed the same with MALDI-TOF MS. Since the accuracy of Sanger sequencing is higher than mNGS, the diagnosis was revised to invasive Fusarium solani infection.

The main manifestations of invasive Fusarium solani infection are fever, skin appearance, and other related changes in invasive organs, but none of them are specific ( 31 ). Once infected, the skin lesions typically present as multiple erythematous papules and painful nodular lesions with central necrosis, which may appear as gangrenous purpura and can involve all parts of the body, especially the extremities. In addition, the lesions have an evolutionary stage, 75% of patients have skin lesions, and approximately 10% of patients have concentric rings. In this case, papular changes emerged at the beginning of admission, and there were cutaneous lesions of the trunk due to Fusarium solani infection. As we all know, invasive Fusarium solani infections have a poor prognosis, and patients are often accompanied by neutropenia and immunocompromised. An analysis of adult case data from the United States and Canada showed that the incidence of invasive Fusarium infection after HSCT was 35.4% ( 11 ). Maged et al. ( 17 ) calculated that the mortality rate of disseminated Fusarium infection reached 50%. A recent study of children with invasive Fusarium infection showed a 90-day survival rate of 77%, which is significantly higher than that of adults, but the primary disease distribution is different ( 2 ).

In this case, the results of MALDI-TOF MS and mNGS were slightly different. The mNGS result was Nectria haematonectria , while culture-based MALDI-TOF MS showed Fusarium solani . To find out the truth, Sanger sequencing was performed for further detection and the results displayed the same with MALDI-TOF MS. In order to clarify the results, many relevant studies have been read, and we have summarized the following reasons: first, Sanger sequencing has a longer read length and higher accuracy than mNGS, second, Fusarium is a sexual fungus of Nectria , so it is difficult to distinguish them using mNGS, and third, the amount of fungi in the original specimen was too small, and the background signal was too high. In addition, due to the thick fungal wall, it is difficult to break the wall to extract genomic DNA. Last but not least, improper storage of specimens led to the degradation of nucleic acids during mNGS sequencing.

Literature review

The incidence of invasive fungal infection is on the rise with a high fatality rate ( 28 ). In order to improve the detection rate of invasive fungal infections and seize the opportunity for early treatment, many novel technologies and methods have been developed in recent years ( 3 – 6 , 8 , 14 , 15 , 19 , 23 , 26 , 27 , 29 , 32 , 34 , 35 ). These novel methods are listed in Table 2 along with traditional invasive fungal detection methods.

Table 2 . A list of traditional and novel invasive fungal detection methods.

The treatment of disseminated Fusarium infection is very difficult, and the mortality rate is extremely high ( 24 ). According to the 2021 global guideline for the diagnosis and management of rare mold infections, voriconazole or a lipid formulation of amphotericin B was strongly recommended for the primary treatment of invasive fusariosis. Amphotericin B deoxycholate should not be used if other active antifungal agents are available. For other agents, a conditional recommendation is indicated. Combination therapy is frequently used in the primary treatment of invasive fusariosis because of the severity of the disease, difficulties in achieving voriconazole trough concentrations within the targeted range, and because minimum inhibitory concentrations for azoles and polyenes are often high.

Primary combination therapy, with the potential for early step-down to monotherapy later (once minimum inhibitory concentrations of azoles and polyenes are available), is an approach strongly recommended ( 9 ). In this case, despite the patient continuing to use micafungin to prevent fungal infection, they developed a Fusarium solani bloodstream infection, indicating resistance to echinocandin drugs. The selection of antifungal drugs should be based on the adjustment plan of antifungal susceptibility testing conducted in vitro . Additionally, elevated neutrophils are an important prognostic factor for this disease. The patient has had extremely low neutrophil levels since admission, with a minimum absolute neutrophil count (ANC) of 0.00 × 10 9 /L, which is also a significant factor for poor prognosis.

In recent years, the incidence rate of disseminated Fusarium disease in patients with hematologic malignancies has increased, and the mortality rate is extremely high. Attention should be paid to the diagnosis of the disease to distinguish it from other infections and to determine the most appropriate antifungal treatment program as soon as possible. As a case study has shown, early intervention using mNGS can be beneficial, especially when conventional treatments have failed; however, it is important to carefully evaluate the accuracy of the identification results. Additionally, a combination of traditional and modern detection methods should be utilized to enhance the diagnosis of invasive fungal infections.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in the article/supplementary material.

Ethics statement

The studies involving humans were approved by Medical Ethics Committee, West China Second University Hospital, Sichuan University. The studies were conducted in accordance with the local legislation and institutional requirements. The human samples used in this study were acquired from primarily isolated as part of your previous study for which ethical approval was obtained. Written informed consent for participation was not required from the participants or the participants’ legal guardians/next of kin in accordance with the national legislation and institutional requirements.

Author contributions

JL: Conceptualization, Writing – review & editing, Methodology, Writing – original draft, Data curation, Validation. LL: Writing – review & editing, Formal analysis, Software. XL: Validation, Writing – review & editing. WW: Funding acquisition, Writing – review & editing. ZY: Funding acquisition, Writing – review & editing. WZ: Data curation, Writing – review & editing. YJ: Project administration, Resources, Writing – review & editing. LK: Funding acquisition, Resources, Supervision, Writing – review & editing.

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This study was supported by the National Science Foundation of China (No. 82300015) and the Sichuan Science and Technology Program (No. 2024NSFSC1942 to XL, No. 2023NSFSC1699 to LK, and No. 2022NSFSC1420 to WW).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. Al-Hatmi, AMS, Bonifaz, A, Ranque, S, Sybren de Hoog, G, Verweij, PE, and Meis, JF. Current antifungal treatment of fusariosis. Int J Antimicrob Agents . (2018) 51:326–32. doi: 10.1016/j.ijantimicag.2017.06.017

Crossref Full Text | Google Scholar

2. Benish, M, Elitzur, S, Arad-Cohen, N, Barg, AA, Ben-Harosh, M, Bielorai, B, et al. Invasive fusariosis in pediatric hematology/oncology and stem cell transplant patients: a report from the Israeli Society of Pediatric Hematology-Oncology. J Fungi . (2022) 8:387. doi: 10.3390/jof8040387

PubMed Abstract | Crossref Full Text | Google Scholar

3. Bhatnagar, I, Mahato, K, Ealla, KKR, Asthana, A, and Chandra, P. Chitosan stabilized gold nanoparticle mediated self-assembled gliP nanobiosensor for diagnosis of invasive Aspergillosis . Int J Biol Macromol . (2018) 110:449–56. doi: 10.1016/j.ijbiomac.2017.12.084

4. Dabrowski, M, Sharma, PS, Iskierko, Z, Noworyta, K, Cieplak, M, Lisowski, W, et al. Early diagnosis of fungal infections using piezomicrogravimetric and electric chemosensors based on polymers molecularly imprinted with d-arabitol. Biosens Bioelectron . (2016) 79:627–35. doi: 10.1016/j.bios.2015.12.088

5. Dina, NE, Gherman, AMR, Chis, V, Sarbu, C, Wieser, A, Bauer, D, et al. Characterization of clinically relevant fungi via SERS fingerprinting assisted by novel chemometric models. Anal Chem . (2018) 90:2484–92. doi: 10.1021/acs.analchem.7b03124

6. Filkins, LM, Bryson, AL, Miller, SA, and Mitchell, SL. Navigating clinical utilization of direct-from-specimen metagenomic pathogen detection: clinical applications, limitations, and testing recommendations. Clin Chem . (2020) 66:1381–95. doi: 10.1093/clinchem/hvaa183

7. Góral, T, Wiśniewska, H, Ochodzki, P, Nielsen, L, Walentyn-Góral, D, and Stępień, Ł. Relationship between Fusarium head blight, kernel damage, concentration of Fusarium biomass, and Fusarium toxins in grain of winter wheat inoculated with Fusarium culmorum . Toxins . (2018) 11:2. doi: 10.3390/toxins11010002

8. Han, D, Li, R, Shi, J, Tan, P, Zhang, R, and Li, J. Liquid biopsy for infectious diseases: a focus on microbial cell-free DNA sequencing. Theranostics . (2020) 10:5501–13. doi: 10.7150/thno.45554

9. Hoenigl, M, Salmanton-Garcia, J, Walsh, TJ, Nucci, M, Neoh, CF, Jenks, JD, et al. Global guideline for the diagnosis and management of rare mould infections: an initiative of the European Confederation of Medical Mycology in cooperation with the International Society for Human and Animal Mycology and the American Society for Microbiology. Lancet Infect Dis . (2021) 21:e246–57. doi: 10.1016/S1473-3099(20)30784-2

10. Hof, H . The medical relevance of Fusarium spp. J Fungi . (2020) 6:117. doi: 10.3390/jof6030117

11. Horn, DL, Freifeld, AG, Schuster, MG, Azie, NE, Franks, B, and Kauffman, CA. Treatment and outcomes of invasive fusariosis: review of 65 cases from the PATH Alliance( ® ) registry. Mycoses . (2014) 57:652–8. doi: 10.1111/myc.12212

12. Kaur, R, and Maheshwari, M. Hyperkeratotic warty skin lesion of foot caused by Fusarium oxysporum . Indian J Dermatol . (2013) 58:159. doi: 10.4103/0019-5154.108082

13. Khwaja, A, Bjorkholm, M, Gale, RE, Levine, RL, Jordan, CT, Ehninger, G, et al. Acute myeloid leukaemia. Nat Rev Dis Primers . (2016) 2:16010. doi: 10.1038/nrdp.2016.10

14. Kwasny, D, Tehrani, SE, Almeida, C, Schjodt, I, Dimaki, M, and Svendsen, WE. Direct detection of Candida albicans with a membrane based electrochemical impedance spectroscopy sensor. Sensors . (2018) 18:2214. doi: 10.3390/s18072214

15. Mery, A, Sendid, B, Francois, N, Cornu, M, Poissy, J, Guerardel, Y, et al. Application of mass spectrometry technology to early diagnosis of invasive fungal infections. J Clin Microbiol . (2016) 54:2786–97. doi: 10.1128/JCM.01655-16

16. Meyer, JC, Birr, T, Hennies, I, Wessels, D, and Schwarz, K. Reduction of deoxynivalenol, T-2 and HT-2 toxins and associated Fusarium species during commercial and laboratory de-hulling of milling oats. Food Addit Contam A . (2022) 39:1163–83. doi: 10.1080/19440049.2022.2059576

17. Muhammed, M, Anagnostou, T, Desalermos, A, Kourkoumpetis, TK, Carneiro, HA, Glavis-Bloom, J, et al. Fusarium infection: report of 26 cases and review of 97 cases from the literature. Medicine . (2013) 92:305–16. doi: 10.1097/MD.0000000000000008

18. Nucci, M, and Anaissie, E. Fusarium infections in immunocompromised patients. Clin Microbiol Rev . (2007) 20:695–704. doi: 10.1128/CMR.00014-07

19. Obrucova, H, Kotaskova, I, Tihelkova, R, Hola, V, Ruzicka, F, and Freiberger, T. Fluorescent capillary electrophoresis is superior to culture in detecting Candida species from samples of urinary catheters and ureteral stents with mono- or polyfungal biofilm growth. J Clin Microbiol . (2019) 57:e01861. doi: 10.1128/JCM.01861-18

20. Pana, ZD, Roilides, E, Warris, A, Groll, AH, and Zaoutis, T. Epidemiology of invasive fungal disease in children. J Pediatric Infect Dis Soc . (2017) 6:S3–S11. doi: 10.1093/jpids/pix046

21. Pizzo, PA . Management of Patients with fever and neutropenia through the arc of time: a narrative review. Ann Intern Med . (2019) 170:389–97. doi: 10.7326/M18-3192

22. Poole, GJ, Smiley, RW, Walker, C, Huggins, D, Rupp, R, Abatzoglou, J, et al. Effect of climate on the distribution of Fusarium spp. causing crown rot of wheat in the Pacific Northwest of the United States. Phytopathology . (2013) 103:1130–40. doi: 10.1094/PHYTO-07-12-0181-R

23. Ribes, A, Aznar, E, Santiago-Felipe, S, Xifre-Perez, E, Tormo-Mas, MA, Peman, J, et al. Selective and sensitive probe based in oligonucleotide-capped nanoporous alumina for the rapid screening of infection produced by Candida albicans . ACS Sensors . (2019) 4:1291–8. doi: 10.1021/acssensors.9b00169

24. Salmanton-Garcia, J, Hoenigl, M, Gangneux, JP, Segal, E, Alastruey-Izquierdo, A, Arikan Akdagli, S, et al. The current state of laboratory mycology and access to antifungal treatment in Europe: a European Confederation of Medical Mycology survey. Lancet Microbe . (2023) 4:e47–56. doi: 10.1016/S2666-5247(22)00261-0

25. Schollenberger, M, Drochner, W, and Muller, HM. Fusarium toxins of the scirpentriol subgroup: a review. Mycopathologia . (2007) 164:101–18. doi: 10.1007/s11046-007-9036-5

26. Serin, I, and Dogu, MH. Serum Aspergillus galactomannan lateral flow assay for the diagnosis of invasive aspergillosis: a single-centre study. Mycoses . (2021) 64:678–83. doi: 10.1111/myc.13265

27. Shi, X, Zhang, X, Yao, Q, and He, F. A novel method for the rapid detection of microbes in blood using pleurocidin antimicrobial peptide functionalized piezoelectric sensor. J Microbiol Methods . (2017) 133:69–75. doi: 10.1016/j.mimet.2016.12.005

28. Supatharawanich, S, Narkbunnam, N, Vathana, N, Takpradit, C, Phuakpet, K, Pongtanakul, B, et al. Invasive fungal diseases in children with acute leukemia and severe aplastic anemia. Mediterr J Hematol Infect Dis . (2021) 13:e2021039. doi: 10.4084/MJHID.2021.039

29. Swetha, PDP, Nikitha, A, Shenoy, MM, Shim, YB, and Prasad, KS. Ni/Ni(OH) 2 -rGO nanocomposites sensor for the detection of long forgotten mycotoxin, xanthomegnin. Talanta . (2023) 253:123953. doi: 10.1016/j.talanta.2022.123953

30. Thornton, CR . Detection of the ‘Big Five’ mold killers of humans: Aspergillus , Fusarium , Lomentospora , Scedosporium and Mucormycetes . Adv Appl Microbiol . (2020) 110:1–61. doi: 10.1016/bs.aambs.2019.10.003

31. Tortorano, AM, Richardson, M, Roilides, E, van Diepeningen, A, Caira, M, Munoz, P, et al. ESCMID and ECMM joint guidelines on diagnosis and management of hyalohyphomycosis: Fusarium spp., Scedosporium spp. and others. Clin Microbiol Infect . (2014) 20:27–46. doi: 10.1111/1469-0691.12465

32. Tsai, SM, Goshia, T, Chen, YC, Kagiri, A, Sibal, A, Chiu, MH, et al. High-throughput label-free microcontact printing graphene-based biosensor for valley fever. Colloids Surf B . (2018) 170:219–23. doi: 10.1016/j.colsurfb.2018.06.011

33. Tupaki-Sreepurna, A, and Kindo, AJ. Fusarium : the versatile pathogen. Indian J Med Microbiol . (2018) 36:8–17. doi: 10.4103/ijmm.IJMM_16_24

34. Wang, J, Meng, S, Lin, K, Yi, X, Sun, Y, Xu, X, et al. Leveraging single-cell Raman spectroscopy and single-cell sorting for the detection and identification of yeast infections. Anal Chim Acta . (2023) 1239:340658. doi: 10.1016/j.aca.2022.340658

35. Wang, Y, Yang, M, Xia, Y, Yan, J, Zou, J, and Zhang, D. Application and evaluation of nucleic acid sequence-based amplification, PCR and cryptococcal antigen test for diagnosis of cryptococcosis. BMC Infect Dis . (2021) 21:1020. doi: 10.1186/s12879-021-06678-4

Keywords: Fusarium solani , invasive fungal infections, detection, MALDI-TOF MS, Sanger sequencing, mNGS

Citation: Ling J, Liang L, Liu X, Wu W, Yan Z, Zhou W, Jiang Y and Kuang L (2024) Invasive Fusarium solani infection diagnosed by traditional microbial detection methods and metagenomic next-generation sequencing in a pediatric patient: a case report and literature review. Front. Med . 11:1322700. doi: 10.3389/fmed.2024.1322700

Received: 16 October 2023; Accepted: 19 June 2024; Published: 08 July 2024.

Reviewed by:

Copyright © 2024 Ling, Liang, Liu, Wu, Yan, Zhou, Jiang and Kuang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Linghan Kuang, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Indian J Sex Transm Dis AIDS
• v.35(2); Jul-Dec 2014

Reviewing literature for research: Doing it the right way

Shital amin poojary.

Department of Dermatology, K J Somaiya Medical College, Mumbai, Maharashtra, India

Jimish Deepak Bagadia

In an era of information overload, it is important to know how to obtain the required information and also to ensure that it is reliable information. Hence, it is essential to understand how to perform a systematic literature search. This article focuses on reliable literature sources and how to make optimum use of these in dermatology and venereology.

INTRODUCTION

A thorough review of literature is not only essential for selecting research topics, but also enables the right applicability of a research project. Most importantly, a good literature search is the cornerstone of practice of evidence based medicine. Today, everything is available at the click of a mouse or at the tip of the fingertips (or the stylus). Google is often the Go-To search website, the supposed answer to all questions in the universe. However, the deluge of information available comes with its own set of problems; how much of it is actually reliable information? How much are the search results that the search string threw up actually relevant? Did we actually find what we were looking for? Lack of a systematic approach can lead to a literature review ending up as a time-consuming and at times frustrating process. Hence, whether it is for research projects, theses/dissertations, case studies/reports or mere wish to obtain information; knowing where to look, and more importantly, how to look, is of prime importance today.

Literature search

Fink has defined research literature review as a “systematic, explicit and reproducible method for identifying, evaluating, and synthesizing the existing body of completed and recorded work produced by researchers, scholars and practitioners.”[ 1 ]

Review of research literature can be summarized into a seven step process: (i) Selecting research questions/purpose of the literature review (ii) Selecting your sources (iii) Choosing search terms (iv) Running your search (v) Applying practical screening criteria (vi) Applying methodological screening criteria/quality appraisal (vii) Synthesizing the results.[ 1 ]

This article will primarily concentrate on refining techniques of literature search.

Sources for literature search are enumerated in Table 1 .

Sources for literature search

PubMed is currently the most widely used among these as it contains over 23 million citations for biomedical literature and has been made available free by National Center for Biotechnology Information (NCBI), U.S. National Library of Medicine. However, the availability of free full text articles depends on the sources. Use of options such as advanced search, medical subject headings (MeSH) terms, free full text, PubMed tutorials, and single citation matcher makes the database extremely user-friendly [ Figure 1 ]. It can also be accessed on the go through mobiles using “PubMed Mobile.” One can also create own account in NCBI to save searches and to use certain PubMed tools.

PubMed home page showing location of different tools which can be used for an efficient literature search

Tips for efficient use of PubMed search:[ 2 , 3 , 4 ]

Use of field and Boolean operators

When one searches using key words, all articles containing the words show up, many of which may not be related to the topic. Hence, the use of operators while searching makes the search more specific and less cumbersome. Operators are of two types: Field operators and Boolean operators, the latter enabling us to combine more than one concept, thereby making the search highly accurate. A few key operators that can be used in PubMed are shown in Tables ​ Tables2 2 and ​ and3 3 and illustrated in Figures ​ Figures2 2 and ​ and3 3 .

Field operators used in PubMed search

Boolean operators used in PubMed search

PubMed search results page showing articles on donovanosis using the field operator [TIAB]; it shows all articles which have the keyword “donovanosis” in either title or abstract of the article

PubMed search using Boolean operators ‘AND’, ‘NOT’; To search for articles on treatment of lepra reaction other than steroids, after clicking the option ‘Advanced search’ on the home page, one can build the search using ‘AND’ option for treatment and ‘NOT’ option for steroids to omit articles on steroid treatment in lepra reaction

Use of medical subject headings terms

These are very specific and standardized terms used by indexers to describe every article in PubMed and are added to the record of every article. A search using MeSH will show all articles about the topic (or keywords), but will not show articles only containing these keywords (these articles may be about an entirely different topic, but still may contain your keywords in another context in any part of the article). This will make your search more specific. Within the topic, specific subheadings can be added to the search builder to refine your search [ Figure 4 ]. For example, MeSH terms for treatment are therapy and therapeutics.

PubMed search using medical subject headings (MeSH) terms for management of gonorrhea. Click on MeSH database ( Figure 1 ) →In the MeSH search box type gonorrhea and click search. Under the MeSH term gonorrhea, there will be a list of subheadings; therapy, prevention and control, click the relevant check boxes and add to search builder →Click on search →All articles on therapy, prevention and control of gonorrhea will be displayed. Below the subheadings, there are two options: (1) Restrict to medical subject headings (MeSH) major topic and (2) do not include MeSH terms found below this term in the MeSH hierarchy. These can be used to further refine the search results so that only articles which are majorly about treatment of gonorrhea will be displayed

Two additional options can be used to further refine MeSH searches. These are located below the subheadings for a MeSH term: (1) Restrict to MeSH major topic; checking this box will retrieve articles which are majorly about the search term and are therefore, more focused and (2) Do not include MeSH terms found below this term in the MeSH hierarchy. This option will again give you more focused articles as it excludes the lower specific terms [ Figure 4 ].

Similar feature is available with Cochrane library (also called MeSH), EMBASE (known as EMTREE) and PsycINFO (Thesaurus of Psychological Index Terms).

Saving your searches

Any search that one has performed can be saved by using the ‘Send to’ option and can be saved as a simple word file [ Figure 5 ]. Alternatively, the ‘Save Search’ button (just below the search box) can be used. However, it is essential to set up an NCBI account and log in to NCBI for this. One can even choose to have E-mail updates of new articles in the topic of interest.

Saving PubMed searches. A simple option is to click on the dropdown box next to ‘Send to’ option and then choose among the options. It can be saved as a text or word file by choosing ‘File’ option. Another option is the “Save search” option below the search box but this will require logging into your National Center for Biotechnology Information account. This however allows you to set up alerts for E-mail updates for new articles

Single citation matcher

This is another important tool that helps to find the genuine original source of a particular research work (when few details are known about the title/author/publication date/place/journal) and cite the reference in the most correct manner [ Figure 6 ].

Single citation matcher: Click on “Single citation matcher” on PubMed Home page. Type available details of the required reference in the boxes to get the required citation

Full text articles

In any search clicking on the link “free full text” (if present) gives you free access to the article. In some instances, though the published article may not be available free, the author manuscript may be available free of charge. Furthermore, PubMed Central articles are available free of charge.

Managing filters

Filters can be used to refine a search according to type of article required or subjects of research. One can specify the type of article required such as clinical trial, reviews, free full text; these options are available on a typical search results page. Further specialized filters are available under “manage filters:” e.g., articles confined to certain age groups (properties option), “Links” to other databases, article specific to particular journals, etc. However, one needs to have an NCBI account and log in to access this option [ Figure 7 ].

Managing filters. Simple filters are available on the ‘search results’ page. One can choose type of article, e.g., clinical trial, reviews etc. Further options are available in the “Manage filters” option, but this requires logging into National Center for Biotechnology Information account

The Cochrane library

Although reviews are available in PubMed, for systematic reviews and meta-analysis, Cochrane library is a much better resource. The Cochrane library is a collection of full length systematic reviews, which can be accessed for free in India, thanks to Indian Council of Medical Research renewing the license up to 2016, benefitting users all over India. It is immensely helpful in finding detailed high quality research work done in a particular field/topic [ Figure 8 ].

Cochrane library is a useful resource for reliable, systematic reviews. One can choose the type of reviews required, including trials

An important tool that must be used while searching for research work is screening. Screening helps to improve the accuracy of search results. It is of two types: (1) Practical: To identify a broad range of potentially useful studies. Examples: Date of publication (last 5 years only; gives you most recent updates), participants or subjects (humans above 18 years), publication language (English only) (2) methodological: To identify best available studies (for example, excluding studies not involving control group or studies with only randomized control trials).

Selecting the right quality of literature is the key to successful research literature review. The quality can be estimated by what is known as “The Evidence Pyramid.” The level of evidence of references obtained from the aforementioned search tools are depicted in Figure 9 . Systematic reviews obtained from Cochrane library constitute level 1 evidence.

Evidence pyramid: Depicting the level of evidence of references obtained from the aforementioned search tools

Thus, a systematic literature review can help not only in setting up the basis of a good research with optimal use of available information, but also in practice of evidence-based medicine.

Source of Support: Nil.

Conflict of Interest: None declared.

• Search Menu
• Sign in through your institution
• Advance Articles
• Editor's Choice
• Collections
• Supplements
• InSight Papers
• BSR Registers Papers
• Virtual Roundtables
• Author Guidelines
• Submission Site
• Open Access Options
• Self-Archiving Policy
• About Rheumatology
• About the British Society for Rheumatology
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Dispatch Dates
• Terms and Conditions
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, supplementary material, data availability, acknowledgements.

• < Previous

Comparative efficacy and safety of bimekizumab in psoriatic arthritis: a systematic literature review and network meta-analysis

• Article contents
• Figures & tables
• Supplementary Data

Philip J Mease, Dafna D Gladman, Joseph F Merola, Peter Nash, Stacy Grieve, Victor Laliman-Khara, Damon Willems, Vanessa Taieb, Adam R Prickett, Laura C Coates, Comparative efficacy and safety of bimekizumab in psoriatic arthritis: a systematic literature review and network meta-analysis, Rheumatology , Volume 63, Issue 7, July 2024, Pages 1779–1789, https://doi.org/10.1093/rheumatology/kead705

• Permissions Icon Permissions

To understand the relative efficacy and safety of bimekizumab, a selective inhibitor of IL-17F in addition to IL-17A, vs other biologic and targeted synthetic DMARDs (b/tsDMARDs) for PsA using network meta-analysis (NMA).

A systematic literature review (most recent update conducted on 1 January 2023) identified randomized controlled trials (RCTs) of b/tsDMARDs in PsA. Bayesian NMAs were conducted for efficacy outcomes at Weeks 12–24 for b/tsDMARD-naïve and TNF inhibitor (TNFi)-experienced patients. Safety at Weeks 12–24 was analysed in a mixed population. Odds ratios (ORs) and differences of mean change with the associated 95% credible interval (CrI) were calculated for the best-fitting models, and the surface under the cumulative ranking curve (SUCRA) values were calculated to determine relative rank.

The NMA included 41 RCTs for 22 b/tsDMARDs. For minimal disease activity (MDA), bimekizumab ranked 1st in b/tsDMARD-naïve patients and 2nd in TNFi-experienced patients. In b/tsDMARD-naïve patients, bimekizumab ranked 6th, 5th and 3rd for ACR response ACR20/50/70, respectively. In TNFi-experienced patients, bimekizumab ranked 1st, 2nd and 1st for ACR20/50/70, respectively. For Psoriasis Area and Severity Index 90/100, bimekizumab ranked 2nd and 1st in b/tsDMARD-naïve patients, respectively, and 1st and 2nd in TNFi-experienced patients, respectively. Bimekizumab was comparable to b/tsDMARDs for serious adverse events.

Bimekizumab ranked favourably among b/tsDMARDs for efficacy on joint, skin and MDA outcomes, and showed comparable safety, suggesting it may be a beneficial treatment option for patients with PsA.

For joint efficacy, bimekizumab ranked highly among approved biologic/targeted synthetic DMARDs (b/tsDMARDs).

Bimekizumab provides better skin efficacy (Psoriasis Area and Severity Index, PASI100 and PASI90) than many other available treatments in PsA.

For minimal disease activity, bimekizumab ranked highest of all available b/tsDMARDs in b/tsDMARD-naïve and TNF inhibitor–experienced patients.

PsA is a chronic, systemic, inflammatory disease in which patients experience a high burden of illness [ 1–3 ]. PsA has multiple articular and extra-articular disease manifestations including peripheral arthritis, axial disease, enthesitis, dactylitis, skin psoriasis (PSO) and psoriatic nail disease [ 4 , 5 ]. Patients with PsA can also suffer from related inflammatory conditions, uveitis and IBD [ 4 , 5 ]. Approximately one fifth of all PSO patients, increasing to one quarter of patients with moderate to severe PSO, will develop PsA over time [ 6 , 7 ].

The goal of treatment is to control inflammation and prevent structural damage to minimize disease burden, normalize function and social participation, and maximize the quality of life of patients [ 1 , 4 ]. As PsA is a heterogeneous disease, the choice of treatment is guided by individual patient characteristics, efficacy against the broad spectrum of skin and joint symptoms, and varying contraindications to treatments [ 1 , 4 ]. There are a number of current treatments classed as conventional DMARDs such as MTX, SSZ, LEF; biologic (b) DMARDs such as TNF inhibitors (TNFi), IL inhibitors and cytotoxic T lymphocyte antigen 4 (CTLA4)-immunoglobulin; and targeted synthetic (ts) DMARDs which include phosphodiesterase-4 (PDE4) and Janus kinase (JAK) inhibitors [ 1 , 8 ].

Despite the number of available treatment options, the majority of patients with PsA report that they do not achieve remission and additional therapeutic options are needed [ 9 , 10 ]. Thus, the treatment landscape for PsA continues to evolve and treatment decisions increase in complexity, especially as direct comparative data are limited [ 2 ].

Bimekizumab is a monoclonal IgG1 antibody that selectively inhibits IL-17F in addition to IL-17A, which is approved for the treatment of adults with active PsA in Europe [ 11 , 12 ]. Both IL-17A and IL-17F are pro-inflammatory cytokines implicated in PsA [ 11 , 13 ]. IL-17F is structurally similar to IL-17A and expressed by the same immune cells; however, the mechanisms that regulate expression and kinetics differ [ 13 , 14 ]. IL-17A and IL-17F are expressed as homodimers and as IL-17A–IL-17F heterodimers that bind to and signal via the same IL-17 receptor A/C complex [ 13 , 15 ].

In vitro studies have demonstrated that the dual inhibition of both IL-17A and IL-17F with bimekizumab was more effective at suppressing PsA inflammatory genes and T cell and neutrophil migration, and periosteal new bone formation, than blocking IL-17A alone [ 11 , 14 , 16 , 17 ]. Furthermore, IL-17A and IL-17F protein levels are elevated in psoriatic lesions and the superiority of bimekizumab 320 mg every 4 weeks (Q4W) or every 8 weeks (Q8W) over the IL-17A inhibitor, secukinumab, in complete clearance of psoriatic skin was demonstrated in a head-to-head trial in PSO [ 16 , 18 ]. Collectively, this evidence suggests that neutralizing both IL-17F and IL-17A may provide more potent abrogation of IL-17-mediated inflammation than IL-17A alone.

Bimekizumab 160 mg Q4W demonstrated significant improvements in efficacy outcomes compared with placebo, and an acceptable safety profile in adults with PsA in the phase 3 RCTs BE OPTIMAL (NCT03895203) (b/tsDMARD-naïve patients) and BE COMPLETE (NCT03896581) (TNFi inadequate responders) [ 19 , 20 ].

The objective of this study was to establish the comparative efficacy and safety of bimekizumab 160 mg Q4W vs other available PsA treatments, using network meta-analysis (NMA).

Search strategy

A systematic literature review (SLR) was conducted according to the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines [ 21 ] and adhered to the principles outlined in the Cochrane Handbook for Systematic Reviews of Interventions, Centre for Reviews and Dissemination’s Guidance for Undertaking Reviews in Healthcare, and Methods for the Development of National Institute of Health and Care Excellence (NICE) Public Health Guidance [ 22–24 ]. The SLR of English-language publications was originally conducted on 3 December 2015, with updates on 7 January 2020, 2 May 2022 and 1 January 2023 in Medical Literature Analysis and Retrieval System Online (MEDLINE ® ), Excerpta Medica Database (Embase ® ) and the Cochrane Central Register of Controlled Trials (CENTRAL) for literature published from January 1991 onward using the Ovid platform. Additionally, bibliographies of SLRs and meta-analyses identified through database searches were reviewed to ensure any publications not identified in the initial search were included in this SLR. Key clinical conference proceedings not indexed in Ovid (from October 2019 to current) and ClinicalTrials.gov were also manually searched. The search strategy is presented in Supplementary Table S1 (available at Rheumatology online).

Study inclusion

Identified records were screened independently and in duplicate by two reviewers and any discrepancies were reconciled via discussion or a third reviewer. The SLR inclusion criteria were defined by the Patient populations, Interventions, Comparators, Outcome measures, and Study designs (PICOS) Statement ( Supplementary Table S2 , available at Rheumatology online). The SLR included published studies assessing approved therapies for the treatment of PsA. Collected data included study and patient population characteristics, interventions, comparators, and reported clinical and patient-reported outcomes relevant to PsA. For efficacy outcomes, pre-crossover data were extracted in studies where crossover occurred. All publications included in the analysis were evaluated according to the Cochrane risk-of-bias tool for randomized trials as described in the Cochrane Handbook [ 25 ].

Network meta-analysis methods

NMA is the quantitative assessment of relative treatment effects and associated uncertainty of two or more interventions [ 26 , 27 ]. It is used frequently in health technology assessment, guideline development and to inform treatment decision making in clinical practice [ 26 ].

Bimekizumab 160 mg Q4W was compared with current b/tsDMARDs at regulatory-approved doses ( Table 1 ) by NMA. All comparators were selected on the basis they were relevant to clinical practice, i.e. recommended by key clinical guidelines, licensed by key regulatory bodies and/or routinely used.

NMA intervention and comparators

See Supplementary Table S4 , available at Rheumatology online for additional dosing schedules used in included studies. BID: twice daily; CTLA4-Ig: cytotoxic T lymphocyte antigen 4-immunoglobulin; IL-17A/17Fi: IL-17A/17F inhibitor; IL-17Ai: IL-17A inhibitor; IL-12/23i: IL-12/23 inhibitor; IL-23i: IL-23 inhibitor; JAKi: Janus kinase inhibitor; NMA: network meta-analysis; PDE-4i: phosphodiesterase-4 inhibitor; Q1W: once weekly; Q2W: every 2 weeks; Q4W: every 4 weeks; Q8W: every 8 weeks; Q12W: every 12 weeks; QD: once daily; TNFi: TNF inhibitor.

Two sets of primary analyses were conducted, one for a b/tsDMARD-naïve PsA population and one for a TNFi-experienced PsA population. Prior treatment with TNFis has been shown to impact the response to subsequent bDMARD treatments [ 28 ]. In addition, most trials involving b/tsDMARDs for the treatment of PsA (including bimekizumab) report separate data on both b/tsDMARD-naïve and TNFi-experienced subgroups, making NMA in each of these patient populations feasible.

For each population the following outcomes were analysed: American College of Rheumatology response (ACR20/50/70), Psoriasis Area and Severity Index (PASI90/100), and minimal disease activity (MDA). The analysis of serious adverse events (SAE) was conducted using a mixed population (i.e. b/tsDMARD-naïve, TNFi-experienced and mixed population data all were included) as patients’ previous TNFI exposure was not anticipated to impact safety outcomes following discussions with clinicians. The NMA included studies for which data were available at week 16, if 16-week data were not available (or earlier crossover occurred), data available at weeks 12, 14 or 24 were included. Pre-crossover data were included in the analyses for efficacy outcomes to avoid intercurrent events.

Heterogeneity between studies for age, sex, ethnicity, mean time since diagnosis, concomitant MTX, NSAIDs or steroid use was assessed using Grubb’s test, also called the extreme Studentized deviate method, to identify outlier studies.

All univariate analyses involved a 10 000 run-in iteration phase and a 10 000-iteration phase for parameter estimation. All calculations were performed using the R2JAGS package to run Just Another Gibbs Sampler (JAGS) 3.2.3 and the code reported in NICE Decision Support Unit (DSU) Technical Support Document Series [ 29–33 ]. Convergence was confirmed through inspection of the ratios of Monte-Carlo error to the standard deviations of the posteriors; values >5% are strong signs of convergence issues [ 31 ]. In some cases, trials reported outcome results of zero (ACR70, PASI100, SAE) in one or more arms for which a continuity correction was applied to mitigate the issue, as without the correction most models were not convergent or provided a large posterior distribution making little clinical sense [ 31 ].

Four NMA models [fixed effects (FE) unadjusted, FE baseline risk-adjusted, random effects (RE) unadjusted and RE baseline risk-adjusted] were assessed and the best-fit models were chosen using methods described in NICE DSU Technical Support Document 2 [ 31 ]. Odds ratios (ORs) and differences of mean change (MC) with the associated 95% credible intervals (CrIs) were calculated for each treatment comparison in the evidence network for the best fitting models and presented in league tables and forest plots. In addition, the probability of bimekizumab 160 mg Q4W being better than other treatments was calculated using surface under the cumulative ranking curve (SUCRA) to determine relative rank. Conclusions (i.e. better/worse or comparable) for bimekizumab 160 mg Q4W vs comparators were based on whether the pairwise 95% CrIs of the ORs/difference of MC include 1 (dichotomous outcomes), 0 (continuous outcomes) or not. In the case where the 95% CrI included 1 or 0, then bimekizumab 160 mg Q4W and the comparator were considered comparable. If the 95% CrI did not include 1 or 0, then bimekizumab 160 mg Q4W was considered either better or worse depending on the direction of the effect.

Compliance with ethics guidelines

This article is based on previously conducted studies and does not contain any new studies with human participants or animals performed by any of the authors.

Study and patient characteristics

The SLR identified 4576 records through databases and 214 records through grey literature, of which 3143 were included for abstract review. Following the exclusion of a further 1609 records, a total of 1534 records were selected for full-text review. A total of 66 primary studies from 246 records were selected for data extraction. No trial was identified as having a moderate or high risk of bias ( Supplementary Table S3 , available at Rheumatology online).

Of the 66 studies identified in the SLR, 41 studies reported outcomes at weeks 12, 16 or 24 and met the criteria for inclusion in the NMA in either a b/tsDMARD-naïve population ( n  = 20), a TNFi-experienced population ( n  = 5), a mixed population with subgroups ( n  = 13) or a mixed PsA population without subgroups reported ( n  = 3). The PRISMA diagram is presented in Fig. 1 . Included and excluded studies are presented in Supplementary Tables S4 and S5 , respectively (available at Rheumatology online).

PRISMA flow diagram. The PRISMA flow diagram for the SLR conducted to identify published studies assessing approved treatments for the treatment of PsA. cDMARD: conventional DMARD; NMA: network meta-analysis; NR: not reported; PD: pharmacodynamic; PK: pharmacokinetic; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT: randomized controlled trial; SLR: systematic literature review

The baseline study and patient characteristics (where reported) are presented in Supplementary Table S6 (available at Rheumatology online). There were 20–483 patients included in treatment arms. The median age of patients was 48.9 years, the median percentage of males was 50.3% and a median of 92.3% of patients were Caucasian. Patients had a mean time since diagnosis of 7.6 years and a mean PASI score of 8.7. The mean (range) use of concomitant MTX, NSAIDs and steroids were 53.9% (29.1% to 84.0%), 72.4% (33.3% to 100.0%) and 16.8% (9.2% to 30.0%), respectively. Heterogeneity was generally low across studies except for the concomitant use of MTX, NSAIDs and steroids. Using an approach consistent with established NMA methods in PsA [ 34–36 ], a meta-regression model using JAGS code reported in NICE DSU Technical Support Document 3 [ 33 ] was used to account for variation in placebo responses when model-fit statistics suggested that baseline risk-adjusted models provided a better fit to the data.

NMA results

The network diagrams for ACR50 in b/tsDMARD-naïve and TNFi-experienced patients are presented in Fig. 2A and B with network diagrams for other outcomes presented in Supplementary Fig. S1 (available at Rheumatology online). The networks for ACR response were larger, in terms of both number of studies and patients included, than the networks for PASI. Similarly, the networks for b/tsDMARD-naïve patients were larger than TNFi-experienced patients across all outcomes analysed. Placebo was used as a common comparator in all networks and there were a few studies that included more than two arms (OPAL-Broaden, Select-PsA-1, SPIRIT-P1 and BE OPTIMAL) that included adalimumab as the reference arm in b/tsDMARD-naïve patients. Lastly, networks included studies where the primary outcome was evaluated at time points longer than 16 weeks (e.g. EXCEED study at 52 weeks) but as per the methods, 16-week data formed the network.

Network of evidence for ACR50. ( A ) b/tsDMARD-naïve patients. ( B ) TNFi-experienced patients. The size of the circle representing each intervention is proportional to the number of patients included in the analysis. The line width is proportional to the number of studies connecting the interventions. ABA: abatacept; ADA: adalimumab; APR: apremilast; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ: bimekizumab; CZP: certolizumab pegol; ETA: etanercept; GOL: golimumab; GUS: guselkumab; IFX: infliximab; IV: intravenous; IXE: ixekizumab; PBO: placebo; Q4W: every 4 weeks; Q8W: every 8 weeks; RIS: risankizumab; SEC: secukinumab; TNFi-experienced: TNF inhibitor–experienced; TOF: tofacitinib; UPA: upadacitinib; UST: ustekinumab; w/o LD: without loading dose

The best-fit model is noted for each outcome with full model fit statistics for all outcomes presented in Supplementary Table S7 (available at Rheumatology online). Forest plots for ACR50 and PASI100 are presented in Figs 3 and 4 , with forest plots for other outcomes, along with the league tables in Supplementary Fig. S2 and Table S8 , respectively (available at Rheumatology online).

ACR50. The results for the NMA on ACR50 at week 16. ( A ) b/tsDMARD-naïve patients including forest plot and SUCRA values. FE baseline–adjusted model DIC = 469.59. ( B ) TNFi-experienced patients including forest plot and SUCRA values. RE-unadjusted model DIC = 205.33. a Week 24 data were used as week 16 data was not available. * The 95% CrI does not include 1; bimekizumab 160 mg Q4W is considered either better or worse depending on the direction of the effect. ABA: abatacept; ADA: adalimumab; APR: apremilast; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ: bimekizumab; CrI: credible interval; CZP: certolizumab pegol; DIC: deviance information criterion; ETA: etanercept; FE: fixed effects; GOL: golimumab; GUS: guselkumab; IFX: infliximab; IV: intravenous; IXE: ixekizumab; NMA: network meta-analysis; PBO: placebo; Q4W: every 4 weeks; Q8W: every 8 weeks; RE: random effects; RIS: risankizumab; SEC: secukinumab; SUCRA: surface under the cumulative ranking curve; TNFi-experienced: TNF inhibitor–experienced; TOF: tofacitinib; UPA: upadacitinib; UST: ustekinumab; w/o LD: without loading dose

PASI100. The results for the NMA on PASI100 at week 16: ( A ) b/tsDMARD-naïve patients including forest plot and SUCRA values. FE baseline–adjusted model DIC = 150.27. ( B ) TNFi-experienced patients including forest plot and SUCRA values. RE-unadjusted model DIC = 81.76. a Week 24 data were used as week 16 data was not available. * The 95% CrI does not include 1; bimekizumab 160 mg 4W is considered better. ADA: adalimumab; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ, bimekizumab; CrI, credible interval; CZP, certolizumab pegol; DIC, deviance information criterion; FE, fixed effects; GOL, golimumab; GUS, guselkumab; IXE, ixekizumab; NMA, network meta-analysis; PASI, Psoriasis Area and Severity Index; PBO, placebo; Q4W, every 4 weeks; Q8W, every 8 weeks; RE, random effects; SEC, secukinumab; SUCRA, surface under the cumulative ranking curve; TNFi-experienced, TNF inhibitor–experienced; UPA, upadacitinib

Joint outcomes

For ACR50 outcomes, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

b/tsDMARD-naïve patients

Bimekizumab 160 mg Q4W ranked 6th for ACR20 (SUCRA = 0.75), 5th for ACR50 (SUCRA = 0.74) ( Fig. 3A ) and 3rd for ACR70 (SUCRA = 0.80) among 21 treatments. For ACR50, bimekizumab 160 mg Q4W was better than placebo, abatacept 125 mg, guselkumab 100 mg Q4W, ustekinumab 45 mg, risankizumab 150 mg, guselkumab 100 mg Q8W and ustekinumab 90 mg; worse than golimumab 2 mg i.v.; and comparable to the remaining treatments in the network ( Fig. 3A ).

TNFi-experienced patients

Bimekizumab 160 mg Q4W ranked 1st among 16 treatments for ACR20 (SUCRA = 0.96), 2nd among 15 treatments for ACR50 (SUCRA = 0.84) ( Fig. 3B ) and 1st among 16 treatments for ACR70 (SUCRA = 0.83). Bimekizumab 160 mg Q4W was better than placebo, abatacept 125 mg, secukinumab 150 mg without loading dose, tofacitinib 5 mg and secukinumab 150 mg; and comparable to the remaining treatments in the network on ACR50 ( Fig. 3B ).

Skin outcomes

For PASI100 outcomes, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

Bimekizumab 160 mg Q4W ranked 2nd among 15 treatments (SUCRA = 0.89) for PASI90 and 1st among 11 treatments (SUCRA = 0.95) for PASI100 ( Fig. 4A ). Bimekizumab 160 mg Q4W was better than placebo, certolizumab pegol pooled, golimumab 2 mg i.v., secukinumab 150 mg, adalimumab 40 mg, upadacitinib 15 mg, secukinumab 300 mg and ixekizumab 80 mg Q4W; and comparable to the remaining treatments in the network on PASI100 ( Fig. 4A ).

Bimekizumab 160 mg Q4W ranked 1st among 10 treatments (SUCRA = 0.85) for PASI90 and 2nd among 7 treatments (SUCRA = 0.79) for PASI100 ( Fig. 4B ). Bimekizumab 160 mg Q4W was better than placebo, ixekizumab 80 mg Q4W and upadacitinib 15 mg; and comparable to the remaining treatments in the network on PASI100 ( Fig. 4B ).

For MDA, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

Bimekizumab 160 mg Q4W ranked 1st among 13 treatments (SUCRA = 0.91) and was better than placebo [OR (95% CrI) 6.31 (4.61–8.20)], guselkumab 100 mg Q4W [2.06 (1.29–3.10)], guselkumab 100 mg Q8W [1.76 (1.09–2.69)], risankizumab 150 mg [1.99 (1.40–2.76)] and adalimumab 40 mg [1.41 (1.01–1.93)]; and comparable to the remaining treatments in the network ( Supplementary Fig. S2G , available at Rheumatology online).

Bimekizumab 160 mg Q4W ranked 1st among 11 treatments (SUCRA = 0.83) and was better than placebo [12.10 (5.31–28.19)] and tofacitinib 5 mg [6.81 (2.14–21.35)]; and comparable to the remaining treatments in the network ( Supplementary Fig. S2H , available at Rheumatology online).

The network for SAEs for a mixed population included 23 treatments and the best-fit model was an RE-unadjusted model (due to study populations and time point reporting heterogeneity). Bimekizumab 160 mg Q4W showed comparable safety to all treatments in the network ( Supplementary Fig. S2I , available at Rheumatology online).

The treatment landscape for PsA is complex, with numerous treatment options and limited direct comparative evidence. Bimekizumab 160 mg Q4W has recently been approved for the treatment of active PsA by the European Medicines Agency and recommended by NICE in the UK, and the published phase 3 results warrant comparison with existing therapies by NMA.

This NMA included 41 studies evaluating 22 b/tsDMARDs including the novel IL-17F and IL-17A inhibitor, bimekizumab. Overall, bimekizumab 160 mg Q4W ranked favourably among b/tsDMARDS for efficacy in joint, skin and disease activity outcomes in PsA across both b/tsDMARD-naïve and TNFi-experienced populations. The safety of bimekizumab 160 mg Q4W was similar to the other b/tsDMARDs.

The Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA) and EULAR provide evidence-based recommendations for the treatment of PsA [ 1 , 2 ]. To treat peripheral arthritis symptoms in PsA, efficacy across the classes of current b/tsDMARDs are considered similar by both GRAPPA and EULAR, in part due to a lack of data comparing licensed therapies in a head-to-head trial setting [ 1 , 2 ]. EULAR recommends the use of JAK inhibitors in the case of inadequate response, intolerance or when a bDMARD is not appropriate [ 1 ]. This recommendation was made when tofacitinib was the only available JAK inhibitor, but reflects current marketing authorizations for tofacitinib and upadacitinib which indicate use in patients with an inadequate response or prior intolerance to TNFis (USA) or bDMARDs (Europe) [ 37–40 ]. This NMA suggests that bimekizumab 160 mg Q4W may have an advantage over current treatments, including IL-23 inhibitors in b/tsDMARD naïve patients, and secukinumab 150 mg and tofacitinib in TNFi-experienced patients, as evidenced by our analysis of ACR50 for which the pairwise comparisons were significantly in favour of bimekizumab 160 mg Q4W.

For the treatment of skin symptoms in PsA, IL-23, IL-12/23 and IL-17A inhibitors are currently recommended due to their greater efficacy compared with TNFis [ 1 , 4 ]. GRAPPA also suggests considering efficacy demonstrated in direct comparative studies in PSO when selecting a treatment for PsA skin symptoms [ 2 ]. In our analysis of complete skin clearance as measured by PASI100, bimekizumab 160 mg Q4W demonstrated the likelihood of significantly greater efficacy than IL-17A, JAK inhibitors and TNFis in b/tsDMARD-naïve patients and IL-17A and JAK inhibitors in TNFi-experienced patients. Furthermore, the NMA results for skin clearance in PsA are in alignment with previous studies in PSO that demonstrated superiority of bimekizumab 320 mg Q4W or Q8W vs secukinumab, ustekinumab and adalimumab ( P  < 0.001) (note that the dosing of bimekizumab in PSO differs from that in PsA) [ 12 , 18 , 41 , 42 ].

There are similarities between our results and other recently published NMAs of b/tsDMARDs in PsA, although methodological heterogeneity across all NMAs makes comparisons challenging [ 34–36 , 43–45 ]. Among recent NMAs, the largest evaluated 21 treatments [ 34 ] and only four considered subgroups of b/tsDMARD-naïve and TNFi-experienced patients or those with inadequate response [ 35 , 36 , 43 , 45 ]. Furthermore, different or pooled levels of response were evaluated for ACR and PASI outcomes.

Previous NMAs also support IL-17, IL-12/23 and IL-23 inhibitors having greater efficacy for skin symptoms than TNFis [ 35 , 36 ]. In an overall PsA population, McInnes et al. demonstrated that secukinumab 300 mg, ixekizumab 80 mg Q4W, and ustekinumab 45 mg and 90 mg were likely more efficacious than TNFis for PASI90 [ 35 ]. In another NMA by Ruyssen-Witrand et al. , results suggested that ixekizumab 80 mg Q4W had significantly greater efficacy than adalimumab, certolizumab pegol pooled, and etanercept 25 mg twice weekly/50 mg once weekly for any PASI score (50%, 75%, 90% and 100% reduction) in bDMARD-naïve patients [ 36 ].

For joint outcomes, Mease et al. compared guselkumab Q4W and Q8W with other b/tsDMARDs in a network of 21 treatments in an overall PsA population for ACR50 [ 34 ]. Both guselkumab dosing schedules were better than abatacept and apremilast, but golimumab 2 mg i.v. had a higher likelihood of ACR50 response than guselkumab Q8W [ 34 ]. Despite MDA being assessed in clinical trials for bDMARD therapies and a treatment target in PsA [ 46 ], evidence for comparative efficacy for this outcome is limited. None of the most recent NMAs before this one included an analysis of MDA [ 34–36 ]. With regard to safety outcomes, previous NMAs evaluating SAEs also resulted in either no difference between b/tsDMARDs vs placebo or other b/tsDMARDs [ 34 , 36 , 44 , 45 ].

This study has a number of strengths. To our knowledge this NMA represents the most comprehensive and in-depth comparative efficacy analysis of approved treatments in PsA to date. The evidence was derived from a recent SLR, ensuring that new RCTs and updated results from previously published RCTs were included. It is also the first NMA to include the phase 3 BE COMPLETE and BE OPTIMAL trials of bimekizumab [ 19 , 20 ]. Our NMA used robust methods and accounted for variation in placebo response through network meta-regression in accordance with NICE DSU Technical Support Documents [ 31–33 ]. As an acknowledgement of the evolution of treatment advances, separate analyses of b/tsDMARD-naïve and TNFi-experienced subgroups were conducted with the intent to assist healthcare decision-making in different clinical settings. In addition, a panel of clinical experts were consulted from project inception and are authors of this paper, ensuring inclusion of a comprehensive set of clinically meaningful outcomes, including the composite, treat-to-target outcome of MDA.

Despite the robust evidence base and methodology, this NMA has limitations. Indirect treatment comparisons such as this NMA are not a substitute for head-to-head trials. There was heterogeneity in the endpoints and reporting in the included studies. Fewer studies reporting PASI outcomes resulted in smaller networks compared with the network of studies evaluating ACR response criteria. Not all trials reported outcomes at the same timepoint, thereby reducing the comparability of trial results, which has been transparently addressed by noting where week 24 data were used vs week 12, 14 or 16 data. The analyses for the TNFi-experienced population were limited by potential heterogeneity, especially in the analyses where fewer studies were included in the networks, as this group could include patients who had an inadequate response to TNFi or discontinued TNFi treatment due to other reasons (e.g. lost access). Also, in the analyses for the TNFi-experienced population, very low patient numbers for some treatments resulted in less statistical power. Additionally, the data included in the analysis were derived exclusively from RCTs, for which the study populations may not reflect a typical patient population seen in real-world practice. For example, trial results may be different in patients with oligoarthritis who are not well-represented in clinical trials.

Over the years covering our SLR, we acknowledge that patient populations and the PsA treatment landscape have evolved. After a thorough review of baseline patient characteristics, no significant differences were observed across the studies included in the NMA. To further mitigate uncertainty, baseline regression was used to actively correct for changes in the placebo rate over time ensuring a consistent and fair comparison across all included treatments. In addition, our analyses were conducted in separate b/tsDMARD-naïve and TNFi-experienced populations that reflect the evolving PsA patient population over time. Radiographic progression was not within the purview of this NMA because the NMA focused on a shorter timeframe than the 52-week duration typically recommended by the literature for investigating radiographic progression. Furthermore, there is existing literature on this topic, as exemplified by the work of Wang et al. in 2022 [ 47 ]. Nevertheless, the comprehensive and current evidence base, examination of multiple endpoints, and consistency with previous reported NMAs lend credence to our results.

Overall, the results of this NMA demonstrated the favourable relative efficacy and safety of bimekizumab 160 mg Q4W vs all approved treatments for PsA. Bimekizumab ranked high in terms of efficacy on joint, skin and MDA outcomes in both b/tsDMARD-naïve and TNFi-experienced patient populations, and showed comparable safety to other treatments. In the evolving PsA treatment landscape, bimekizumab 160 mg Q4W is a potentially beneficial treatment option for patients with PsA.

Supplementary material is available at Rheumatology online.

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

This study was funded in full by UCB Pharma.

Disclosure statement : P.J.M.: has received research grants from AbbVie, Amgen, BMS, Eli Lilly, Gilead, Janssen, Novartis, Pfizer, Sun Pharma and UCB Pharma; consultancy fees from AbbVie, Acelyrin, Aclaris, Amgen, BMS, Boehringer Ingelheim, Eli Lilly, Galapagos, Gilead, GSK, Janssen, Moonlake Pharma, Novartis, Pfizer, Sun Pharma and UCB Pharma; and speakers’ bureau for AbbVie, Amgen, Eli Lilly, Janssen, Novartis, Pfizer and UCB Pharma. L.C.C.: received grants/research support from AbbVie, Amgen, Celgene, Eli Lilly, Janssen, Novartis, Pfizer and UCB; worked as a paid consultant for AbbVie, Amgen, Bristol Myers Squibb, Celgene, Eli Lilly, Gilead, Galapagos, Janssen, Moonlake, Novartis, Pfizer and UCB; and has been paid as a speaker for AbbVie, Amgen, Biogen, Celgene, Eli Lilly, Galapagos, Gilead, GSK, Janssen, Medac, Novartis, Pfizer and UCB. D.D.G.: consultant and/or received grant support from Abbvie, Amgen, BMS, Celgene, Eli Lilly, Galapagos, Gilead, Janssen, Novartis, Pfizer and UCB. J.F.M.: consultant and/or investigator for AbbVie, Amgen, Biogen, BMS, Dermavant, Eli Lilly, Janssen, LEO Pharma, Novartis, Pfizer, Regeneron, Sanofi, Sun Pharma and UCB Pharma. P.N.: research grants, clinical trials and honoraria for advice and lectures on behalf of AbbVie, Boehringer Ingelheim, BMS, Eli Lilly, Galapagos/Gilead, GSK, Janssen, Novartis, Pfizer, Samsung, Sanofi and UCB Pharma. S.G. and V.L.-K.: employees of Cytel, Inc. which served as a consultant on the project. A.R.P., D.W. and V.T.: employees and stockholders of UCB Pharma.

The authors acknowledge Leah Wiltshire of Cytel for medical writing and editorial assistance based on the authors’ input and direction, Heather Edens (UCB Pharma, Smyrna, GA, USA) for publication coordination and Costello Medical for review management, which were funded by UCB Pharma. This analysis was funded by UCB Pharma in accordance with Good Publication Practice (GPP 2022) guidelines ( http://www.ismpp.org/gpp-2022 ). Data were previously presented at ISPOR-US 2023 (Boston, MA, USA, 7–10 May 2023).

Gossec L , Baraliakos X , Kerschbaumer A et al.  EULAR recommendations for the management of psoriatic arthritis with pharmacological therapies: 2019 update . Ann Rheum Dis 2020 ; 79 : 700 – 12 .

Google Scholar

Coates LC , Soriano ER , Corp N et al. ; GRAPPA Treatment Recommendations domain subcommittees . Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA): updated treatment recommendations for psoriatic arthritis 2021 . Nat Rev Rheumatol 2022 ; 18 : 465 – 79 .

Fitzgerald O , Ogdie A , Chandran V et al.  Psoriatic arthritis . Nat Rev Dis Primers 2021 ; 7 : 59 .

Coates LC , Soriano ER , Corp N et al.  Treatment recommendations domain subcommittees. Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA): updated treatment recommendations for psoriatic arthritis 2021 . Nat Rev Rheumatol 2022 ; 27 : 1 – 15 .

Najm A , Goodyear CS , McInnes IB , Siebert S. Phenotypic heterogeneity in psoriatic arthritis: towards tissue pathology-based therapy . Nat Rev Rheumatol 2023 ; 19 : 153 – 65 .

Ogdie A , Weiss P. The epidemiology of psoriatic arthritis . Rheum Dis Clin North Am 2015 ; 41 : 545 – 68 .

Alinaghi F , Calov M , Kristensen LE et al.  Prevalence of psoriatic arthritis in patients with psoriasis: a systematic review and meta-analysis of observational and clinical studies . J Am Acad Dermatol 2019 ; 80 : 251 – 65.e19 .

Singh JA , Guyatt G , Ogdie A et al.  Special Article: 2018 American college of rheumatology/national psoriasis foundation guideline for the treatment of psoriatic arthritis . Arthritis Rheumatol 2019 ; 71 : 5 – 32 .

Coates LC , Robinson DE , Orbai AM et al.  What influences patients' opinion of remission and low disease activity in psoriatic arthritis? Principal component analysis of an international study . Rheumatology (Oxford) 2021 ; 60 : 5292 – 9 .

Gondo G , Mosca M , Hong J et al.  Demographic and clinical factors associated with patient-reported remission in psoriatic arthritis . Dermatol Ther (Heidelb) 2022 ; 12 : 1885 – 95 .

Glatt S , Baeten D , Baker T et al.  Dual IL-17A and IL-17F neutralisation by bimekizumab in psoriatic arthritis: evidence from preclinical experiments and a randomised placebo-controlled clinical trial that IL-17F contributes to human chronic tissue inflammation . Ann Rheum Dis 2018 ; 77 : 523 – 32 .

UCB Pharma S.A . Bimzelx ® (bimekizumab): Summary of Product Characteristics. 2023 . https://www.ema.europa.eu/en/medicines/human/EPAR/bimzelx (26 June 2023, date last accessed).

Adams R , Maroof A , Baker T et al.  Bimekizumab, a novel humanized IgG1 antibody that neutralizes both IL-17A and IL-17F . Front Immunol 2020 ; 11 : 1894 .

Burns LA , Maroof A , Marshall D et al.  Presence, function, and regulation of IL-17F-expressing human CD4(+) T cells . Eur J Immunol 2020 ; 50 : 568 – 80 .

Kuestner R , Taft D , Haran A et al.  Identification of the IL-17 receptor related molecule IL-17RC as the receptor for IL-17F . J Immunol 2007 ; 179 : 5462 – 73 .

Johansen C , Usher PA , Kjellerup RB et al.  Characterization of the interleukin-17 isoforms and receptors in lesional psoriatic skin . Br J Dermatol 2009 ; 160 : 319 – 24 .

Shah M , Maroof A , Gikas P et al.  Dual neutralisation of IL-17F and IL-17A with bimekizumab blocks inflammation-driven osteogenic differentiation of human periosteal cells . RMD Open 2020 ; 6 : e001306 .

Reich K , Warren RB , Lebwohl M et al.  Bimekizumab versus Secukinumab in Plaque Psoriasis . New Engl J Med 2021 ; 385 : 142 – 52 .

McInnes IB , Asahina A , Coates LC et al.  Bimekizumab in patients with psoriatic arthritis, naive to biologic treatment: a randomised, double-blind, placebo-controlled, phase 3 trial (BE OPTIMAL) . Lancet 2023 ; 401 : 25 – 37 .

Merola JF , Landewe R , McInnes IB et al.  Bimekizumab in patients with active psoriatic arthritis and previous inadequate response or intolerance to tumour necrosis factor-alpha inhibitors: a randomised, double-blind, placebo-controlled, phase 3 trial (BE COMPLETE) . Lancet 2023 ; 401 : 38 – 48 .

Page MJ , McKenzie JE , Bossuyt PM et al.  The PRISMA 2020 statement: an updated guideline for reporting systematic reviews . BMJ 2021 ; 372 : n71 .

Higgins JP. Cochrane handbook for systematic reviews of interventions. Vol. 2. 2nd edn. Chichester, UK: John Wiley & Sons, 2019 .

National Institute for Health and Care Excellence . The guidelines manual: Process and methods [PMG6]. 2012 . https://www.nice.org.uk/process/pmg6/chapter/introduction (3 March 2023, date last accessed).

Booth AM , Wright KE , Outhwaite H. Centre for Reviews and Dissemination databases: value, content, and developments . Int J Technol Assess Health Care 2010 ; 26 : 470 – 2 .

Sterne JAC , Savovic J , Page MJ et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials . BMJ 2019 ; 366 : l4898 .

Daly C , Dias S , Welton N , Anwer S , Ades A. NICE Guidelines Technical Support Unit. Meta-Analysis: Guideline Methodology Document 1 (Version 1). 2021 . http://www.bristol.ac.uk/population-health-sciences/centres/cresyda/mpes/nice/guideline-methodology-documents-gmds/ (1 March 2023, date last accessed).

Dias S , Caldwell DM. Network meta-analysis explained . Archi Dis Childhood Fetal Neonatal Ed 2019 ; 104 : F8 – F12 .

Merola JF , Lockshin B , Mody EA. Switching biologics in the treatment of psoriatic arthritis . Semin Arthritis Rheum 2017 ; 47 : 29 – 37 .

Openbugs (website) 2014 . http://www.openbugs.net/w/FrontPage (6 April 2023, date last accessed).

Lunn D , Spiegelhalter D , Thomas A , Best N. The BUGS project: evolution, critique and future directions . Stat Med 2009 ; 28 : 3049 – 67 .

Dias S , Welton NJ , Sutton AJ , Ades AE. NICE DSU technical support document 2: a generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. London. 2016 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Dias S , Welton N , Sutton AJ , Caldwell DM , Lu G , Ades AE. NICE DSU technical support document 4: inconsistency in networks of evidence based on randomised controlled trials. 2011 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Dias S , Sutton AJ , Welton N , Ades AE. NICE DSU Technical support document 3: heterogeneity: subgroups, meta-regression, bias and bias-adjustment. 2011 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Mease PJ , McInnes IB , Tam LS et al.  Comparative effectiveness of guselkumab in psoriatic arthritis: results from systematic literature review and network meta-analysis . Rheumatology (Oxford) 2021 ; 60 : 2109 – 21 .

McInnes IB , Sawyer LM , Markus K et al.  Targeted systemic therapies for psoriatic arthritis: a systematic review and comparative synthesis of short-term articular, dermatological, enthesitis and dactylitis outcomes . RMD Open 2022 ; 8 : e002074 .

Ruyssen-Witrand A , Perry R , Watkins C et al.  Efficacy and safety of biologics in psoriatic arthritis: a systematic literature review and network meta-analysis . RMD Open 2020 ; 6 : e001117 .

Pfizer Inc . XELJANZ ® (tofacitinib): Summary of Product Characteristics. 2022 . https://www.ema.europa.eu/en/medicines/human/EPAR/xeljanz (4 May 2023, date last accessed).

Pfizer Inc . XELJANZ ® (tofacitinib): US Prescribing Information. 2022 . https://labeling.pfizer.com/ShowLabeling.aspx?id=959 (4 May 2023, date last accessed).

Abbvie Inc . RINVOQ ® (upadacitinib) extended-release tablets, for oral use: US Prescribing Information. 2023 . https://www.rxabbvie.com/pdf/rinvoq_pi.pdf (4 May 2023, date last accessed).

AbbVie Deutschland GmbH & Co . KG. RINVOQ ® (upadacitinib): Summary of Product Characteristics. 2023 . https://www.ema.europa.eu/en/medicines/human/EPAR/rinvoq (4 May 2023, date last accessed).

Warren RB , Blauvelt A , Bagel J et al.  Bimekizumab versus Adalimumab in Plaque Psoriasis . N Engl J Med 2021 ; 385 : 130 – 41 .

Reich K , Papp KA , Blauvelt A et al.  Bimekizumab versus ustekinumab for the treatment of moderate to severe plaque psoriasis (BE VIVID): efficacy and safety from a 52-week, multicentre, double-blind, active comparator and placebo controlled phase 3 trial . Lancet 2021 ; 397 : 487 – 98 .

Gladman DD , Orbai AM , Gomez-Reino J et al.  Network meta-analysis of tofacitinib, biologic disease-modifying antirheumatic drugs, and apremilast for the treatment of psoriatic arthritis . Curr Ther Res Clin Exp 2020 ; 93 : 100601 .

Qiu M , Xu Z , Gao W et al.  Fourteen small molecule and biological agents for psoriatic arthritis: a network meta-analysis of randomized controlled trials . Medicine (Baltimore) 2020 ; 99 : e21447 .

Kawalec P , Holko P , Mocko P , Pilc A. Comparative effectiveness of abatacept, apremilast, secukinumab and ustekinumab treatment of psoriatic arthritis: a systematic review and network meta-analysis . Rheumatol Int 2018 ; 38 : 189 – 201 .

Gossec L , McGonagle D , Korotaeva T et al.  Minimal disease activity as a treatment target in psoriatic arthritis: a review of the literature . J Rheumatol 2018 ; 45 : 6 – 13 .

Wang SH , Yu CL , Wang TY , Yang CH , Chi CC. Biologic disease-modifying antirheumatic drugs for preventing radiographic progression in psoriatic arthritis: a systematic review and network meta-analysis . Pharmaceutics 2022 ; 14 .

Supplementary data

Email alerts

Citing articles via.

• Rheumatology X/Twitter
• BSR Instagram
• BSR Facebook
• Recommend to Your Librarian

Affiliations

• Online ISSN 1462-0332
• Print ISSN 1462-0324
• Copyright © 2024 British Society for Rheumatology
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

The current role of contrast-enhanced ultrasound (ceus) in the diagnosis and staging of bladder cancer: a review of the available literature.

1. Introduction

2. imaging modalities for the diagnosis and staging of bladder cancer: current guidelines, 3. the rationale behind the use of ceus in bladder cancer.

• Wash-in time or rise time (RT), measured in seconds: the first point of signal intensity above the baseline;
• Peak intensity (PI), measured in dB: the maximum signal intensity in the selected ROIs;
• Time to peak (TTP), measured in seconds;
• Time from peak to one-half the signal intensity (TPH), measured in seconds: time from PI to one-half of PI;
• Mean transit time (MTT), measured in seconds: the time from the start of the rise of the TIC, past the PI and back to 50% of PI;
• Semi-descending slope (DS), measured in dB/second: the descending slope from maximum peak to half its intensity;
• Wash-out time: the length of time that precedes the moment at which the TIC returns to or nears zero after reaching the PI.

4. CEUS in Bladder Cancer: Review of the Available Literature

5. conclusions, author contributions, conflicts of interest.

• Antoni, S.; Ferlay, J.; Soerjomataram, I.; Znaor, A.; Jemal, A.; Bray, F. Bladder Cancer Incidence and Mortality: A Global Overview and Recent Trends. Eur. Urol. 2017 , 71 , 96–108. [ Google Scholar ] [ CrossRef ]
• Dobruch, J.; Daneshmand, S.; Fisch, M.; Lotan, Y.; Noon, A.P.; Resnick, M.J.; Shariat, S.F.; Zlotta, A.R.; Boorjian, S.A. Gender and Bladder Cancer: A Collaborative Review of Etiology, Biology, and Outcomes. Eur. Urol. 2016 , 69 , 300–310. [ Google Scholar ] [ CrossRef ]
• Matulewicz, R.S.; Steinberg, G.D. Non-muscle-invasive Bladder Cancer: Overview and Contemporary Treatment Landscape of Neoadjuvant Chemoablative Therapies. Rev. Urol. 2020 , 22 , 43–51. [ Google Scholar ] [ PubMed ] [ PubMed Central ]
• Raspollini, M.R.; Comperat, E.M.; Lopez-Beltran, A.; Montironi, R.; Cimadamore, A.; Tsuzuki, T.; Netto, G.J. News in the classification of WHO 2022 bladder tumors. Pathologica 2022 , 115 , 32–40. [ Google Scholar ] [ CrossRef ]
• Brierley, J.D.; Gospodarowicz, M.K.; Wittekind, C. TNM Classification of Malignant Tumours ; John Wiley & Sons: Hoboken, NJ, USA, 2017. [ Google Scholar ]
• Goonewardene, S.S.; Persad, R.; Motiwala, H.; Albala, D. Management of Non-Muscle Invasive Bladder Cancer ; Springer Nature: Berlin/Heidelberg, Germany, 2019. [ Google Scholar ]
• Porter, M.P.; Kerrigan, M.C.; Donato, B.M.; Ramsey, S.D. Patterns of use of systemic chemotherapy for Medicare beneficiaries with urothelial bladder cancer. Urol. Oncol. 2011 , 29 , 252–258. [ Google Scholar ] [ CrossRef ]
• Kurth, K.; Schellhammer, P.; Okajima, E.; Akdas, A.; Jakse, G.; Hen, H.; da Silva, F.C.; Fukushima, S.; Nagayama, T. Current methods of assessing and treating carcinoma in situ of the bladder with or without involvement of the prostatic urethra. Int. J. Urol. 1995 , 2 (Suppl. 2), 8–22. [ Google Scholar ] [ CrossRef ]
• Stav, K.; Leibovici, D.; Goren, E.; Livshitz, A.; I Siegel, Y.; Lindner, A.; Zisman, A. Adverse effects of cystoscopy and its impact on patients’ quality of life and sexual performance. Isr. Med. Assoc. J. 2004 , 6 , 474–478. [ Google Scholar ]
• Awiwi, M.O.; Vikram, R. Radiologic Diagnosis and Staging of Bladder Cancer: An Update. J. Comput. Assist. Tomogr. 2022 , 46 , 530–544. [ Google Scholar ] [ CrossRef ]
• Herment, A.; Guglielmi, J.P.; Dumee, P.; Peronneau, P.; Delouche, P. Limitations of ultrasound imaging and image restoration. Ultrasonics 1987 , 25 , 267–273. [ Google Scholar ] [ CrossRef ]
• Srivastava, S.; Van Roekel, D.; Wright, J.L.; Bruce, M.; Dighe, M. Contrast-enhanced ultrasound (CEUS) in the evaluation of bladder pathologies: Review. WFUMB Ultrasound Open. 2023 , 1 , 100019. [ Google Scholar ] [ CrossRef ]
• Drudi, F.M.; Di Leo, N.; Maghella, F.; Malpassini, F.; Iera, J.; Rubini, A.; Orsogna, N.; D’Ambrosio, F. CEUS in the study of bladder, method, administration and evaluation, a technical note. J. Ultrasound 2014 , 17 , 57–63. [ Google Scholar ] [ CrossRef ]
• Sidhu, P.S.; Cantisani, V.; Dietrich, C.F.; Gilja, O.H.; Saftoiu, A.; Bartels, E.; Bertolotto, M.; Calliada, F.; Clevert, D.-A.; Cosgrove, D.; et al. The EFSUMB Guidelines and Recommendations for the Clinical Practice of Contrast-Enhanced Ultrasound (CEUS) in Non-Hepatic Applications: Update 2017 (Long Version). Ultraschall Med. 2018 , 39 , e2–e44. [ Google Scholar ]
• Witjes, J.A.; Babjuk, M.; Gontero, P.; Jacqmin, D.; Karl, A.; Kruck, S.; Mariappan, P.; Redorta, J.P.; Stenzl, A.; van Velthoven, R.; et al. Clinical and cost effectiveness of hexaminolevulinate-guided blue-light cystoscopy: Evidence review and updated expert recommendations. Eur. Urol. 2014 , 66 , 863–871. [ Google Scholar ] [ CrossRef ]
• Stamatiou, K.; Papadoliopoulos, I.; Dahanis, S.; Zafiropoulos, G.; Polizois, K. The accuracy of ultrasonography in the diagnosis of superficial bladder tumors in patients presenting with hematuria. Ann. Saudi Med. 2009 , 29 , 134–137. [ Google Scholar ] [ CrossRef ]
• Abu-Yousef, M.M.; Narayana, A.S.; Franken, E.A., Jr.; Brown, R.C. Urinary bladder tumors studied by cystosonography. Part I: Detection. Radiology 1984 , 153 , 223–226. [ Google Scholar ] [ CrossRef ]
• Abu-Yousef, M.M.; Narayana, A.S.; Brown, R.C.; Franken, E.A., Jr. Urinary bladder tumors studied by cystosonography. Part II: Staging. Radiology 1984 , 153 , 227–231. [ Google Scholar ] [ CrossRef ]
• Wang, L.-J.; Wong, Y.-C.; Ng, K.-F.; Chuang, C.-K.; Lee, S.-Y.; Wan, Y.-L. Tumor characteristics of urothelial carcinoma on multidetector computerized tomography urography. J. Urol. 2010 , 183 , 2154–2160. [ Google Scholar ] [ CrossRef ]
• Galgano, S.J.; Porter, K.K.; Burgan, C.; Rais-Bahrami, S. The Role of Imaging in Bladder Cancer Diagnosis and Staging. Diagnostics 2020 , 10 , 703. [ Google Scholar ] [ CrossRef ]
• Del Giudice, F.; Pecoraro, M.; Vargas, H.A.; Cipollari, S.; De Berardinis, E.; Bicchetti, M.; Chung, B.I.; Catalano, C.; Narumi, Y.; Catto, J.W.F.; et al. Systematic Review and Meta-Analysis of Vesical Imaging-Reporting and Data System (VI-RADS) Inter-Observer Reliability: An Added Value for Muscle Invasive Bladder Cancer Detection. Cancers 2020 , 12 , 2994. [ Google Scholar ] [ CrossRef ]
• Gurram, S.; Muthigi, A.; Egan, J.; Stamatakis, L. Imaging in Localized Bladder Cancer: Can Current Diagnostic Modalities Provide Accurate Local Tumor Staging? Curr. Urol. Rep. 2019 , 20 , 82. [ Google Scholar ] [ CrossRef ]
• Zhang-Yin, J.; Girard, A.; Marchal, E.; Lebret, T.; Seban, M.H.; Uhl, M.; Bertaux, M. PET Imaging in Bladder Cancer: An Update and Future Direction. Pharmaceuticals 2023 , 16 , 606. [ Google Scholar ] [ CrossRef ]
• Revisiting the Safety Profile of Echocardiography Contrast Agents. Available online: https://www.acc.org/latest-in-cardiology/articles/2016/06/23/08/23/revisiting-the-safety-profile-of-echocardiography-contrast-agents (accessed on 1 June 2024).
• Jokisch, F.; Buchner, A.; Schulz, G.B.; Grimm, T.; Weinhold, P.; Pfitzinger, P.L.; Chaloupka, M.; Stief, C.G.; Schlenker, B.; Clevert, D.-A. Prospective evaluation of 4-D contrast-enhanced-ultrasound (CEUS) imaging in bladder tumors. Clin. Hemorheol. Microcirc. 2020 , 74 , 1–12. [ Google Scholar ] [ CrossRef ]
• Bracco. SonoVue. Available online: https://www.bracco.com/it-it/sonovue (accessed on 4 November 2023).
• Guo, S.; Xu, P.; Zhou, A.; Wang, G.; Chen, W.; Mei, J.; Xiao, F.; Liu, J.; Zhang, C. Contrast-Enhanced Ultrasound Differentiation between Low- and High-Grade Bladder Urothelial Carcinoma and Correlation with Tumor Microvessel Density. J. Ultrasound Med. 2017 , 36 , 2287–2297. [ Google Scholar ] [ CrossRef ]
• Hu, C.; Feng, Y.; Huang, P.; Jin, J. Adverse reactions after the use of SonoVue contrast agent: Characteristics and nursing care experience. Medicine 2019 , 98 , e17745. [ Google Scholar ] [ CrossRef ]
• Gettle, L.M.; Revzin, M.V. Innovations in Vascular Ultrasound. Radiol. Clin. N. Am. 2020 , 58 , 653–669. [ Google Scholar ] [ CrossRef ]
• Caruso, G.; Salvaggio, G.; Campisi, A.; Melloni, D.; Midiri, M.; Bertolotto, M.; Lagalla, R. Bladder tumor staging: Comparison of contrast-enhanced and gray-scale ultrasound. AJR Am. J. Roentgenol. 2010 , 194 , 151–156. [ Google Scholar ] [ CrossRef ]
• Liu, Q.; Gong, H.; Zhu, H.; Yuan, C.; Hu, B. Contrast-Enhanced Ultrasound in the Bladder: Critical Features to Differentiate Occupied Lesions. Comput. Math. Methods Med. 2021 , 2021 , 1047948. [ Google Scholar ] [ CrossRef ]
• Alexander, A.A.; Nazarian, L.N.; Capuzzi, D.M.; Rawool, N.M.; Kurtz, A.B.; Mastrangelo, M.J. Color Doppler sonographic detection of tumor flow in superficial melanoma metastases: Histologic correlation. J. Ultrasound Med. 1998 , 17 , 123–126. [ Google Scholar ] [ CrossRef ]
• Nicolau, C.; Bunesch, L.; Peri, L.; Salvador, R.; Corral, J.M.; Mallofre, C.; Sebastia, C. Accuracy of contrast-enhanced ultrasound in the detection of bladder cancer. Br. J. Radiol. 2011 , 84 , 1091–1099. [ Google Scholar ] [ CrossRef ]
• Srivastava, S.; Dhyani, M.; Dighe, M. Contrast-enhanced ultrasound (CEUS): Applications from the kidneys to the bladder. Abdom. Radiol. 2024 . [ Google Scholar ] [ CrossRef ]
• Li, W.; Li, C.; Gu, Z.; Ni, P.; Zhang, W.; Yang, F.; Yao, X.; Chen, Y. The value of contrast-enhanced ultrasound and magnetic resonance imaging in the diagnosis of bladder cancer. J. Cancer Res. Ther. 2021 , 17 , 1179–1185. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Drudi, F.M.; Cantisani, V.; Liberatore, M.; Iori, F.; Erturk, S.M.; Cristini, C.; Di Pierro, G.; D’ambrosio, U.; Malpassini, F.; De Felice, C.; et al. Role of low-mechanical index CEUS in the differentiation between low and high grade bladder carcinoma: A pilot study. Ultraschall Med. 2010 , 31 , 589–595. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Drudi, F.M.; Di Leo, N.; Malpassini, F.; Antonini, F.; Corongiu, E.; Iori, F. CEUS in the differentiation between low and high-grade bladder carcinoma. J. Ultrasound 2012 , 15 , 247–251. [ Google Scholar ] [ CrossRef ]
• Gupta, V.G.; Kumar, S.; Singh, S.K.; Lal, A.; Kakkar, N. Contrast enhanced ultrasound in urothelial carcinoma of urinary bladder: An underutilized staging and grading modality. Cent. Eur. J. Urol. 2016 , 69 , 360–365. [ Google Scholar ]
• Ge, X.; Lan, Z.-K.; Chen, J.; Zhu, S.-Y. Effectiveness of contrast-enhanced ultrasound for detecting the staging and grading of bladder cancer: A systematic review and meta-analysis. Med. Ultrason. 2021 , 23 , 29–35. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Huang, L.; Kong, Q.; Liu, Z.; Wang, J.; Kang, Z.; Zhu, Y. The Diagnostic Value of MR Imaging in Differentiating T Staging of Bladder Cancer: A Meta-Analysis. Radiology 2018 , 286 , 502–511. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Fu, W.; Liu, D.; Xiong, Y.; Liu, C.; Liu, J.; Yi, X.; Zhao, T. The comparison between contrast-enhanced ultrasound and contrast-enhanced magnetic resonance imaging in diagnosing bladder urothelial carcinoma. Int. Urol. Nephrol. 2022 , 55 , 1073–1079. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, Q.Y.; Tang, J.; He, E.H.; Li, Y.M.; Zhou, Y.; Zhang, X.; Chen, G. Clinical utility of three-dimensional contrast-enhanced ultrasound in the differentiation between noninvasive and invasive neoplasms of urinary bladder. Eur. J. Radiol. 2012 , 81 , 2936–2942. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, Q.; Tang, J.; He, E.; Li, Y.; Zhou, Y.; Wang, B. Differentiation between high- and low-grade urothelial carcinomas using contrast enhanced ultrasound. Oncotarget 2017 , 8 , 70883–70889. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ PubMed Central ]
• Smajerova, M.; Petrasova, H.; Little, J.; Ovesna, P.; Andrasina, T.; Valek, V.; Nemcova, E.; Miklosova, B. Contrast-enhanced ultrasonography in the evaluation of incidental focal liver lesions: A cost-effectiveness analysis. World J. Gastroenterol. 2016 , 22 , 8605–8614. [ Google Scholar ] [ CrossRef ]
• Quaia, E. Microbubble ultrasound contrast agents: An update. Eur. Radiol. 2007 , 17 , 1995–2008. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Thomsen, H.S.; Webb, J.A.W. Contrast Media: Safety Issues and ESUR Guidelines ; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2009. [ Google Scholar ]
• Lopes, R.I.; Nogueira, L.; Albertotti, C.J.; Takahashi, D.Y.; Lopes, R.N. Comparison of virtual cystoscopy and transabdominal ultrasonography with conventional cystoscopy for bladder tumor detection. J. Endourol. 2008 , 22 , 1725–1729. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Quaia, E.; Alaimo, V.; Baratella, E.; Pizzolato, R.; Cester, G.; Medeot, A.; Cova, M.A. Effect of observer experience in the differentiation between benign and malignant liver tumors after ultrasound contrast agent injection. J. Ultrasound Med. 2010 , 29 , 25–36. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, J.; Li, H.; Guan, L.; Lu, Y.; Zhan, W.; Dong, Y.; Gu, P.; Liu, J.; Cheng, W.; Na, Z.; et al. The value of preoperative sentinel lymph node contrast-enhanced ultrasound for breast cancer: A large, multicenter trial. BMC Cancer 2022 , 22 , 455. [ Google Scholar ] [ CrossRef ]
• Tufano, A.; Rosati, D.; Moriconi, M.; Santarelli, V.; Canale, V.; Salciccia, S.; Sciarra, A.; Franco, G.; Cantisani, V.; Di Pierro, G.B. Diagnostic Accuracy of Contrast-Enhanced Ultrasound (CEUS) in the Detection of Muscle-Invasive Bladder Cancer: A Systematic Review and Diagnostic Meta-Analysis. Curr. Oncol. 2024 , 31 , 818–827. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ PubMed Central ]

Click here to enlarge figure

Share and Cite

Santarelli, V.; Rosati, D.; Canale, V.; Salciccia, S.; Di Lascio, G.; Bevilacqua, G.; Tufano, A.; Sciarra, A.; Cantisani, V.; Franco, G.; et al. The Current Role of Contrast-Enhanced Ultrasound (CEUS) in the Diagnosis and Staging of Bladder Cancer: A Review of the Available Literature. Life 2024 , 14 , 857. https://doi.org/10.3390/life14070857

Santarelli V, Rosati D, Canale V, Salciccia S, Di Lascio G, Bevilacqua G, Tufano A, Sciarra A, Cantisani V, Franco G, et al. The Current Role of Contrast-Enhanced Ultrasound (CEUS) in the Diagnosis and Staging of Bladder Cancer: A Review of the Available Literature. Life . 2024; 14(7):857. https://doi.org/10.3390/life14070857

Santarelli, Valerio, Davide Rosati, Vittorio Canale, Stefano Salciccia, Giovanni Di Lascio, Giulio Bevilacqua, Antonio Tufano, Alessandro Sciarra, Vito Cantisani, Giorgio Franco, and et al. 2024. "The Current Role of Contrast-Enhanced Ultrasound (CEUS) in the Diagnosis and Staging of Bladder Cancer: A Review of the Available Literature" Life 14, no. 7: 857. https://doi.org/10.3390/life14070857

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals