Jump to navigation

Home

Cochrane Training

Chapter 3: defining the criteria for including studies and how they will be grouped for the synthesis.

Joanne E McKenzie, Sue E Brennan, Rebecca E Ryan, Hilary J Thomson, Renea V Johnston, James Thomas

Key Points:

  • The scope of a review is defined by the types of population (participants), types of interventions (and comparisons), and the types of outcomes that are of interest. The acronym PICO (population, interventions, comparators and outcomes) helps to serve as a reminder of these.
  • The population, intervention and comparison components of the question, with the additional specification of types of study that will be included, form the basis of the pre-specified eligibility criteria for the review. It is rare to use outcomes as eligibility criteria: studies should be included irrespective of whether they report outcome data, but may legitimately be excluded if they do not measure outcomes of interest, or if they explicitly aim to prevent a particular outcome.
  • Cochrane Reviews should include all outcomes that are likely to be meaningful and not include trivial outcomes. Critical and important outcomes should be limited in number and include adverse as well as beneficial outcomes.
  • Review authors should plan at the protocol stage how the different populations, interventions, outcomes and study designs within the scope of the review will be grouped for analysis.

Cite this chapter as: McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

3.1 Introduction

One of the features that distinguishes a systematic review from a narrative review is that systematic review authors should pre-specify criteria for including and excluding studies in the review (eligibility criteria, see MECIR Box 3.2.a ).

When developing the protocol, one of the first steps is to determine the elements of the review question (including the population, intervention(s), comparator(s) and outcomes, or PICO elements) and how the intervention, in the specified population, produces the expected outcomes (see Chapter 2, Section 2.5.1 and Chapter 17, Section 17.2.1 ). Eligibility criteria are based on the PICO elements of the review question plus a specification of the types of studies that have addressed these questions. The population, interventions and comparators in the review question usually translate directly into eligibility criteria for the review, though this is not always a straightforward process and requires a thoughtful approach, as this chapter shows. Outcomes usually are not part of the criteria for including studies, and a Cochrane Review would typically seek all sufficiently rigorous studies (most commonly randomized trials) of a particular comparison of interventions in a particular population of participants, irrespective of the outcomes measured or reported. It should be noted that some reviews do legitimately restrict eligibility to specific outcomes. For example, the same intervention may be studied in the same population for different purposes; or a review may specifically address the adverse effects of an intervention used for several conditions (see Chapter 19 ).

Eligibility criteria do not exist in isolation, but should be specified with the synthesis of the studies they describe in mind. This will involve making plans for how to group variants of the PICO elements for synthesis. This chapter describes the processes by which the structure of the synthesis can be mapped out at the beginning of the review, and the interplay between the review question, considerations for the analysis and their operationalization in terms of eligibility criteria. Decisions about which studies to include (and exclude), and how they will be combined in the review’s synthesis, should be documented and justified in the review protocol.

A distinction between three different stages in the review at which the PICO construct might be used is helpful for understanding the decisions that need to be made. In Chapter 2, Section 2.3 , we introduced the ideas of a review PICO (on which eligibility of studies is based), the PICO for each synthesis (defining the question that each specific synthesis aims to answer) and the PICO of the included studies (what was actually investigated in the included studies). In this chapter, we focus on the review PICO and the PICO for each synthesis as a basis for specifying which studies should be included in the review and planning its syntheses. These PICOs should relate clearly and directly to the questions or hypotheses that are posed when the review is formulated (see Chapter 2 ) and will involve specifying the population in question, and a set of comparisons between the intervention groups.

An integral part of the process of setting up the review is to specify which characteristics of the interventions (e.g. individual compounds of a drug), populations (e.g. acute and chronic conditions), outcomes (e.g. different depression measurement scales) and study designs, will be grouped together. Such decisions should be made independent of knowing which studies will be included and the methods of synthesis that will be used (e.g. meta-analysis). There may be a need to modify the comparisons and even add new ones at the review stage in light of the data that are collected. For example, important variations in the intervention may be discovered only after data are collected, or modifying the comparison may facilitate the possibility of synthesis when only one or few studies meet the comparison PICO. Planning for the latter scenario at the protocol stage may lead to less post-hoc decision making ( Chapter 2, Section 2.5.3 ) and, of course, any changes made during the conduct of the review should be recorded and documented in the final report.

3.2 Articulating the review and comparison PICO

3.2.1 defining types of participants: which people and populations.

The criteria for considering types of people included in studies in a review should be sufficiently broad to encompass the likely diversity of studies and the likely scenarios in which the interventions will be used, but sufficiently narrow to ensure that a meaningful answer can be obtained when studies are considered together; they should be specified in advance (see MECIR Box 3.2.a ). As discussed in Chapter 2, Section 2.3.1 , the degree of breadth will vary, depending on the question being asked and the analytical approach to be employed. A range of evidence may inform the choice of population characteristics to examine, including theoretical considerations, evidence from other interventions that have a similar mechanism of action, and in vitro or animal studies. Consideration should be given to whether the population characteristic is at the level of the participant (e.g. age, severity of disease) or the study (e.g. care setting, geographical location), since this has implications for grouping studies and for the method of synthesis ( Chapter 10, Section 10.11.5 ). It is often helpful to consider the types of people that are of interest in three steps.

MECIR Box 3.2.a Relevant expectations for conduct of intervention reviews

First, the diseases or conditions of interest should be defined using explicit criteria for establishing their presence (or absence). Criteria that will force the unnecessary exclusion of studies should be avoided. For example, diagnostic criteria that were developed more recently – which may be viewed as the current gold standard for diagnosing the condition of interest – will not have been used in earlier studies. Expensive or recent diagnostic tests may not be available in many countries or settings, and time-consuming tests may not be practical in routine healthcare settings.

Second, the broad population and setting of interest should be defined . This involves deciding whether a specific population group is within scope, determined by factors such as age, sex, race, educational status or the presence of a particular condition such as angina or shortness of breath. Interest may focus on a particular setting such as a community, hospital, nursing home, chronic care institution, or outpatient setting. Box 3.2.a outlines some factors to consider when developing population criteria.

Whichever criteria are used for defining the population and setting of interest, it is common to encounter studies that only partially overlap with the review’s population. For example, in a review focusing on children, a cut-point of less than 16 years might be desirable, but studies may be identified with participants aged from 12 to 18. Unless the study reports separate data from the eligible section of the population (in which case data from the eligible participants can be included in the review), review authors will need a strategy for dealing with these studies (see MECIR Box 3.2.a ). This will involve balancing concerns about reduced applicability by including participants who do not meet the eligibility criteria, against the loss of data when studies are excluded. Arbitrary rules (such as including a study if more than 80% of the participants are under 16) will not be practical if detailed information is not available from the study. A less stringent rule, such as ‘the majority of participants are under 16’ may be sufficient. Although there is a risk of review authors’ biases affecting post-hoc inclusion decisions (which is why many authors endeavour to pre-specify these rules), this may be outweighed by a common-sense strategy in which eligibility decisions keep faith with the objectives of the review rather than with arbitrary rules. Difficult decisions should be documented in the review, checked with the advisory group (if available, see Chapter 1 ), and sensitivity analyses can assess the impact of these decisions on the review’s findings (see Chapter 10, Section 10.14 and MECIR Box 3.2.b ).

Box 3.2.a Factors to consider when developing criteria for ‘Types of participants’

MECIR Box 3.2.b Relevant expectations for conduct of intervention reviews

Third, there should be consideration of whether there are population characteristics that might be expected to modify the size of the intervention effects (e.g. different severities of heart failure). Identifying subpopulations may be important for implementation of the intervention. If relevant subpopulations are identified, two courses of action are possible: limiting the scope of the review to exclude certain subpopulations; or maintaining the breadth of the review and addressing subpopulations in the analysis.

Restricting the review with respect to specific population characteristics or settings should be based on a sound rationale. It is important that Cochrane Reviews are globally relevant, so the rationale for the exclusion of studies based on population characteristics should be justified. For example, focusing a review of the effectiveness of mammographic screening on women between 40 and 50 years old may be justified based on biological plausibility, previously published systematic reviews and existing controversy. On the other hand, focusing a review on a particular subgroup of people on the basis of their age, sex or ethnicity simply because of personal interests, when there is no underlying biologic or sociological justification for doing so, should be avoided, as these reviews will be less useful to decision makers and readers of the review.

Maintaining the breadth of the review may be best when it is uncertain whether there are important differences in effects among various subgroups of people, since this allows investigation of these differences (see Chapter 10, Section 10.11.5 ). Review authors may combine the results from different subpopulations in the same synthesis, examining whether a given subdivision explains variation (heterogeneity) among the intervention effects. Alternatively, the results may be synthesized in separate comparisons representing different subpopulations. Splitting by subpopulation risks there being too few studies to yield a useful synthesis (see Table 3.2.a and Chapter 2, Section 2.3.2 ). Consideration needs to be given to the subgroup analysis method, particularly for population characteristics measured at the participant level (see Chapter 10 and Chapter 26 , Fisher et al 2017). All subgroup analyses should ideally be planned a priori and stated as a secondary objective in the protocol, and not driven by the availability of data.

In practice, it may be difficult to assign included studies to defined subpopulations because of missing information about the population characteristic, variability in how the population characteristic is measured across studies (e.g. variation in the method used to define the severity of heart failure), or because the study does not wholly fall within (or report the results separately by) the defined subpopulation. The latter issue mainly applies for participant characteristics but can also arise for settings or geographic locations where these vary within studies. Review authors should consider planning for these scenarios (see example reviews Hetrick et al 2012, Safi et al 2017; Table 3.2.b , column 3).

Table 3.2.a Examples of population attributes and characteristics

3.2.2 Defining interventions and how they will be grouped

In some reviews, predefining the intervention ( MECIR Box 3.2.c ) may be straightforward. For example, in a review of the effect of a given anticoagulant on deep vein thrombosis, the intervention can be defined precisely. A more complicated definition might be required for a multi-component intervention composed of dietary advice, training and support groups to reduce rates of obesity in a given population.

The inherent complexity present when defining an intervention often comes to light when considering how it is thought to achieve its intended effect and whether the effect is likely to differ when variants of the intervention are used. In the first example, the anticoagulant warfarin is thought to reduce blood clots by blocking an enzyme that depends on vitamin K to generate clotting factors. In the second, the behavioural intervention is thought to increase individuals’ self-efficacy in their ability to prepare healthy food. In both examples, we cannot assume that all forms of the intervention will work in the same way. When defining drug interventions, such as anticoagulants, factors such as the drug preparation, route of administration, dose, duration, and frequency should be considered. For multi-component interventions (such as interventions to reduce rates of obesity), the common or core features of the interventions must be defined, so that the review authors can clearly differentiate them from other interventions not included in the review.

MECIR Box 3.2.c Relevant expectations for conduct of intervention reviews

In general, it is useful to consider exactly what is delivered, who delivers it, how it is delivered, where it is delivered, when and how much is delivered, and whether the intervention can be adapted or tailored , and to consider this for each type of intervention included in the review (see the TIDieR checklist (Hoffmann et al 2014)). As argued in Chapter 17 , separating interventions into ‘simple’ and ‘complex’ is a false dichotomy; all interventions can be complex in some ways. The critical issue for review authors is to identify the most important factors to be considered in a specific review. Box 3.2.b outlines some factors to consider when developing broad criteria for the ‘Types of interventions’ (and comparisons).

Box 3.2.b Factors to consider when developing criteria for ‘Types of interventions’

Once interventions eligible for the review have been broadly defined, decisions should be made about how variants of the intervention will be handled in the synthesis. Differences in intervention characteristics across studies occur in all reviews. If these reflect minor differences in the form of the intervention used in practice (such as small differences in the duration or content of brief alcohol counselling interventions), then an overall synthesis can provide useful information for decision makers. Where differences in intervention characteristics are more substantial (such as delivery of brief alcohol counselling by nurses versus doctors), and are expected to have a substantial impact on the size of intervention effects, these differences should be examined in the synthesis. What constitutes an important difference requires judgement, but in general differences that alter decisions about how an intervention is implemented or whether the intervention is used or not are likely to be important. In such circumstances, review authors should consider specifying separate groups (or subgroups) to examine in their synthesis.

Clearly defined intervention groups serve two main purposes in the synthesis. First, the way in which interventions are grouped for synthesis (meta-analysis or other synthesis) is likely to influence review findings. Careful planning of intervention groups makes best use of the available data, avoids decisions that are influenced by study findings (which may introduce bias), and produces a review focused on questions relevant to decision makers. Second, the intervention groups specified in a protocol provide a standardized terminology for describing the interventions throughout the review, overcoming the varied descriptions used by study authors (e.g. where different labels are used for the same intervention, or similar labels used for different techniques) (Michie et al 2013). This standardization enables comparison and synthesis of information about intervention characteristics across studies (common characteristics and differences) and provides a consistent language for reporting that supports interpretation of review findings.

Table 3.2.b   outlines a process for planning intervention groups as a basis for/precursor to synthesis, and the decision points and considerations at each step. The table is intended to guide, rather than to be prescriptive and, although it is presented as a sequence of steps, the process is likely to be iterative, and some steps may be done concurrently or in a different sequence. The process aims to minimize data-driven approaches that can arise once review authors have knowledge of the findings of the included studies. It also includes principles for developing a flexible plan that maximizes the potential to synthesize in circumstances where there are few studies, many variants of an intervention, or where the variants are difficult to anticipate. In all stages, review authors should consider how to categorize studies whose reports contain insufficient detail.

Table 3.2.b A process for planning intervention groups for synthesis

3.2.3 Defining which comparisons will be made

When articulating the PICO for each synthesis, defining the intervention groups alone is not sufficient for complete specification of the planned syntheses. The next step is to define the comparisons that will be made between the intervention groups. Setting aside for a moment more complex analyses such as network meta-analyses, which can simultaneously compare many groups ( Chapter 11 ), standard meta-analysis ( Chapter 10 ) aims to draw conclusions about the comparative effects of two groups at a time (i.e. which of two intervention groups is more effective?). These comparisons form the basis for the syntheses that will be undertaken if data are available. Cochrane Reviews sometimes include one comparison, but most often include multiple comparisons. Three commonly identified types of comparisons include the following (Davey et al 2011).

  • newer generation antidepressants versus placebo (Hetrick et al 2012); and
  • vertebroplasty for osteoporotic vertebral compression fractures versus placebo (sham procedure) (Buchbinder et al 2018).
  • chemotherapy or targeted therapy plus best supportive care (BSC) versus BSC for palliative treatment of esophageal and gastroesophageal-junction carcinoma (Janmaat et al 2017); and
  • personalized care planning versus usual care for people with long-term conditions (Coulter et al 2015).
  • early (commenced at less than two weeks of age) versus late (two weeks of age or more) parenteral zinc supplementation in term and preterm infants (Taylor et al 2017);
  • high intensity versus low intensity physical activity or exercise in people with hip or knee osteoarthritis (Regnaux et al 2015);
  • multimedia education versus other education for consumers about prescribed and over the counter medications (Ciciriello et al 2013).

The first two types of comparisons aim to establish the effectiveness of an intervention, while the last aims to compare the effectiveness of two interventions. However, the distinction between the placebo and control is often arbitrary, since any differences in the care provided between trials with a control arm and those with a placebo arm may be unimportant , especially where ‘usual care’ is provided to both. Therefore, placebo and control groups may be determined to be similar enough to be combined for synthesis.

In reviews including multiple intervention groups, many comparisons are possible. In some of these reviews, authors seek to synthesize evidence on the comparative effectiveness of all their included interventions, including where there may be only indirect comparison of some interventions across the included studies ( Chapter 11, Section 11.2.1 ). However, in many reviews including multiple intervention groups, a limited subset of the possible comparisons will be selected. The chosen subset of comparisons should address the most important clinical and research questions. For example, if an established intervention (or dose of an intervention) is used in practice, then the synthesis would ideally compare novel or alternative interventions to this established intervention, and not, for example, to no intervention.

3.2.3.1 Dealing with co-interventions

Planning is needed for the special case where the same supplementary intervention is delivered to both the intervention and comparator groups. A supplementary intervention is an additional intervention delivered alongside the intervention of interest, such as massage in a review examining the effects of aromatherapy (i.e. aromatherapy plus massage versus massage alone). In many cases, the supplementary intervention will be unimportant and can be ignored. In other situations, the effect of the intervention of interest may differ according to whether participants receive the supplementary therapy. For example, the effect of aromatherapy among people who receive a massage may differ from the effect of the aromatherapy given alone. This will be the case if the intervention of interest interacts with the supplementary intervention leading to larger (synergistic) or smaller (dysynergistic/antagonistic) effects than the intervention of interest alone (Squires et al 2013). While qualitative interactions are rare (where the effect of the intervention is in the opposite direction when combined with the supplementary intervention), it is possible that there will be more variation in the intervention effects (heterogeneity) when supplementary interventions are involved, and it is important to plan for this. Approaches for dealing with this in the statistical synthesis may include fitting a random-effects meta-analysis model that encompasses heterogeneity ( Chapter 10, Section 10.10.4 ), or investigating whether the intervention effect is modified by the addition of the supplementary intervention through subgroup analysis ( Chapter 10, Section 10.11.2 ).

3.2.4 Selecting, prioritizing and grouping review outcomes

3.2.4.1 selecting review outcomes.

Broad outcome domains are decided at the time of setting up the review PICO (see Chapter 2 ). Once the broad domains are agreed, further specification is required to define the domains to facilitate reporting and synthesis (i.e. the PICO for comparison) (see Chapter 2, Section 2.3 ). The process for specifying and grouping outcomes largely parallels that used for specifying intervention groups.

Reporting of outcomes should rarely determine study eligibility for a review. In particular, studies should not be excluded because they do not report results of an outcome they may have measured, or provide ‘no usable data’ ( MECIR Box 3.2.d ). This is essential to avoid bias arising from selective reporting of findings by the study authors (see Chapter 13 ). However, in some circumstances, the measurement of certain outcomes may be a study eligibility criterion. This may be the case, for example, when the review addresses the potential for an intervention to prevent a particular outcome, or when the review addresses a specific purpose of an intervention that can be used in the same population for different purposes (such as hormone replacement therapy, or aspirin).

MECIR Box 3.2.d Relevant expectations for conduct of intervention reviews

In general, systematic reviews should aim to include outcomes that are likely to be meaningful to the intended users and recipients of the reviewed evidence. This may include clinicians, patients (consumers), the general public, administrators and policy makers. Outcomes may include survival (mortality), clinical events (e.g. strokes or myocardial infarction), behavioural outcomes (e.g. changes in diet, use of services), patient-reported outcomes (e.g. symptoms, quality of life), adverse events, burdens (e.g. demands on caregivers, frequency of tests, restrictions on lifestyle) and economic outcomes (e.g. cost and resource use). It is critical that outcomes used to assess adverse effects as well as outcomes used to assess beneficial effects are among those addressed by a review (see Chapter 19 ).

Outcomes that are trivial or meaningless to decision makers should not be included in Cochrane Reviews. Inclusion of outcomes that are of little or no importance risks overwhelming and potentially misleading readers. Interim or surrogate outcomes measures, such as laboratory results or radiologic results (e.g. loss of bone mineral content as a surrogate for fractures in hormone replacement therapy), while potentially helpful in explaining effects or determining intervention integrity (see Chapter 5, Section 5.3.4.1 ), can also be misleading since they may not predict clinically important outcomes accurately. Many interventions reduce the risk for a surrogate outcome but have no effect or have harmful effects on clinically relevant outcomes, and some interventions have no effect on surrogate measures but improve clinical outcomes.

Various sources can be used to develop a list of relevant outcomes, including input from consumers and advisory groups (see Chapter 2 ), the clinical experiences of the review authors, and evidence from the literature (including qualitative research about outcomes important to those affected (see Chapter 21 )). A further driver of outcome selection is consideration of outcomes used in related reviews. Harmonization of outcomes across reviews addressing related questions facilitates broader evidence synthesis questions being addressed through the use of Overviews of reviews (see Chapter V ).

Outcomes considered to be meaningful, and therefore addressed in a review, may not have been reported in the primary studies. For example, quality of life is an important outcome, perhaps the most important outcome, for people considering whether or not to use chemotherapy for advanced cancer, even if the available studies are found to report only survival (see Chapter 18 ). A further example arises with timing of the outcome measurement, where time points determined as clinically meaningful in a review are not measured in the primary studies. Including and discussing all important outcomes in a review will highlight gaps in the primary research and encourage researchers to address these gaps in future studies.

3.2.4.2 Prioritizing review outcomes

Once a full list of relevant outcomes has been compiled for the review, authors should prioritize the outcomes and select the outcomes of most relevance to the review question. The GRADE approach to assessing the certainty of evidence (see Chapter 14 ) suggests that review authors separate outcomes into those that are ‘critical’, ‘important’ and ‘not important’ for decision making.

The critical outcomes are the essential outcomes for decision making, and are those that would form the basis of a ‘Summary of findings’ table or other summary versions of the review, such as the Abstract or Plain Language Summary. ‘Summary of findings’ tables provide key information about the amount of evidence for important comparisons and outcomes, the quality of the evidence and the magnitude of effect (see Chapter 14, Section 14.1 ). There should be no more than seven outcomes included in a ‘Summary of findings’ table, and those outcomes that will be included in summaries should be specified at the protocol stage. They should generally not include surrogate or interim outcomes. They should not be chosen on the basis of any anticipated or observed magnitude of effect, or because they are likely to have been addressed in the studies to be reviewed. Box 3.2.c summarizes the principal factors to consider when selecting and prioritizing review outcomes.

Box 3.2.c Factors to consider when selecting and prioritizing review outcomes

3.2.4.3 Defining and grouping outcomes for synthesis

Table 3.2.c outlines a process for planning for the diversity in outcome measurement that may be encountered in the studies included in a review and which can complicate, and sometimes prevent, synthesis. Research has repeatedly documented inconsistency in the outcomes measured across trials in the same clinical areas (Harrison et al 2016, Williamson et al 2017). This inconsistency occurs across all aspects of outcome measurement, including the broad domains considered, the outcomes measured, the way these outcomes are labelled and defined, and the methods and timing of measurement. For example, a review of outcome measures used in 563 studies of interventions for dementia and mild cognitive impairment found that 321 unique measurement methods were used for 1278 assessments of cognitive outcomes (Harrison et al 2016). Initiatives like COMET ( Core Outcome Measures in Effectiveness Trials ) aim to encourage standardization of outcome measurement across trials (Williamson et al 2017), but these initiatives are comparatively new and review authors will inevitably encounter diversity in outcomes across studies.

The process begins by describing the scope of each outcome domain in sufficient detail to enable outcomes from included studies to be categorized ( Table 3.2.c Step 1). This step may be straightforward in areas for which core outcome sets (or equivalent systems) exist ( Table 3.2.c Step 2). The methods and timing of outcome measurement also need to be specified, giving consideration to how differences across studies will be handled ( Table 3.2.c Steps 3 and 4). Subsequent steps consider options for dealing with studies that report multiple measures within an outcome domain ( Table 3.2.c Step 5), planning how outcome domains will be used in synthesis ( Table 3.2.c Step 6), and building in contingencies to maximize potential to synthesize ( Table 3.2.c Step 7).

Table 3.2.c A process for planning outcome groups for synthesis

3.3 Determining which study designs to include

Some study designs are more appropriate than others for answering particular questions. Authors need to consider a priori what study designs are likely to provide reliable data with which to address the objectives of their review ( MECIR Box 3.3.a ). Sections 3.3.1 and 3.3.2 cover randomized and non-randomized designs for assessing treatment effects; Chapter 17, Section 17.2.5  discusses other study designs in the context of addressing intervention complexity.

MECIR Box 3.3.a Relevant expectations for conduct of intervention reviews

3.3.1 Including randomized trials

Because Cochrane Reviews address questions about the effects of health care, they focus primarily on randomized trials and randomized trials should be included if they are feasible for the interventions of interest ( MECIR Box 3.3.b ). Randomization is the only way to prevent systematic differences between baseline characteristics of participants in different intervention groups in terms of both known and unknown (or unmeasured) confounders (see Chapter 8 ), and claims about cause and effect can be based on their findings with far more confidence than almost any other type of study. For clinical interventions, deciding who receives an intervention and who does not is influenced by many factors, including prognostic factors. Empirical evidence suggests that, on average, non-randomized studies produce effect estimates that indicate more extreme benefits of the effects of health care than randomized trials. However, the extent, and even the direction, of the bias is difficult to predict. These issues are discussed at length in Chapter 24 , which provides guidance on when it might be appropriate to include non-randomized studies in a Cochrane Review.

Practical considerations also motivate the restriction of many Cochrane Reviews to randomized trials. In recent decades there has been considerable investment internationally in establishing infrastructure to index and identify randomized trials. Cochrane has contributed to these efforts, including building up and maintaining a database of randomized trials, developing search filters to aid their identification, working with MEDLINE to improve tagging and identification of randomized trials, and using machine learning and crowdsourcing to reduce author workload in identifying randomized trials ( Chapter 4, Section 4.6.6.2 ). The same scale of organizational investment has not (yet) been matched for the identification of other types of studies. Consequently, identifying and including other types of studies may require additional efforts to identify studies and to keep the review up to date, and might increase the risk that the result of the review will be influenced by publication bias. This issue and other bias-related issues that are important to consider when defining types of studies are discussed in detail in Chapter 7 and Chapter 13 .

Specific aspects of study design and conduct should be considered when defining eligibility criteria, even if the review is restricted to randomized trials. For example, whether cluster-randomized trials ( Chapter 23, Section 23.1 ) and crossover trials ( Chapter 23, Section 23.2 ) are eligible, as well as other criteria for eligibility such as use of a placebo comparison group, evaluation of outcomes blinded to allocation sequence, or a minimum period of follow-up. There will always be a trade-off between restrictive study design criteria (which might result in the inclusion of studies that are at low risk of bias, but very few in number) and more liberal design criteria (which might result in the inclusion of more studies, but at a higher risk of bias). Furthermore, excessively broad criteria might result in the inclusion of misleading evidence. If, for example, interest focuses on whether a therapy improves survival in patients with a chronic condition, it might be inappropriate to look at studies of very short duration, except to make explicit the point that they cannot address the question of interest.

MECIR Box 3.3.b Relevant expectations for conduct of intervention reviews

3.3.2 Including non-randomized studies

The decision of whether non-randomized studies (and what type) will be included is decided alongside the formulation of the review PICO. The main drivers that may lead to the inclusion of non-randomized studies include: (i) when randomized trials are unable to address the effects of the intervention on harm and long-term outcomes or in specific populations or settings; or (ii) for interventions that cannot be randomized (e.g. policy change introduced in a single or small number of jurisdictions) (see Chapter 24 ). Cochrane, in collaboration with others, has developed guidance for review authors to support their decision about when to look for and include non-randomized studies (Schünemann et al 2013).

Non-randomized designs have the commonality of not using randomization to allocate units to comparison groups, but their different design features mean that they are variable in their susceptibility to bias. Eligibility criteria should be based on explicit study design features, and not the study labels applied by the primary researchers (e.g. case-control, cohort), which are often used inconsistently (Reeves et al 2017; see Chapter 24 ).

When non-randomized studies are included, review authors should consider how the studies will be grouped and used in the synthesis. The Cochrane Non-randomized Studies Methods Group taxonomy of design features (see Chapter 24 ) may provide a basis for grouping together studies that are expected to have similar inferential strength and for providing a consistent language for describing the study design.

Once decisions have been made about grouping study designs, planning of how these will be used in the synthesis is required. Review authors need to decide whether it is useful to synthesize results from non-randomized studies and, if so, whether results from randomized trials and non-randomized studies should be included in the same synthesis (for the purpose of examining whether study design explains heterogeneity among the intervention effects), or whether the effects should be synthesized in separate comparisons (Valentine and Thompson 2013). Decisions should be made for each of the different types of non-randomized studies under consideration. Review authors might anticipate increased heterogeneity when non-randomized studies are synthesized, and adoption of a meta-analysis model that encompasses heterogeneity is wise (Valentine and Thompson 2013) (such as a random effects model, see Chapter 10, Section 10.10.4 ). For further discussion of non-randomized studies, see Chapter 24 .

3.4 Eligibility based on publication status and language

Chapter 4 contains detailed guidance on how to identify studies from a range of sources including, but not limited to, those in peer-reviewed journals. In general, a strategy to include studies reported in all types of publication will reduce bias ( Chapter 7 ). There would need to be a compelling argument for the exclusion of studies on the basis of their publication status ( MECIR Box 3.4.a ), including unpublished studies, partially published studies, and studies published in ‘grey’ literature sources. Given the additional challenge in obtaining unpublished studies, it is possible that any unpublished studies identified in a given review may be an unrepresentative subset of all the unpublished studies in existence. However, the bias this introduces is of less concern than the bias introduced by excluding all unpublished studies, given what is known about the impact of reporting biases (see Chapter 13 on bias due to missing studies, and Chapter 4, Section 4.3 for a more detailed discussion of searching for unpublished and grey literature).

Likewise, while searching for, and analysing, studies in any language can be extremely resource-intensive, review authors should consider carefully the implications for bias (and equity, see Chapter 16 ) if they restrict eligible studies to those published in one specific language (usually English). See Chapter 4, Section 4.4.5 , for further discussion of language and other restrictions while searching.

MECIR Box 3.4.a Relevant expectations for conduct of intervention reviews

3.5 Chapter information

Authors: Joanne E McKenzie, Sue E Brennan, Rebecca E Ryan, Hilary J Thomson, Renea V Johnston, James Thomas

Acknowledgements: This chapter builds on earlier versions of the Handbook . In particular, Version 5, Chapter 5 , edited by Denise O’Connor, Sally Green and Julian Higgins.

Funding: JEM is supported by an Australian National Health and Medical Research Council (NHMRC) Career Development Fellowship (1143429). SEB and RER’s positions are supported by the NHMRC Cochrane Collaboration Funding Program. HJT is funded by the UK Medical Research Council (MC_UU_12017-13 and MC_UU_12017-15) and Scottish Government Chief Scientist Office (SPHSU13 and SPHSU15). RVJ’s position is supported by the NHMRC Cochrane Collaboration Funding Program and Cabrini Institute. JT is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

3.6 References

Bailey JV, Murray E, Rait G, Mercer CH, Morris RW, Peacock R, Cassell J, Nazareth I. Interactive computer-based interventions for sexual health promotion. Cochrane Database of Systematic Reviews 2010; 9 : CD006483.

Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL, Thorlund K. Attention should be given to multiplicity issues in systematic reviews. Journal of Clinical Epidemiology 2008; 61 : 857–865.

Buchbinder R, Johnston RV, Rischin KJ, Homik J, Jones CA, Golmohammadi K, Kallmes DF. Percutaneous vertebroplasty for osteoporotic vertebral compression fracture. Cochrane Database of Systematic Reviews 2018; 4 : CD006349.

Caldwell DM, Welton NJ. Approaches for synthesising complex mental health interventions in meta-analysis. Evidence-Based Mental Health 2016; 19 : 16–21.

Chamberlain C, O’Mara-Eves A, Porter J, Coleman T, Perlen S, Thomas J, McKenzie J. Psychosocial interventions for supporting women to stop smoking in pregnancy. Cochrane Database of Systematic Reviews 2017; 2 : CD001055.

Ciciriello S, Johnston RV, Osborne RH, Wicks I, deKroo T, Clerehan R, O’Neill C, Buchbinder R. Multimedia educational interventions for consumers about prescribed and over-the-counter medications. Cochrane Database of Systematic Reviews 2013; 4 : CD008416.

Cochrane Consumers & Communication Group. Outcomes of Interest to the Cochrane Consumers & Communication Group: taxonomy. http://cccrg.cochrane.org/ .

COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative. COSMIN database of systematic reviews of outcome measurement instruments. https://database.cosmin.nl/ .

Coulter A, Entwistle VA, Eccles A, Ryan S, Shepperd S, Perera R. Personalised care planning for adults with chronic or long-term health conditions. Cochrane Database of Systematic Reviews 2015; 3 : CD010523.

Davey J, Turner RM, Clarke MJ, Higgins JPT. Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis. BMC Medical Research Methodology 2011; 11 : 160.

Desroches S, Lapointe A, Ratte S, Gravel K, Legare F, Turcotte S. Interventions to enhance adherence to dietary advice for preventing and managing chronic diseases in adults. Cochrane Database of Systematic Reviews 2013; 2 : CD008722.

Deyo RA, Dworkin SF, Amtmann D, Andersson G, Borenstein D, Carragee E, Carrino J, Chou R, Cook K, DeLitto A, Goertz C, Khalsa P, Loeser J, Mackey S, Panagis J, Rainville J, Tosteson T, Turk D, Von Korff M, Weiner DK. Report of the NIH Task Force on research standards for chronic low back pain. Journal of Pain 2014; 15 : 569–585.

Dodd S, Clarke M, Becker L, Mavergames C, Fish R, Williamson PR. A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery. Journal of Clinical Epidemiology 2018; 96 : 84–92.

Fisher DJ, Carpenter JR, Morris TP, Freeman SC, Tierney JF. Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? BMJ 2017; 356 : j573.

Fransen M, McConnell S, Harmer AR, Van der Esch M, Simic M, Bennell KL. Exercise for osteoarthritis of the knee. Cochrane Database of Systematic Reviews 2015; 1 : CD004376.

Guise JM, Chang C, Viswanathan M, Glick S, Treadwell J, Umscheid CA. Systematic reviews of complex multicomponent health care interventions. Report No. 14-EHC003-EF . Rockville, MD: Agency for Healthcare Research and Quality; 2014.

Harrison JK, Noel-Storr AH, Demeyere N, Reynish EL, Quinn TJ. Outcomes measures in a decade of dementia and mild cognitive impairment trials. Alzheimer’s Research and Therapy 2016; 8 : 48.

Hedges LV, Tipton E, Johnson M, C. Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods 2010; 1 : 39–65.

Hetrick SE, McKenzie JE, Cox GR, Simmons MB, Merry SN. Newer generation antidepressants for depressive disorders in children and adolescents. Cochrane Database of Systematic Reviews 2012; 11 : CD004851.

Higgins JPT, López-López JA, Becker BJ, Davies SR, Dawson S, Grimshaw JM, McGuinness LA, Moore THM, Rehfuess E, Thomas J, Caldwell DM. Synthesizing quantitative evidence in systematic reviews of complex health interventions. BMJ Global Health 2019; 4 : e000858.

Hoffmann T, Glasziou P, Barbour V, Macdonald H. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014; 1687 : 1-13.

Hollands GJ, Shemilt I, Marteau TM, Jebb SA, Lewis HB, Wei Y, Higgins JPT, Ogilvie D. Portion, package or tableware size for changing selection and consumption of food, alcohol and tobacco. Cochrane Database of Systematic Reviews 2015; 9 : CD011045.

Howe TE, Shea B, Dawson LJ, Downie F, Murray A, Ross C, Harbour RT, Caldwell LM, Creed G. Exercise for preventing and treating osteoporosis in postmenopausal women. Cochrane Database of Systematic Reviews 2011; 7 : CD000333.

ICHOM. The International Consortium for Health Outcomes Measurement 2018. http://www.ichom.org/ .

IPDAS. International Patient Decision Aid Standards Collaboration (IPDAS) standards. www.ipdas.ohri.ca .

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012; 6 : CD000259.

Janmaat VT, Steyerberg EW, van der Gaast A, Mathijssen RH, Bruno MJ, Peppelenbosch MP, Kuipers EJ, Spaander MC. Palliative chemotherapy and targeted therapies for esophageal and gastroesophageal junction cancer. Cochrane Database of Systematic Reviews 2017; 11 : CD004063.

Kendrick D, Kumar A, Carpenter H, Zijlstra GAR, Skelton DA, Cook JR, Stevens Z, Belcher CM, Haworth D, Gawler SJ, Gage H, Masud T, Bowling A, Pearl M, Morris RW, Iliffe S, Delbaere K. Exercise for reducing fear of falling in older people living in the community. Cochrane Database of Systematic Reviews 2014; 11 : CD009848.

Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews? A survey of the Co-ordinating Editors of Cochrane Review Groups. Trials 2013; 14 : 21.

Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods 2011; 2 : 61–76.

Lamb SE, Becker C, Gillespie LD, Smith JL, Finnegan S, Potter R, Pfeiffer K. Reporting of complex interventions in clinical trials: development of a taxonomy to classify and describe fall-prevention interventions. Trials 2011; 12 : 125.

Lewin S, Hendry M, Chandler J, Oxman AD, Michie S, Shepperd S, Reeves BC, Tugwell P, Hannes K, Rehfuess EA, Welch V, Mckenzie JE, Burford B, Petkovic J, Anderson LM, Harris J, Noyes J. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Medical Research Methodology 2017; 17 : 76.

López-López JA, Page MJ, Lipsey MW, Higgins JPT. Dealing with multiplicity of effect sizes in systematic reviews and meta-analyses. Research Synthesis Methods 2018; 9 : 336–351.

Mavridis D, Salanti G. A practical introduction to multivariate meta-analysis. Statistical Methods in Medical Research 2013; 22 : 133–158.

Michie S, van Stralen M, West R. The Behaviour Change Wheel: a new method for characterising and designing behaviour change interventions. Implementation Science 2011; 6 : 42.

Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, Eccles MP, Cane J, Wood CE. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Annals of Behavioral Medicine 2013; 46 : 81–95.

Moraes VY, Lenza M, Tamaoki MJ, Faloppa F, Belloti JC. Platelet-rich therapies for musculoskeletal soft tissue injuries. Cochrane Database of Systematic Reviews 2014; 4 : CD010071.

O'Neill J, Tabish H, Welch V, Petticrew M, Pottie K, Clarke M, Evans T, Pardo Pardo J, Waters E, White H, Tugwell P. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. Journal of Clinical Epidemiology 2014; 67 : 56–64.

Pompoli A, Furukawa TA, Imai H, Tajika A, Efthimiou O, Salanti G. Psychological therapies for panic disorder with or without agoraphobia in adults: a network meta-analysis. Cochrane Database of Systematic Reviews 2016; 4 : CD011004.

Pompoli A, Furukawa TA, Efthimiou O, Imai H, Tajika A, Salanti G. Dismantling cognitive-behaviour therapy for panic disorder: a systematic review and component network meta-analysis. Psychological Medicine 2018; 48 : 1–9.

Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series-paper 5: a checklist for classifying studies evaluating the effects on health interventions – a taxonomy without labels. Journal of Clinical Epidemiology 2017; 89 : 30–42.

Regnaux J-P, Lefevre-Colau M-M, Trinquart L, Nguyen C, Boutron I, Brosseau L, Ravaud P. High-intensity versus low-intensity physical activity or exercise in people with hip or knee osteoarthritis. Cochrane Database of Systematic Reviews 2015; 10 : CD010203.

Richards SH, Anderson L, Jenkinson CE, Whalley B, Rees K, Davies P, Bennett P, Liu Z, West R, Thompson DR, Taylor RS. Psychological interventions for coronary heart disease. Cochrane Database of Systematic Reviews 2017; 4 : CD002902.

Safi S, Korang SK, Nielsen EE, Sethi NJ, Feinberg J, Gluud C, Jakobsen JC. Beta-blockers for heart failure. Cochrane Database of Systematic Reviews 2017; 12 : CD012897.

Santesso N, Carrasco-Labra A, Brignardello-Petersen R. Hip protectors for preventing hip fractures in older people. Cochrane Database of Systematic Reviews 2014; 3 : CD001255.

Shepherd E, Gomersall JC, Tieu J, Han S, Crowther CA, Middleton P. Combined diet and exercise interventions for preventing gestational diabetes mellitus. Cochrane Database of Systematic Reviews 2017; 11 : CD010443.

Squires J, Valentine J, Grimshaw J. Systematic reviews of complex interventions: framing the review question. Journal of Clinical Epidemiology 2013; 66 : 1215–1222.

Stacey D, Légaré F, Lewis K, Barry MJ, Bennett CL, Eden KB, Holmes-Rovner M, Llewellyn-Thomas H, Lyddiatt A, Thomson R, Trevena L. Decision aids for people facing health treatment or screening decisions. Cochrane Database of Systematic Reviews 2017; 4 : CD001431.

Stroke Unit Trialists Collaboration. Organised inpatient (stroke unit) care for stroke. Cochrane Database of Systematic Reviews 2013; 9 : CD000197.

Taylor AJ, Jones LJ, Osborn DA. Zinc supplementation of parenteral nutrition in newborn infants. Cochrane Database of Systematic Reviews 2017; 2 : CD012561.

Valentine JC, Thompson SG. Issues relating to confounding and meta-analysis when including non-randomized studies in systematic reviews on the effects of interventions. Research Synthesis Methods 2013; 4 : 26–35.

Vaona A, Banzi R, Kwag KH, Rigon G, Cereda D, Pecoraro V, Tramacere I, Moja L. E-learning for health professionals. Cochrane Database of Systematic Reviews 2018; 1 : CD011736.

Verheyden GSAF, Weerdesteyn V, Pickering RM, Kunkel D, Lennon S, Geurts ACH, Ashburn A. Interventions for preventing falls in people after stroke. Cochrane Database of Systematic Reviews 2013; 5 : CD008728.

Weisz JR, Kuppens S, Ng MY, Eckshtain D, Ugueto AM, Vaughn-Coaxum R, Jensen-Doss A, Hawley KM, Krumholz Marchette LS, Chu BC, Weersing VR, Fordwood SR. What five decades of research tells us about the effects of youth psychological therapy: a multilevel meta-analysis and implications for science and practice. American Psychologist 2017; 72 : 79–117.

Welch V, Petkovic J, Simeon R, Presseau J, Gagnon D, Hossain A, Pardo Pardo J, Pottie K, Rader T, Sokolovski A, Yoganathan M, Tugwell P, DesMeules M. Interactive social media interventions for health behaviour change, health outcomes, and health equity in the adult population. Cochrane Database of Systematic Reviews 2018; 2 : CD012932.

Welton NJ, Caldwell DM, Adamopoulos E, Vedhara K. Mixed treatment comparison meta-analysis of complex interventions: psychological interventions in coronary heart disease. American Journal of Epidemiology 2009; 169 : 1158–1165.

Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, Clarke M, Gargon E, Gorst S, Harman N, Kirkham JJ, McNair A, Prinsen CAC, Schmitt J, Terwee CB, Young B. The COMET Handbook: version 1.0. Trials 2017; 18 : 280.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

  • Research article
  • Open access
  • Published: 21 November 2014

PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews

  • Abigail M Methley 1 ,
  • Stephen Campbell 1 , 4 ,
  • Carolyn Chew-Graham 2 ,
  • Rosalind McNally 3 &
  • Sudeh Cheraghi-Sohi 1 , 4  

BMC Health Services Research volume  14 , Article number:  579 ( 2014 ) Cite this article

215k Accesses

1036 Citations

59 Altmetric

Metrics details

Qualitative systematic reviews are increasing in popularity in evidence based health care. Difficulties have been reported in conducting literature searches of qualitative research using the PICO search tool. An alternative search tool, entitled SPIDER, was recently developed for more effective searching of qualitative research, but remained untested beyond its development team.

In this article we tested the ‘SPIDER’ search tool in a systematic narrative review of qualitative literature investigating the health care experiences of people with Multiple Sclerosis. Identical search terms were combined into the PICO or SPIDER search tool and compared across Ovid MEDLINE, Ovid EMBASE and EBSCO CINAHL Plus databases. In addition, we added to this method by comparing initial SPIDER and PICO tools to a modified version of PICO with added qualitative search terms (PICOS).

Results showed a greater number of hits from the PICO searches, in comparison to the SPIDER searches, with greater sensitivity. SPIDER searches showed greatest specificity for every database. The modified PICO demonstrated equal or higher sensitivity than SPIDER searches, and equal or lower specificity than SPIDER searches. The modified PICO demonstrated lower sensitivity and greater specificity than PICO searches.

Conclusions

The recommendations for practice are therefore to use the PICO tool for a fully comprehensive search but the PICOS tool where time and resources are limited. Based on these limited findings the SPIDER tool would not be recommended due to the risk of not identifying relevant papers, but has potential due to its greater specificity.

Peer Review reports

Systematic reviews are a crucial method, underpinning evidence based practice and informing health care decisions [ 1 ],[ 2 ]. Traditionally systematic reviews are completed using an objective and primarily quantitative approach [ 3 ] whereby a comprehensive search is conducted, attempting to identify all relevant articles which are then integrated and assimilated through statistical analysis. The comprehensiveness of the search process has been viewed as a key factor in preventing bias and providing a true representation of available research [ 4 ]. Current research investigating the process of quantitative systematic reviews therefore focuses on methods for ensuring the most comprehensive and bias free searches possible [ 5 ]. Because of the time and resources required to complete a systematic and comprehensive search, efforts have been made to investigate the sensitivity of searches, and thus lessen the amount of time spent reviewing irrelevant articles with no benefit [ 6 ].

However, conducting comprehensive searches also forms the bedrock of qualitative or narrative reviews, now commonly referred to as qualitative evidence syntheses [ 7 ]. Qualitative evidence syntheses are now acknowledged as a necessary and valuable type of information to answer health services research questions [ 8 ]. However, difficulties in completing a sensitive yet comprehensive search of qualitative literature have been previously noted [ 9 ]-[ 11 ] including: poor indexing and use of key words of qualitative studies, the common use of titles that lack the keywords describing the article, and unstructured abstracts.

When devising a search strategy, a search tool is used as an organising framework to list terms by the main concepts in the search question, especially in teams where it is not possible to have an experienced information specialist as a member of the review team. The PICO tool focuses on the Population, Intervention, Comparison and Outcomes of a (usually quantitative) article. It is commonly used to identify components of clinical evidence for systematic reviews in evidence based medicine and is endorsed by the Cochrane Collaboration [ 2 ]. Due to its target literature base several of these search terms such as “control group” and “intervention” are not relevant to qualitative research which traditionally does not utilise control groups or interventions, and therefore may not appropriately locate qualitative research. However, these terms may become more relevant in the future as more trials and interventions incorporate qualitative research [ 12 ].

As the PICO tool does not currently accommodate terms relating to qualitative research or specific qualitative designs, it has often been modified in practice to “PICOS” where the “S” refers to the Study design [ 4 ], thus limiting the number of irrelevant articles.

Cooke et al. also addressed this issue of relevance by developing a new search tool entitled “SPIDER” (sample, phenomenon of interest, design, evaluation, research type), designed specifically to identify relevant qualitative and mixed-method studies [ 9 ]. The key features and differences of the SPIDER and PICO search tools are shown in Table  1 . The addition of the “design” and “research type” categories to the SPIDER tool was intended to further increase the ability of this tool to identify qualitative articles, whilst removing irrelevant PICO categories such as the “comparison” group [ 9 ].

Cooke et al. recommended that the SPIDER tool was tested further in qualitative literature searches [ 9 ]. Although it has been used previously in a scoping review to investigate gaps in an evidence base on community participation in rural health care [ 13 ], SPIDER has not yet been tested and evaluated in a qualitative systematic narrative review context. The authors of this article recently completed a systematic review of the qualitative research investigating experiences of health care services for people with Multiple Sclerosis [ 14 ]. On embarking on this review topic we faced many of the difficulties commonly discussed in identifying qualitative literature on a given topic, and identified SPIDER as a potential way of overcoming some of these difficulties. Therefore, the aim of this article was to test SPIDER by broadly replicating the work of Cooke et al. [ 9 ], specifically by comparing the two approaches: 1) the traditional PICO method of searching electronic databases with 2) the newly devised SPIDER tool, developed for qualitative and mixed-method research. In addition we wished to build and expand on the work of Cooke et al. [ 9 ] and so our third aim was to compare PICO and SPIDER to a modified PICO with qualitative study designs (PICOS, see Table  1 by investigating specificity and sensitivity across 3 major databases.

Inclusion and exclusion criteria

Studies eligible for inclusion were those that qualitatively investigated patients’ experiences, views, attitudes to and perceptions of health care services for Multiple Sclerosis. No date restriction was imposed on searches as this was an original review. Qualitative research, for this purpose, was defined by the Cochrane qualitative methods group [ 7 ] as using both a qualitative data collection method and qualitative analysis. Quantitative and mixed method studies were therefore excluded.

We define experience as “ Patients’ reports of how care was organised and delivered to meet their needs p.301” [ 15 ]. Patients’ reports could refer to either experience of health care services delivery and organisation overall or their experiences of care by specific health care personnel. We included studies that investigated adults (aged 18 years old and older) with a diagnosis of Multiple Sclerosis, who had experience of utilising health care services at any time point. There were no restrictions on subtype of Multiple Sclerosis, gender, ethnicity or frequency of use of health care. Health care in this sense referred to routine clinical care (either state funded or privately funded) not trial protocols or interventions. Excluded studies included studies that focussed on self-management and studies that investigated quality of life.

Because of the focus on Multiple Sclerosis, studies were excluded if they used a mixed sample of various conditions (e.g. studies reported a mixed sample of people with neurological conditions) or if they used a sample of mixed respondents (i.e. people with Multiple Sclerosis and their carers) where results of patients with Multiple Sclerosis could not be clearly separated. If an article had a section or subtheme on health care services but this was not the main research area of the article, then that article was included; however only data from the relevant subtheme were extracted and included in the findings. Additional exclusion criteria were articles that only described carer or health care professional experiences not patient experiences. Conference abstracts, editorials and commentaries were not included.

Search strategy

For this systematic search we developed a detailed search strategy in collaboration with a specialist librarian and information specialist. This search strategy was tailored to the three largest medical and nursing databases (Ovid MEDLINE, Ovid EMBASE, and EBSCO CINAHL Plus) as in Cooke et al.’s study [ 9 ] and search terms used a mixture of medical subject headings and keywords. To investigate the benefit of the SPIDER,PICO and PICOS tools we used identical search terms but combined them in different ways as shown in Tables  2 , 3 and 4 below.

One reviewer judged titles and abstracts against the inclusion criteria. If a title and abstract met the inclusion criteria then full text copies of all articles were retrieved for further investigation. Two authors reviewed these full text articles independently for relevance to the search aim (i.e. patients/service users with multiple sclerosis, experiences of health care services and qualitative research). Any disagreements were resolved via discussion. Data from included studies were extracted by both reviewers independently to ensure accuracy and then stored on a Microsoft Excel spread sheet. No ethical approval was required for this study.

All searches spanned from database inception until 12th October 2013. As in Cooke et al. [ 9 ], we reviewed our findings based on two metrics; the number of hits generated and of these, the number relevant to the search aim (see Table  5 ).

Number of articles generated

As found in Cooke et al. [ 9 ], PICO created a much greater number of hits compared to SPIDER. A total of 23758 hits were generated using PICO, 448 hits were generated using PICOS and 239 hits were generated using SPIDER. Overall, the average reduction of hits (% across all three databases) was 98.58% for SPIDER vs. PICO, 97.94% for PICO vs. PICOS and 68.64% for PICOS vs. SPIDER. The time spent screening hits for relevant articles equated to weeks for the PICO hits and hours for the PICOS and SPIDER hits.

Proportion of relevant articles

Articles which met the inclusion criteria after full text review are displayed in Table  6 [ 16 ]-[ 33 ]. Examination of the titles and abstracts of the identified articles resulted in the obtainment of 18 full text articles relevant at full text, across all databases and search tools.

For the PICO tool in CINAHL Plus, 5.78% of hits were deemed relevant after the title and abstract stage (78 articles/1350 articles), and 14/78 articles (17.95%) were confirmed to meet the inclusion criteria after full text review. For the PICO tool in MEDLINE, 0.42% of hits were deemed relevant after the title and abstract stage (34 articles/8158 articles) and 12/34 (35.29%) articles were confirmed to meet the inclusion criteria after full text review. For the PICO tool in EMBASE, 0.25% hits were deemed relevant after the title and abstract stage (35 articles/ 14250 articles) and 14/35(40%) articles were confirmed to meet the inclusion criteria after full text review.

For the PICOS tool in CINAHL Plus, 38.36% of articles were relevant after the title and abstract stage (56 articles/146 articles) and 12/56 (21.43%) were confirmed to meet the inclusion criteria after full text review. For the PICOS tool in MEDLINE 14.16% of articles were relevant after the title and abstract stage (16 articles/ 113 articles) and 6/16 (37.5%) were confirmed to meet the inclusion criteria after full text review. For the PICOS tool in EMBASE 7.94% of articles were deemed relevant after the title and abstract stage (15 articles/189 articles) and 7/15 (46.67%) were confirmed to meet the inclusion criteria after full text review.

SPIDER tool

For the SPIDER tool in CINAHL Plus 38.36% of articles were relevant after the title and abstract stage (56 articles/146 articles) and 12/56 (21.43%) were confirmed to meet the inclusion criteria after full text review. For the SPIDER tool in MEDLINE, 36.81% hits were deemed relevant at the title stage (14 articles/38 articles) and 5/14 articles (35.71%) were confirmed to meet the inclusion criteria after full text review. For the SPIDER tool in EMBASE, 16.36% were relevant at the title stage (9 articles/55 articles) and 3/9 (33.33%) were confirmed to meet the inclusion criteria after full text review.

Sensitivity and specificity

The SPIDER tool identified 13 relevant articles out of 239 articles across all three databases (5.43%) compared to PICOS which identified 13 articles out of 448 articles (2.90%) and PICO which identified 18 articles out of 23758 articles (0.076%). Of the 18 relevant articles identified by the PICO tool, 66.66% came from both MEDLINE and CINAHL Plus (12 articles each), and 72.22% came from EMBASE (13 articles). Of the 13 relevant articles identified by the PICOS tool 46.15% came from MEDLINE (6 articles), 53.84% came from EMBASE (7 articles) and 92.31% came from CINAHL Plus (12 articles). Of the 13 relevant articles identified by SPIDER, 38.46% came from MEDLINE (5 articles) and 23.07% came from EMBASE (3 articles) and 92.30% came from CINAHL Plus (12 articles) Table  7 .

Different articles were found across different tools and databases (as shown in Table  6 ). All three databases were checked for all articles. One article was available in CINAHL Plus but not identified by any of the tools [ 17 ]. Two papers were identified in all databases through all search tools. Five papers were identified in MEDLINE through all search tools, three identified in EMBASE through all search tools and 12 identified in CINAHL through all search tools. Five papers were identified solely in CINAHL Plus, with one of these papers only identified using the PICO search method. One paper was identified by all search tools in EMBASE but not identified by any in MEDLINE. No new studies were identified using the SPIDER or PICOS tools alone in any database.

In this article we addressed the aim of replicating a comparison between the SPIDER, PICOS and PICO search tools. As previously described in Cooke et al. [ 9 ], the SPIDER tool produced a greatly reduced number of initial hits to sift through, however in this study it missed five studies that were identified through the PICO method. This may be partly be explained by the nature of the research question prompting the search. As this study included subthemes of studies whose focus differed from the initial research question (i.e. only a smaller section of the paper related to health care) then it’s possible that these studies were picked up by a broader search but not the highly specific SPIDER search. Other authors researching the process of qualitative literature reviews have previously commented that there appears to be a decision to be made about the benefits of comprehensiveness of findings versus the accuracy of the studies identified [ 11 ]. Given the common nature of using sub-sections of papers for systematic reviews then our findings suggest that comprehensiveness needs to be the key for this type of search.

The PICOS tool was more specific than the PICO tool, but did not identify any additional relevant hits to the SPIDER tool, suggesting it is of approximately equal sensitivity. PICOS identified the same number of papers as the SPIDER tool and both demonstrated a substantially lower number of hits generated than a regular PICO search. The SPIDER tool showed the greatest specificity due the small number of hits generated. This may mean that review teams with very limited resources or time, and who are not aiming for a totally comprehensive search (i.e. in the case of scoping studies), would benefit from using the SPIDER tool. This might be applicable particularly to studies such as qualitative syntheses, where the research aim is theoretical saturation, not a comprehensive search [ 34 ]. In addition, articles written to influence policy often require swift publication, providing another area in which either SPIDER or PICOS might improve current practice.

The issue of time was also related to the number of relevant articles identified per database. Whilst EMBASE generated nearly twice as many hits as MEDLINE, only one additional paper was found. The PICO tool identified all articles, suggesting that where time is not a factor, it might be of more benefit to use this tool, as SPIDER demonstrated lower sensitivity, did not identify any new articles and identified fewer relevant articles than PICO.

Our findings indicate that it is worthwhile testing a chosen search tool across various databases as they produce different results; i.e. CINHAL Plus identified papers not identified in MEDLINE or EMBASE databases. It is therefore important for future research to investigate the potential of the SPIDER vs. PICOS and PICO tools as a base for the recommended comprehensive searching process, by investigating the contribution of the SPIDER and PICOS tools at every stage from the initial search hits, to the final included relevant articles.

As CINAHL is a database dedicated to nursing and allied health research, it was expected that it would produce a greater number of relevant articles than more medically focussed databases [ 10 ], as nursing and allied areas have traditionally been at the forefront of qualitative investigations into Multiple Sclerosis.

SPIDER proved to be a tool designed to formulate search terms easily, as it naturally fits the crucial elements of the search question. However, even though some qualitative keywords are necessary to identify qualitative studies, including the words “ qualitative research ” AND the name of the type of research e.g. “ grounded theory ” might be too restrictive, particularly given the poor use of the qualitative index term, and might partially explain the fewer studies identified by SPIDER in comparison to PICO. Studies not identified by the SPIDER model in MEDLINE and EMBASE databases did not use keywords such as “ qualitative ”, but some described qualitative methods, such as “ phenomenological-hermeneutic ” [ 16 ] or “ interview(s) ” [ 20 ],[ 23 ].

In all PICO searches for MEDLINE and EMBASE the word “ qualitative ” combined with the phrase “ multiple sclerosis ” identified many quantitative studies reporting brain scan assessments that were wholly unrelated to the search aim. This was because the word “ qualitative ” in this context referred to using a qualitative method to provide information about the quality of the scan and any potential flaws [ 35 ]. This caused a problem with specificity, resulting in thousands of inappropriate hits as there was no way to exclude studies with the word “ qualitative ” unless all articles clearly utilised and indexed qualitative research methods in the title, abstract and keywords.

Many studies were excluded at the full text stage on the basis that the samples were mixed: being comprised of either various neurological conditions or mixed groups of people i.e. patients and carers/patients and health care professionals and so forth. Without clearer titles and abstracts, and potentially an indexing phrase that indicates mixed samples, there is no way of avoiding this issue. Excluding the phrases “ caregivers ” or “ health care professionals ” would have excluded any studies that used these phrases (for example in the introduction or implication for future research sections) and therefore it is difficult to see how this could be prevented. A strength and limitation of our study is that whilst it details a real world example of evidence searching, it only addresses one topic. Further research should test these search tools against a wider variety of narrative review and meta-synthesis topics.

SPIDER greatly reduced the initial number of articles identified on a given search due to increased specificity, however because of lower sensitivity omitted many relevant papers. The PICOS tool resulted in an overall more sensitive search, but still demonstrated poor specificity on this topic. Further investigations of the specificity and sensitivity of SPIDER and PICOS on varied topics will be of benefit to research teams with limited time and resources or articles necessary to impact on policy or change current practice. However, where comprehensiveness is a key factor we suggest that the PICO tool should be used preferentially. Part of the lower identification rate for SPIDER (in comparison to PICO) was poor labelling and use of qualitative keywords in indexing studies. As both individual research submissions and journal/database indexers improve, or standardise, the indexing of qualitative studies, it is likely that the relevance of the SPIDER tool will increase. The recommendation for current practice therefore is to use the PICO tool across a variety of databases. In this article we have shown that SPIDER is relevant for those researchers completing systematic narrative reviews of qualitative literature but not as effective as PICO. Future research should investigate the use of SPIDER and PICOS across varied databases.

Authors’ information

Caroly Chew-Graham is part-funded by the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care West Midlands.

Stephens KR: Systematic reviews: the heart of evidence-based practice. Am Assoc Crit Care Nurs Clin Issu. 2001, 12 (4): 529-538.

Google Scholar  

Higgins JPT, Green S: Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration. 2013

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olson R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 3-10.1186/1471-2288-6-35.

Article   Google Scholar  

Centre for reviews and dissemination: Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. 2006, University of York, York

Helmer D, Savoie I, Green C, Kazanjian A: Evidence-based practice: extending the search to find material for the systematic review. Bull Med Libr Assoc. 2001, 89 (4): 346-352.

CAS   PubMed   PubMed Central   Google Scholar  

Stevinson C, Lawlor DA: Searching multiple databases for systematic reviews: added value or diminishing returns?. Complement Ther Med. 2004, 12: 228-232. 10.1016/j.ctim.2004.09.003.

Article   CAS   PubMed   Google Scholar  

Noyes J: Never mind the qualitative feel the depth! The evolving role of qualitative research in Cochrane intervention reviews. J Res Nurs. 2010, 15: 525-534. 10.1177/1744987110381696.

Noyes J, Popay J, Pearson A, Hannes K, Booth A: Chapter 20: qualitative research and cochrane reviews. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. 2008, 20.1-20.18

Cooke A, Smith D, Booth A: Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012, 22: 1435-1443. 10.1177/1049732312452938.

Article   PubMed   Google Scholar  

Evans D: Database searches for qualitative research. J Med Library Assoc. 2002, 9: 290-293.

Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, Jones DR, Dixon-Woods M: Finding qualitative research: an evaluation of search strategies. BMC Med Res Methodol. 2004, 4: 5-10.1186/1471-2288-4-5.

Article   PubMed   PubMed Central   Google Scholar  

Lewin S, Glenton C, Oxman AD: Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ 2009, 339(b3496). http://www.bmj.com/content/339/bmj.b3496 ..

Kenny A, Hyett N, Sawtell J, Dickson-Swift V, Farmer J, O’Meara P: Community participation in rural health: a scoping review. BMC Health Serv Res. 2013, 13: 64-10.1186/1472-6963-13-64.

Methley AM, Chew-Graham C, Campbell S, Cheraghi-Sohi S: Experiences of UK health care services for people with Multiple Sclerosis: a systematic narrative review. Health Expect 2014, epub ahead of print..

Sinfield P, Baker R, Camosso- Stefinovic J, Colman AM, Tarrant C, Mellon JK, Steward W, Kockelbergh R, Agarwal S: Men’s and carer’s experiences of care for prostate cancer: a narrative literature review. Health Expect. 2009, 12: 301-312. 10.1111/j.1369-7625.2009.00546.x.

Lohne V, Aasgaard T, Caspari S, Slettebo A, Naden D: The lonely battle for dignity: individuals struggling with multiple sclerosis. Nurs Ethics. 2010, 17 (3): 301-311. 10.1177/0969733010361439.

Mackereth PA, Booth K, Hillier VF, Caress A: Reflexology and progressive muscle relaxation training for people with multiple sclerosis: a crossover trial. Complement Ther Clin Pract. 2008, 15: 14-21. 10.1016/j.ctcp.2008.07.002.

Isaksson AK, Ahlström G: Managing chronic sorrow: experiences of patients with multiple sclerosis. J Neurosci Nurs. 2008, 40 (3): 180-191. 10.1097/01376517-200806000-00009.

Edwards RG, Barlow JH, Turner AP: Experiences of diagnosis and treatment among people with multiple sclerosis. J Eval Clin Pract. 2008, 14 (3): 460-464. 10.1111/j.1365-2753.2007.00902.x.

Barker-Collo S, Cartwright C, Read J: Into the unknown: The experiences of individuals living with multiple sclerosis. J Neurosci Nurs. 2006, 38 (6): 435-441. 10.1097/01376517-200612000-00008.

Isaksson AK, Ahlström G: From symptom to diagnosis: illness experiences of multiple sclerosis patients. J Neurosci Nurs. 2006, 38 (4): 229-237. 10.1097/01376517-200608000-00005.

Miller CE, Jezewski MA: Relapsing MS patients’ experiences with glatiramer acetate treatment: a phenomenologic study. J Neurosci Nurs. 2006, 38 (1): 37-41. 10.1097/01376517-200602000-00008.

Johnson J: On receiving the diagnosis of multiple sclerosis: managing the transition. Mult Scler. 2003, 9 (1): 82-88. 10.1191/1352458503ms856oa.

Miller CE, Jezewski MA: A phenomenologic assessment of relapsing MS patients’ experiences during treatment with Interferon Beta-1(*). J Neurosci Nurs. 2001, 33 (5): 240-244. 10.1097/01376517-200110000-00004.

Miller CM: The lived experience of relapsing multiple sclerosis: a phenomenological study. J Neurosci Nurs. 1997, 29 (5): 294-304. 10.1097/01376517-199710000-00003.

Aars H, Bruusgaard D: Chronic disease and sexuality: an interview study on sexual dysfunction in patients with multiple sclerosis. Tidsskrift Den Norske Laegeforening. 1989, 109 (32): 3352-3354.

CAS   Google Scholar  

Rintell DJ, Frankel D, Minden SL, Glanz BI: Patients’ perspectives on quality of mental health care for people with MS. Gen Hosp Psychiatry. 2012, 34 (6): 604-610. 10.1016/j.genhosppsych.2012.04.001.

Laidlaw A, Henwood S: Patients with multiple sclerosis: their experiences and perceptions of the MRI investigation. J Diagn Radiogr Imaging. 2003, 5 (1): 19-25. 10.1017/S146047280300004X.

Koopman W, Schweitzer A: The journey to multiple sclerosis: a qualitative study. J Neurosci Nurs. 1999, 31 (1): 17-26. 10.1097/01376517-199902000-00003.

Hansen AK, Krogh H, Bangsgaard L, Aabling S: Facing the diagnosis. Sygeplejersken. 2008, 4: 52-56.

Loveland CA: The experiences of African americans and euro-americans with multiple sclerosis. Sex Disabil. 1999, 17 (1): 19-35. 10.1023/A:1021499628918.

Moriya R, Suzuki S: A qualitative study relating to the experiences of people with MS: differences by disease severity. Br J Neurosci Nurs. 2011, 7 (4): 593-600. 10.12968/bjnn.2011.7.4.593.

Classen S, Lou JQ: Exploring rehabilitation and wellness needs of people with MS living in South Florida: a pilot study. Intern J MS Care. 2004, 1: 26-31. 10.7224/1537-2073-6.1.26.

Booth A: Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative research? In Qualitative Evidence-Based Practice Conference. Coventry; 2001. [ http://www.leeds.ac.uk/educol/documents/00001724.htm ]

Chalavi S, Simmons A, Dijkstra H, Barker GJ, Reindeers AATS: Quantitative and qualitative assessment of structural magnetic resonance imaging data in a two-center study. BMC Med Imaging. 2012, 12: 27-10.1186/1471-2342-12-27.

Download references

Acknowledgements

This study was funded by a School for Primary Care Research PhD studentship from the National Institute of Health Research. Support in selecting search terms is acknowledged from Olivia Walsby, Academic Engagement Librarian at the University of Manchester. We are grateful to Professor Peter Bower for his comments on the protocol.

Author information

Authors and affiliations.

University of Manchester, Centre for Primary Care, Williamson Building, Oxford Road, Manchester, M13 9PL, UK

Abigail M Methley, Stephen Campbell & Sudeh Cheraghi-Sohi

Institute of Primary Care and Health Sciences, Keele University, Keele, UK

Carolyn Chew-Graham

Central Manchester Hospitals Site, Manchester Mental Health and Social Care Trust, Research and Innovation 3rd Floor, Rawnsley Building, Hathersage Road, Manchester, M13 9WL, UK

Rosalind McNally

NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre, Institute of Population Health, The University of Manchester, Manchester, M13 9WL, UK

Stephen Campbell & Sudeh Cheraghi-Sohi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Abigail M Methley .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

AM designed the study, conducted all searches, appraised all potential studies and wrote and revised the draft manuscript and subsequent manuscripts. SC made significant contributions to the conception and design of the study, assisted with the presentation of findings and assisted with drafting and revising the manuscript. CCG and RM made significant contributions to the conception and design of the study, assisted with the presentation of findings and assisted with drafting and revising the manuscript. SCS conceived and designed the study, assisted with searches, appraised relevant studies and assisted with drafting and revising the manuscript. All authors read and approved the final manuscript.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/4.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Methley, A.M., Campbell, S., Chew-Graham, C. et al. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 14 , 579 (2014). https://doi.org/10.1186/s12913-014-0579-0

Download citation

Received : 03 February 2014

Accepted : 03 November 2014

Published : 21 November 2014

DOI : https://doi.org/10.1186/s12913-014-0579-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Health care
  • Users’ experiences
  • Multiple sclerosis (MS)
  • Research evaluation
  • Qualitative
  • Systematic reviews

BMC Health Services Research

ISSN: 1472-6963

systematic review pico tools

The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review

Affiliations.

  • 1 University Library, University of Southern Denmark, Campusvej 55, 5230 Odense M, Denmark.
  • 2 Department of Design and Communication, University of Southern Denmark, Universitetsparken 1, DK-6000 Kolding, Denmark.
  • PMID: 30271283
  • PMCID: PMC6148624
  • DOI: 10.5195/jmla.2018.345

Objective: This review aimed to determine if the use of the patient, intervention, comparison, outcome (PICO) model as a search strategy tool affects the quality of a literature search.

Methods: A comprehensive literature search was conducted in PubMed, Embase, CINAHL, PsycINFO, Cochrane Library, Web of Science, Library and Information Science Abstracts (LISA), Scopus, and the National Library of Medicine (NLM) catalog up until January 9, 2017. Reference lists were scrutinized, and citation searches were performed on the included studies. The primary outcome was the quality of literature searches and the secondary outcome was time spent on the literature search when the PICO model was used as a search strategy tool, compared to the use of another conceptualizing tool or unguided searching.

Results: A total of 2,163 records were identified, and after removal of duplicates and initial screening, 22 full-text articles were assessed. Of these, 19 studies were excluded and 3 studies were included, data were extracted, risk of bias was assessed, and a qualitative analysis was conducted. The included studies compared PICO to the PIC truncation or links to related articles in PubMed, PICOS, and sample, phenomenon of interest, design, evaluation, research type (SPIDER). One study compared PICO to unguided searching. Due to differences in intervention, no quantitative analysis was performed.

Conclusions: Only few studies exist that assess the effect of the PICO model vis-a-vis other available models or even vis-a-vis the use of no model. Before implications for current practice can be drawn, well-designed studies are needed to evaluate the role of the tool used to devise a search strategy.

Publication types

  • Systematic Review
  • Evidence-Based Medicine / standards*
  • Information Storage and Retrieval / methods*
  • Periodicals as Topic / statistics & numerical data*
  • Quality Assurance, Health Care / statistics & numerical data*
  • United States

PolyU Library

  • The Hong Kong Polytechnic University
  • Guides & Tutorials

Systematic Search for Systematic Review

  • Formulate Research Question Using PICO
  • Introduction
  • Find Systematic Reviews (SR)
  • Databases Selection for Conducting SR
  • Step 1. Set Preferences in EndNote
  • Step 2. Create Groups in EndNote
  • Step 3. Export Search Results from Databases to EndNote
  • Step 4. Add Name of Database to References
  • Step 5. Remove Duplicate Records
  • Step 6. Share References with Teammates
  • Step 7. Find Full Text Articles
  • [Optional] Export References to Excel

Worksheets for Documenting & Reporting Search Process

Here are some resources for you to document and report your search process in a systematic review. 

  • Workbook for documenting systematic search
  • PRISMA Flow Diagram A flow diagram to depict the flow of information through the different phases of a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions.

Understanding SR

  • What are systematic reviews? (Cochrane)
  • Intro to Systematic Reviews & Meta-Analyses
  • Using PICO to formulate a search question   (CEBM)
  • Turning search terms into a search   (CEBM)
  • Turning your search strategy into results: PubMed demonstration   (CEBM)

Understanding study design

  • What is a randomised trial?
  • Epidemiology Study Types: Randomized Control Trial
  • Epidemiology Study Types: Cohort and Case-Control
  • Cohort, Case-Control, Meta-Analysis, Cross-sectional Study Designs & Definition 

Copyright Disclaimer

Creative Commons License

Except where otherwise noted, the content of this guide is licensed under a  CC BY-NC 4.0 License .

A systematic review aims to answer a specific research (clinical) question. A well-formulated question will guide many aspects of the review process, including determining eligibility criteria, searching for studies, collecting data from included studies, and presenting findings ( Cochrane Handbook , Sec. 5.1.1).

To define a  researchable  question, the most commonly used structure is  PICO , which specifies the type of P atient or P opulation, type of I nterventions (and C omparisons if there is any), and the type of O utcomes that are of interest. 

The table below gives an example on how a research question is framed using the PICO structure. You may also use the PICO components to write the objective and title of your review, and later to structure your inclusion and exclusion criteria for study selection. This ensures that the whole review process is guided by your research question. 

Type of Question and Study Design

While formulating your research question, it's also important to consider the  type of question  you are asking because this will affect the type of studies (or study design ) to be included in your review.

Each type of question defines its type of studies in order to provide the best evidence. For example, to answer a therapeutic question, you need to include as many Randomized Controlled Trials (RCTs) as possible, because RCTs are considered to have the highest  level of evidence  (least bias) for solving a therapeutic problem. 

The table below suggests the best designs for specific type of question. The Level of Evidence pyramid, which is widely adopted in the medical research area, shows a hierarchy of the quality of medical research evidence in different type of studies ( Level of Evidence (2011), Oxford Centre for Evidence-based Medicine, CEBM ).

Usually, the study design of a research work will be clearly indicated either in its title or abstract, especially for RCT. Some databases also allow to search or refine results to one or a few study designs, which helps you locate as many as possible the relevant studies. If you are not sure the study design of a research work, refer to this brief guide for spotting study designs  (by CEBM).

Learn to Build a Good Clinical Question

Learn to build a good clinical question  from this  EBP Tutorial: Module 1:  "Introduction to Evidence-Based Practice"

It is provided by Duke University and University of North Carolina at Chapel Hill, USA.

PICO Framework and the Question Statement The above named section  in the Library guide:  Evidence-Based Practice in Health , provided by the University of Canberra Library, explains the PICO framework with examples and in various question types.

Documenting Your Search Process

Systematic review requires a detailed and structured reporting of the search strategy and selection criteria used in the review. Therefore we strongly advise you to document your search process from the very beginning. You may use this workbook  to help you with the documentation.

The documentation should include:

  • Research concepts in PICO structure and research question ,
  • Type of studies you intend to include, and
  • Inclusion and exclusion criteria in PICO structure

and the whole search process, including:

  • Databases searched (hosting platforms) , including journals and other sources covered in handsearching
  • Date of search
  • Search strategy , including keywords and subject headings used, the combination of searches (usually copy-paste from database search page)
  • Filters used in initial search or refine results, including year coverage, type of studies, age, etc.
  • Number of results retrieved after each search and refinement in each database
  • Total number of results from all databases searched
  • Duplicates identified from all results
  • Number of results with full text

Eventually, you will need to include the information above when you start writing your review. A highly recommended structure for reporting the search process is the PRISMA Flow Diagram . You may also use PRISMA Flow Diagram Generator to generate a diagram in a different format (based on your input). 

  • << Previous: Find Systematic Reviews (SR)
  • Next: Databases Selection for Conducting SR >>
  • Last Updated: Apr 22, 2024 10:53 AM
  • URL: https://libguides.lb.polyu.edu.hk/syst_review

Synthesize evidence at rapid speed

Intelligently accelerate your research and innovation with PICO Portal.

systematic review pico tools

PICO Portal's AI-powered Systematic Literature Review platform and managed services let you focus on developing high quality research, generating evidence you can trust - faster than ever.

systematic review pico tools

Supercharged Systematic Reviews

Relevant articles first.

Review the most relevant articles first to conduct the screening process efficiently leveraging our ML-prediction model.

Fast and Accurate

Reduce review timeline by 40-60% compared to traditional tools with the most accurate prediction and deduplication technology on the market.

Fully Customizable Reviews

One size does not fit all. Set up your workflow to suit your needs. Follow single or dual review, create tags, customize keywords, complete PRISMA, and more.

Professional Services

Staff your reviews how and when you need to with our certified methodologists and flexible mananged services

PICO Portal in numbers

Institutions

Institutions Trust Us

systematic review pico tools

Using AI prediction we identified 95% to 99% of all of eligible citations by screening fewer than 10% of our initial citations. We're all used to the standard of two people having to independently screen every citation. Do we need to rethink that?

systematic review pico tools

We had 15 committee members at 15 different institutions that all needed to help with this review. We identified 5,172 references and the AI showed likely includes before likely excludes. That helped us get the literature out to the community sooner.

systematic review pico tools

The value for us was the tagging function. It was just a matter of adding a couple of clicks of the mouse and we were able to watch the distribution of literature across this wide range and approach it as evidence ecology. We looked at the full range of research designs from RCTs to qualitative research.

Tiered pricing for teams of all sizes.

Get the power of an enterprise solution without the high price tag. Ready to partner with us for your upcoming research?

Available on All Devices

PICO Portal's SLR platform supports screening using your desktop browser such as Chrome, Safari or Firefox, your tablet browser and your mobile device. The mobile support can help screen your citations and full text on the go!

Desktop-HomePage

Your Partner in Evidence Synthesis

With the most comprehensive technology including artificial intelligence, deduplication, and natural language processing, PICO Portal helps institutions af all kinds in citation screening and systematic review.

systematic review pico tools

US Department of Veterans Affairs

National academy of sciences.

systematic review pico tools

Johns Hopkins University

systematic review pico tools

University of Colorado

systematic review pico tools

Abilene Christian University

systematic review pico tools

University of Arizona

systematic review pico tools

Oregon Health and Science University

systematic review pico tools

Cornell University

systematic review pico tools

Florida Atlantic University

systematic review pico tools

Florida Gulf Coast University

systematic review pico tools

Indiana University

systematic review pico tools

Loma Linda University

systematic review pico tools

Monash University

systematic review pico tools

University of Minnesota

systematic review pico tools

University of Toronto

systematic review pico tools

Royal Victoria Regional Health Center

systematic review pico tools

University of Cologne

systematic review pico tools

University of South Florida

What our customers are saying.

“PICO Portal has greatly streamlined the review process: de-duplication, highlighting importing keywords during abstract review, and presenting the full text for full text review. All the members of the team were able to access the portal at the same time, thus enabling seamless collaboration across teams and locations. I would absolutely recommend it to other researchers for their reviews because it is easy to set up, intuitive to use, it standardizes the process, and it saves time.“

Researcher, Univeristy of South Florida

“It was easy to observe the efficiency of PICO Portal for the logistics management for multiple party classification of the same judgements, because I’ve built systems to do such a thing. But here I didn’t have to build the system, and it was very smooth. Knowing what it takes to build that system, that was a very efficient way to get it done. Multiple people could jump in and screen in a streamlined way.“

Brian S. Alper, MD, MSPH, FAAFP, FAMIA

Founder, COVID-19 Knowledge Accelerator (COKA)

“PICO Portal is easy to use, efficient, and time saving; especially when collaborating with colleagues. At Coreva Scientific, we conduct many literature reviews, and therefore we greatly appreciate that the PICO Portal team had an open ear to our suggestions. We were also impressed by how quickly they implemented additional functions we needed to streamline our review process. Functions such as tags, extensive reports, and highlighting of key words are very helpful.“

Juliane Hafermann

HEOR Medical Writer, Coreva

“All in all, I would recommend PICO Portal and would use it again. It makes it easy to manage the large volume of studies in my review. It is also helpful when working with a large team with members from multiple institutions who may not have access to the same shared drives or intranet."

Researcher in a Government Organization

“My experience with PICO Portal has been extremely positive. PICO Portal has efficiently and expediently helped us work through the process of systematically reviewing articles. The team at PICO Portal is helpful and I would recommend PICO Portal to other researchers as well."

Researcher at a University in Florida

“Highlighting and bolding of words specific to our review made things so much faster. It made it quick to look – you see it right there. You know within seconds it’s something you want to include. You click ‘include’ and that’s all you have to do."

Researcher in University of Minnesota

Becker Medical Library logotype

  • Library Hours
  • (314) 362-7080
  • [email protected]
  • Library Website
  • Electronic Books & Journals
  • Database Directory
  • Catalog Home
  • Library Home

Systematic Review Guide

  • Preliminary Search
  • PICO format
  • Full Text Retrieval
  • Covidence Screening Application
  • Evidence Grading & Appraisal
  • Risk of Bias Tools
  • Manuscript Preparation
  • Other Review Types

On the Systematic Review Request form you will be asked to outline your research question in PICO format. This allows us to easily understand the main concepts of your research question. Here is what PICO stands for:

P = Problem/Population

I = Intervention (or the experimental variable)

C = Comparison (or the control variable) [Optional]

O = Outcome

If your research question does not fit neatly into PICO that is okay. Just try to include the elements of your question as closely as possible into the format. Your collaborating librarian will discuss any questions or concerns about your research topic before putting together your systematic review search strategy.

  • << Previous: Preliminary Search
  • Next: Full Text Retrieval >>
  • Last Updated: Mar 15, 2024 11:45 AM
  • URL: https://beckerguides.wustl.edu/SystematicReviews
  • Open access
  • Published: 17 August 2023

Data visualisation in scoping reviews and evidence maps on health topics: a cross-sectional analysis

  • Emily South   ORCID: orcid.org/0000-0003-2187-4762 1 &
  • Mark Rodgers 1  

Systematic Reviews volume  12 , Article number:  142 ( 2023 ) Cite this article

3614 Accesses

13 Altmetric

Metrics details

Scoping reviews and evidence maps are forms of evidence synthesis that aim to map the available literature on a topic and are well-suited to visual presentation of results. A range of data visualisation methods and interactive data visualisation tools exist that may make scoping reviews more useful to knowledge users. The aim of this study was to explore the use of data visualisation in a sample of recent scoping reviews and evidence maps on health topics, with a particular focus on interactive data visualisation.

Ovid MEDLINE ALL was searched for recent scoping reviews and evidence maps (June 2020-May 2021), and a sample of 300 papers that met basic selection criteria was taken. Data were extracted on the aim of each review and the use of data visualisation, including types of data visualisation used, variables presented and the use of interactivity. Descriptive data analysis was undertaken of the 238 reviews that aimed to map evidence.

Of the 238 scoping reviews or evidence maps in our analysis, around one-third (37.8%) included some form of data visualisation. Thirty-five different types of data visualisation were used across this sample, although most data visualisations identified were simple bar charts (standard, stacked or multi-set), pie charts or cross-tabulations (60.8%). Most data visualisations presented a single variable (64.4%) or two variables (26.1%). Almost a third of the reviews that used data visualisation did not use any colour (28.9%). Only two reviews presented interactive data visualisation, and few reported the software used to create visualisations.

Conclusions

Data visualisation is currently underused by scoping review authors. In particular, there is potential for much greater use of more innovative forms of data visualisation and interactive data visualisation. Where more innovative data visualisation is used, scoping reviews have made use of a wide range of different methods. Increased use of these more engaging visualisations may make scoping reviews more useful for a range of stakeholders.

Peer Review reports

Scoping reviews are “a type of evidence synthesis that aims to systematically identify and map the breadth of evidence available on a particular topic, field, concept, or issue” ([ 1 ], p. 950). While they include some of the same steps as a systematic review, such as systematic searches and the use of predetermined eligibility criteria, scoping reviews often address broader research questions and do not typically involve the quality appraisal of studies or synthesis of data [ 2 ]. Reasons for conducting a scoping review include the following: to map types of evidence available, to explore research design and conduct, to clarify concepts or definitions and to map characteristics or factors related to a concept [ 3 ]. Scoping reviews can also be undertaken to inform a future systematic review (e.g. to assure authors there will be adequate studies) or to identify knowledge gaps [ 3 ]. Other evidence synthesis approaches with similar aims have been described as evidence maps, mapping reviews or systematic maps [ 4 ]. While this terminology is used inconsistently, evidence maps can be used to identify evidence gaps and present them in a user-friendly (and often visual) way [ 5 ].

Scoping reviews are often targeted to an audience of healthcare professionals or policy-makers [ 6 ], suggesting that it is important to present results in a user-friendly and informative way. Until recently, there was little guidance on how to present the findings of scoping reviews. In recent literature, there has been some discussion of the importance of clearly presenting data for the intended audience of a scoping review, with creative and innovative use of visual methods if appropriate [ 7 , 8 , 9 ]. Lockwood et al. suggest that innovative visual presentation should be considered over dense sections of text or long tables in many cases [ 8 ]. Khalil et al. suggest that inspiration could be drawn from the field of data visualisation [ 7 ]. JBI guidance on scoping reviews recommends that reviewers carefully consider the best format for presenting data at the protocol development stage and provides a number of examples of possible methods [ 10 ].

Interactive resources are another option for presentation in scoping reviews [ 9 ]. Researchers without the relevant programming skills can now use several online platforms (such as Tableau [ 11 ] and Flourish [ 12 ]) to create interactive data visualisations. The benefits of using interactive visualisation in research include the ability to easily present more than two variables [ 13 ] and increased engagement of users [ 14 ]. Unlike static graphs, interactive visualisations can allow users to view hierarchical data at different levels, exploring both the “big picture” and looking in more detail ([ 15 ], p. 291). Interactive visualizations are often targeted at practitioners and decision-makers [ 13 ], and there is some evidence from qualitative research that they are valued by policy-makers [ 16 , 17 , 18 ].

Given their focus on mapping evidence, we believe that scoping reviews are particularly well-suited to visually presenting data and the use of interactive data visualisation tools. However, it is unknown how many recent scoping reviews visually map data or which types of data visualisation are used. The aim of this study was to explore the use of data visualisation methods in a large sample of recent scoping reviews and evidence maps on health topics. In particular, we were interested in the extent to which these forms of synthesis use any form of interactive data visualisation.

This study was a cross-sectional analysis of studies labelled as scoping reviews or evidence maps (or synonyms of these terms) in the title or abstract.

The search strategy was developed with help from an information specialist. Ovid MEDLINE® ALL was searched in June 2021 for studies added to the database in the previous 12 months. The search was limited to English language studies only.

The search strategy was as follows:

Ovid MEDLINE(R) ALL

(scoping review or evidence map or systematic map or mapping review or scoping study or scoping project or scoping exercise or literature mapping or evidence mapping or systematic mapping or literature scoping or evidence gap map).ab,ti.

limit 1 to english language

(202006* or 202007* or 202008* or 202009* or 202010* or 202011* or 202012* or 202101* or 202102* or 202103* or 202104* or 202105*).dt.

The search returned 3686 records. Records were de-duplicated in EndNote 20 software, leaving 3627 unique records.

A sample of these reviews was taken by screening the search results against basic selection criteria (Table 1 ). These criteria were piloted and refined after discussion between the two researchers. A single researcher (E.S.) screened the records in EPPI-Reviewer Web software using the machine-learning priority screening function. Where a second opinion was needed, decisions were checked by a second researcher (M.R.).

Our initial plan for sampling, informed by pilot searching, was to screen and data extract records in batches of 50 included reviews at a time. We planned to stop screening when a batch of 50 reviews had been extracted that included no new types of data visualisation or after screening time had reached 2 days. However, once data extraction was underway, we found the sample to be richer in terms of data visualisation than anticipated. After the inclusion of 300 reviews, we took the decision to end screening in order to ensure the study was manageable.

Data extraction

A data extraction form was developed in EPPI-Reviewer Web, piloted on 50 reviews and refined. Data were extracted by one researcher (E. S. or M. R.), with a second researcher (M. R. or E. S.) providing a second opinion when needed. The data items extracted were as follows: type of review (term used by authors), aim of review (mapping evidence vs. answering specific question vs. borderline), number of visualisations (if any), types of data visualisation used, variables/domains presented by each visualisation type, interactivity, use of colour and any software requirements.

When categorising review aims, we considered “mapping evidence” to incorporate all of the six purposes for conducting a scoping review proposed by Munn et al. [ 3 ]. Reviews were categorised as “answering a specific question” if they aimed to synthesise study findings to answer a particular question, for example on effectiveness of an intervention. We were inclusive with our definition of “mapping evidence” and included reviews with mixed aims in this category. However, some reviews were difficult to categorise (for example where aims were unclear or the stated aims did not match the actual focus of the paper) and were considered to be “borderline”. It became clear that a proportion of identified records that described themselves as “scoping” or “mapping” reviews were in fact pseudo-systematic reviews that failed to undertake key systematic review processes. Such reviews attempted to integrate the findings of included studies rather than map the evidence, and so reviews categorised as “answering a specific question” were excluded from the main analysis. Data visualisation methods for meta-analyses have been explored previously [ 19 ]. Figure  1 shows the flow of records from search results to final analysis sample.

figure 1

Flow diagram of the sampling process

Data visualisation was defined as any graph or diagram that presented results data, including tables with a visual mapping element, such as cross-tabulations and heat maps. However, tables which displayed data at a study level (e.g. tables summarising key characteristics of each included study) were not included, even if they used symbols, shading or colour. Flow diagrams showing the study selection process were also excluded. Data visualisations in appendices or supplementary information were included, as well as any in publicly available dissemination products (e.g. visualisations hosted online) if mentioned in papers.

The typology used to categorise data visualisation methods was based on an existing online catalogue [ 20 ]. Specific types of data visualisation were categorised in five broad categories: graphs, diagrams, tables, maps/geographical and other. If a data visualisation appeared in our sample that did not feature in the original catalogue, we checked a second online catalogue [ 21 ] for an appropriate term, followed by wider Internet searches. These additional visualisation methods were added to the appropriate section of the typology. The final typology can be found in Additional file 1 .

We conducted descriptive data analysis in Microsoft Excel 2019 and present frequencies and percentages. Where appropriate, data are presented using graphs or other data visualisations created using Flourish. We also link to interactive versions of some of these visualisations.

Almost all of the 300 reviews in the total sample were labelled by review authors as “scoping reviews” ( n  = 293, 97.7%). There were also four “mapping reviews”, one “scoping study”, one “evidence mapping” and one that was described as a “scoping review and evidence map”. Included reviews were all published in 2020 or 2021, with the exception of one review published in 2018. Just over one-third of these reviews ( n  = 105, 35.0%) included some form of data visualisation. However, we excluded 62 reviews that did not focus on mapping evidence from the following analysis (see “ Methods ” section). Of the 238 remaining reviews (that either clearly aimed to map evidence or were judged to be “borderline”), 90 reviews (37.8%) included at least one data visualisation. The references for these reviews can be found in Additional file 2 .

Number of visualisations

Thirty-six (40.0%) of these 90 reviews included just one example of data visualisation (Fig.  2 ). Less than a third ( n  = 28, 31.1%) included three or more visualisations. The greatest number of data visualisations in one review was 17 (all bar or pie charts). In total, 222 individual data visualisations were identified across the sample of 238 reviews.

figure 2

Number of data visualisations per review

Categories of data visualisation

Graphs were the most frequently used category of data visualisation in the sample. Over half of the reviews with data visualisation included at least one graph ( n  = 59, 65.6%). The least frequently used category was maps, with 15.6% ( n  = 14) of these reviews including a map.

Of the total number of 222 individual data visualisations, 102 were graphs (45.9%), 34 were tables (15.3%), 23 were diagrams (10.4%), 15 were maps (6.8%) and 48 were classified as “other” in the typology (21.6%).

Types of data visualisation

All of the types of data visualisation identified in our sample are reported in Table 2 . In total, 35 different types were used across the sample of reviews.

The most frequently used data visualisation type was a bar chart. Of 222 total data visualisations, 78 (35.1%) were a variation on a bar chart (either standard bar chart, stacked bar chart or multi-set bar chart). There were also 33 pie charts (14.9% of data visualisations) and 24 cross-tabulations (10.8% of data visualisations). In total, these five types of data visualisation accounted for 60.8% ( n  = 135) of all data visualisations. Figure  3 shows the frequency of each data visualisation category and type; an interactive online version of this treemap is also available ( https://public.flourish.studio/visualisation/9396133/ ). Figure  4 shows how users can further explore the data using the interactive treemap.

figure 3

Data visualisation categories and types. An interactive version of this treemap is available online: https://public.flourish.studio/visualisation/9396133/ . Through the interactive version, users can further explore the data (see Fig.  4 ). The unit of this treemap is the individual data visualisation, so multiple data visualisations within the same scoping review are represented in this map. Created with flourish.studio ( https://flourish.studio )

figure 4

Screenshots showing how users of the interactive treemap can explore the data further. Users can explore each level of the hierarchical treemap ( A Visualisation category >  B Visualisation subcategory >  C Variables presented in visualisation >  D Individual references reporting this category/subcategory/variable permutation). Created with flourish.studio ( https://flourish.studio )

Data presented

Around two-thirds of data visualisations in the sample presented a single variable ( n  = 143, 64.4%). The most frequently presented single variables were themes ( n  = 22, 9.9% of data visualisations), population ( n  = 21, 9.5%), country or region ( n  = 21, 9.5%) and year ( n  = 20, 9.0%). There were 58 visualisations (26.1%) that presented two different variables. The remaining 21 data visualisations (9.5%) presented three or more variables. Figure  5 shows the variables presented by each different type of data visualisation (an interactive version of this figure is available online).

figure 5

Variables presented by each data visualisation type. Darker cells indicate a larger number of reviews. An interactive version of this heat map is available online: https://public.flourish.studio/visualisation/10632665/ . Users can hover over each cell to see the number of data visualisations for that combination of data visualisation type and variable. The unit of this heat map is the individual data visualisation, so multiple data visualisations within a single scoping review are represented in this map. Created with flourish.studio ( https://flourish.studio )

Most reviews presented at least one data visualisation in colour ( n  = 64, 71.1%). However, almost a third ( n  = 26, 28.9%) used only black and white or greyscale.

Interactivity

Only two of the reviews included data visualisations with any level of interactivity. One scoping review on music and serious mental illness [ 22 ] linked to an interactive bubble chart hosted online on Tableau. Functionality included the ability to filter the studies displayed by various attributes.

The other review was an example of evidence mapping from the environmental health field [ 23 ]. All four of the data visualisations included in the paper were available in an interactive format hosted either by the review management software or on Tableau. The interactive versions linked to the relevant references so users could directly explore the evidence base. This was the only review that provided this feature.

Software requirements

Nine reviews clearly reported the software used to create data visualisations. Three reviews used Tableau (one of them also used review management software as discussed above) [ 22 , 23 , 24 ]. Two reviews generated maps using ArcGIS [ 25 ] or ArcMap [ 26 ]. One review used Leximancer for a lexical analysis [ 27 ]. One review undertook a bibliometric analysis using VOSviewer [ 28 ], and another explored citation patterns using CitNetExplorer [ 29 ]. Other reviews used Excel [ 30 ] or R [ 26 ].

To our knowledge, this is the first systematic and in-depth exploration of the use of data visualisation techniques in scoping reviews. Our findings suggest that the majority of scoping reviews do not use any data visualisation at all, and, in particular, more innovative examples of data visualisation are rare. Around 60% of data visualisations in our sample were simple bar charts, pie charts or cross-tabulations. There appears to be very limited use of interactive online visualisation, despite the potential this has for communicating results to a range of stakeholders. While it is not always appropriate to use data visualisation (or a simple bar chart may be the most user-friendly way of presenting the data), these findings suggest that data visualisation is being underused in scoping reviews. In a large minority of reviews, visualisations were not published in colour, potentially limiting how user-friendly and attractive papers are to decision-makers and other stakeholders. Also, very few reviews clearly reported the software used to create data visualisations. However, 35 different types of data visualisation were used across the sample, highlighting the wide range of methods that are potentially available to scoping review authors.

Our results build on the limited research that has previously been undertaken in this area. Two previous publications also found limited use of graphs in scoping reviews. Results were “mapped graphically” in 29% of scoping reviews in any field in one 2014 publication [ 31 ] and 17% of healthcare scoping reviews in a 2016 article [ 6 ]. Our results suggest that the use of data visualisation has increased somewhat since these reviews were conducted. Scoping review methods have also evolved in the last 10 years; formal guidance on scoping review conduct was published in 2014 [ 32 ], and an extension of the PRISMA checklist for scoping reviews was published in 2018 [ 33 ]. It is possible that an overall increase in use of data visualisation reflects increased quality of published scoping reviews. There is also some literature supporting our findings on the wide range of data visualisation methods that are used in evidence synthesis. An investigation of methods to identify, prioritise or display health research gaps (25/139 included studies were scoping reviews; 6/139 were evidence maps) identified 14 different methods used to display gaps or priorities, with half being “more advanced” (e.g. treemaps, radial bar plots) ([ 34 ], p. 107). A review of data visualisation methods used in papers reporting meta-analyses found over 200 different ways of displaying data [ 19 ].

Only two reviews in our sample used interactive data visualisation, and one of these was an example of systematic evidence mapping from the environmental health field rather than a scoping review (in environmental health, systematic evidence mapping explicitly involves producing a searchable database [ 35 ]). A scoping review of papers on the use of interactive data visualisation in population health or health services research found a range of examples but still limited use overall [ 13 ]. For example, the authors noted the currently underdeveloped potential for using interactive visualisation in research on health inequalities. It is possible that the use of interactive data visualisation in academic papers is restricted by academic publishing requirements; for example, it is currently difficult to incorporate an interactive figure into a journal article without linking to an external host or platform. However, we believe that there is a lot of potential to add value to future scoping reviews by using interactive data visualisation software. Few reviews in our sample presented three or more variables in a single visualisation, something which can easily be achieved using interactive data visualisation tools. We have previously used EPPI-Mapper [ 36 ] to present results of a scoping review of systematic reviews on behaviour change in disadvantaged groups, with links to the maps provided in the paper [ 37 ]. These interactive maps allowed policy-makers to explore the evidence on different behaviours and disadvantaged groups and access full publications of the included studies directly from the map.

We acknowledge there are barriers to use for some of the data visualisation software available. EPPI-Mapper and some of the software used by reviews in our sample incur a cost. Some software requires a certain level of knowledge and skill in its use. However numerous online free data visualisation tools and resources exist. We have used Flourish to present data for this review, a basic version of which is currently freely available and easy to use. Previous health research has been found to have used a range of different interactive data visualisation software, much of which does not required advanced knowledge or skills to use [ 13 ].

There are likely to be other barriers to the use of data visualisation in scoping reviews. Journal guidelines and policies may present barriers for using innovative data visualisation. For example, some journals charge a fee for publication of figures in colour. As previously mentioned, there are limited options for incorporating interactive data visualisation into journal articles. Authors may also be unaware of the data visualisation methods and tools that are available. Producing data visualisations can be time-consuming, particularly if authors lack experience and skills in this. It is possible that many authors prioritise speed of publication over spending time producing innovative data visualisations, particularly in a context where there is pressure to achieve publications.

Limitations

A limitation of this study was that we did not assess how appropriate the use of data visualisation was in our sample as this would have been highly subjective. Simple descriptive or tabular presentation of results may be the most appropriate approach for some scoping review objectives [ 7 , 8 , 10 ], and the scoping review literature cautions against “over-using” different visual presentation methods [ 7 , 8 ]. It cannot be assumed that all of the reviews that did not include data visualisation should have done so. Likewise, we do not know how many reviews used methods of data visualisation that were not well suited to their data.

We initially relied on authors’ own use of the term “scoping review” (or equivalent) to sample reviews but identified a relatively large number of papers labelled as scoping reviews that did not meet the basic definition, despite the availability of guidance and reporting guidelines [ 10 , 33 ]. It has previously been noted that scoping reviews may be undertaken inappropriately because they are seen as “easier” to conduct than a systematic review ([ 3 ], p.6), and that reviews are often labelled as “scoping reviews” while not appearing to follow any established framework or guidance [ 2 ]. We therefore took the decision to remove these reviews from our main analysis. However, decisions on how to classify review aims were subjective, and we did include some reviews that were of borderline relevance.

A further limitation is that this was a sample of published reviews, rather than a comprehensive systematic scoping review as have previously been undertaken [ 6 , 31 ]. The number of scoping reviews that are published has increased rapidly, and this would now be difficult to undertake. As this was a sample, not all relevant scoping reviews or evidence maps that would have met our criteria were included. We used machine learning to screen our search results for pragmatic reasons (to reduce screening time), but we do not see any reason that our sample would not be broadly reflective of the wider literature.

Data visualisation, and in particular more innovative examples of it, is currently underused in published scoping reviews on health topics. The examples that we have found highlight the wide range of methods that scoping review authors could draw upon to present their data in an engaging way. In particular, we believe that interactive data visualisation has significant potential for mapping the available literature on a topic. Appropriate use of data visualisation may increase the usefulness, and thus uptake, of scoping reviews as a way of identifying existing evidence or research gaps by decision-makers, researchers and commissioners of research. We recommend that scoping review authors explore the extensive free resources and online tools available for data visualisation. However, we also think that it would be useful for publishers to explore allowing easier integration of interactive tools into academic publishing, given the fact that papers are now predominantly accessed online. Future research may be helpful to explore which methods are particularly useful to scoping review users.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Organisation formerly known as Joanna Briggs Institute

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Munn Z, Pollock D, Khalil H, Alexander L, McLnerney P, Godfrey CM, Peters M, Tricco AC. What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid Synth. 2022;20:950–952.

Peters MDJ, Marnie C, Colquhoun H, Garritty CM, Hempel S, Horsley T, Langlois EV, Lillie E, O’Brien KK, Tunçalp Ӧ, et al. Scoping reviews: reinforcing and advancing the methodology and application. Syst Rev. 2021;10:263.

Article   PubMed   PubMed Central   Google Scholar  

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18:143.

Sutton A, Clowes M, Preston L, Booth A. Meeting the review family: exploring review types and associated information retrieval requirements. Health Info Libr J. 2019;36:202–22.

Article   PubMed   Google Scholar  

Miake-Lye IM, Hempel S, Shanman R, Shekelle PG. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev. 2016;5:28.

Tricco AC, Lillie E, Zarin W, O’Brien K, Colquhoun H, Kastner M, Levac D, Ng C, Sharpe JP, Wilson K, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16:15.

Khalil H, Peters MDJ, Tricco AC, Pollock D, Alexander L, McInerney P, Godfrey CM, Munn Z. Conducting high quality scoping reviews-challenges and solutions. J Clin Epidemiol. 2021;130:156–60.

Lockwood C, dos Santos KB, Pap R. Practical guidance for knowledge synthesis: scoping review methods. Asian Nurs Res. 2019;13:287–94.

Article   Google Scholar  

Pollock D, Peters MDJ, Khalil H, McInerney P, Alexander L, Tricco AC, Evans C, de Moraes ÉB, Godfrey CM, Pieper D, et al. Recommendations for the extraction, analysis, and presentation of results in scoping reviews. JBI Evidence Synthesis. 2022;10:11124.

Google Scholar  

Peters MDJ GC, McInerney P, Munn Z, Tricco AC, Khalil, H. Chapter 11: Scoping reviews (2020 version). In: Aromataris E MZ, editor. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . Accessed 1 Feb 2023.

Tableau Public. https://www.tableau.com/en-gb/products/public . Accessed 24 January 2023.

flourish.studio. https://flourish.studio/ . Accessed 24 January 2023.

Chishtie J, Bielska IA, Barrera A, Marchand J-S, Imran M, Tirmizi SFA, Turcotte LA, Munce S, Shepherd J, Senthinathan A, et al. Interactive visualization applications in population health and health services research: systematic scoping review. J Med Internet Res. 2022;24: e27534.

Isett KR, Hicks DM. Providing public servants what they need: revealing the “unseen” through data visualization. Public Adm Rev. 2018;78:479–85.

Carroll LN, Au AP, Detwiler LT, Fu T-c, Painter IS, Abernethy NF. Visualization and analytics tools for infectious disease epidemiology: a systematic review. J Biomed Inform. 2014;51:287–298.

Lundkvist A, El-Khatib Z, Kalra N, Pantoja T, Leach-Kemon K, Gapp C, Kuchenmüller T. Policy-makers’ views on translating burden of disease estimates in health policies: bridging the gap through data visualization. Arch Public Health. 2021;79:17.

Zakkar M, Sedig K. Interactive visualization of public health indicators to support policymaking: an exploratory study. Online J Public Health Inform. 2017;9:e190–e190.

Park S, Bekemeier B, Flaxman AD. Understanding data use and preference of data visualization for public health professionals: a qualitative study. Public Health Nurs. 2021;38:531–41.

Kossmeier M, Tran US, Voracek M. Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis. BMC Med Res Methodol. 2020;20:26.

Ribecca, S. The Data Visualisation Catalogue. https://datavizcatalogue.com/index.html . Accessed 23 November 2021.

Ferdio. Data Viz Project. https://datavizproject.com/ . Accessed 23 November 2021.

Golden TL, Springs S, Kimmel HJ, Gupta S, Tiedemann A, Sandu CC, Magsamen S. The use of music in the treatment and management of serious mental illness: a global scoping review of the literature. Front Psychol. 2021;12: 649840.

Keshava C, Davis JA, Stanek J, Thayer KA, Galizia A, Keshava N, Gift J, Vulimiri SV, Woodall G, Gigot C, et al. Application of systematic evidence mapping to assess the impact of new research when updating health reference values: a case example using acrolein. Environ Int. 2020;143: 105956.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Jayakumar P, Lin E, Galea V, Mathew AJ, Panda N, Vetter I, Haynes AB. Digital phenotyping and patient-generated health data for outcome measurement in surgical care: a scoping review. J Pers Med. 2020;10:282.

Qu LG, Perera M, Lawrentschuk N, Umbas R, Klotz L. Scoping review: hotspots for COVID-19 urological research: what is being published and from where? World J Urol. 2021;39:3151–60.

Article   CAS   PubMed   Google Scholar  

Rossa-Roccor V, Acheson ES, Andrade-Rivas F, Coombe M, Ogura S, Super L, Hong A. Scoping review and bibliometric analysis of the term “planetary health” in the peer-reviewed literature. Front Public Health. 2020;8:343.

Hewitt L, Dahlen HG, Hartz DL, Dadich A. Leadership and management in midwifery-led continuity of care models: a thematic and lexical analysis of a scoping review. Midwifery. 2021;98: 102986.

Xia H, Tan S, Huang S, Gan P, Zhong C, Lu M, Peng Y, Zhou X, Tang X. Scoping review and bibliometric analysis of the most influential publications in achalasia research from 1995 to 2020. Biomed Res Int. 2021;2021:8836395.

Vigliotti V, Taggart T, Walker M, Kusmastuti S, Ransome Y. Religion, faith, and spirituality influences on HIV prevention activities: a scoping review. PLoS ONE. 2020;15: e0234720.

van Heemskerken P, Broekhuizen H, Gajewski J, Brugha R, Bijlmakers L. Barriers to surgery performed by non-physician clinicians in sub-Saharan Africa-a scoping review. Hum Resour Health. 2020;18:51.

Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5:371–85.

Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, McInerney P, Godfrey CM, Khalil H. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18:2119–26.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MDJ, Horsley T, Weeks L, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169:467–73.

Nyanchoka L, Tudur-Smith C, Thu VN, Iversen V, Tricco AC, Porcher R. A scoping review describes methods used to identify, prioritize and display gaps in health research. J Clin Epidemiol. 2019;109:99–110.

Wolffe TAM, Whaley P, Halsall C, Rooney AA, Walker VR. Systematic evidence maps as a novel tool to support evidence-based decision-making in chemicals policy and risk management. Environ Int. 2019;130:104871.

Digital Solution Foundry and EPPI-Centre. EPPI-Mapper, Version 2.0.1. EPPI-Centre, UCL Social Research Institute, University College London. 2020. https://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3790 .

South E, Rodgers M, Wright K, Whitehead M, Sowden A. Reducing lifestyle risk behaviours in disadvantaged groups in high-income countries: a scoping review of systematic reviews. Prev Med. 2022;154: 106916.

Download references

Acknowledgements

We would like to thank Melissa Harden, Senior Information Specialist, Centre for Reviews and Dissemination, for advice on developing the search strategy.

This work received no external funding.

Author information

Authors and affiliations.

Centre for Reviews and Dissemination, University of York, York, YO10 5DD, UK

Emily South & Mark Rodgers

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors conceptualised and designed the study and contributed to screening, data extraction and the interpretation of results. ES undertook the literature searches, analysed data, produced the data visualisations and drafted the manuscript. MR contributed to revising the manuscript, and both authors read and approved the final version.

Corresponding author

Correspondence to Emily South .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Typology of data visualisation methods.

Additional file 2.

References of scoping reviews included in main dataset.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

South, E., Rodgers, M. Data visualisation in scoping reviews and evidence maps on health topics: a cross-sectional analysis. Syst Rev 12 , 142 (2023). https://doi.org/10.1186/s13643-023-02309-y

Download citation

Received : 21 February 2023

Accepted : 07 August 2023

Published : 17 August 2023

DOI : https://doi.org/10.1186/s13643-023-02309-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Scoping review
  • Evidence map
  • Data visualisation

Systematic Reviews

ISSN: 2046-4053

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

systematic review pico tools

  • Systematic Review
  • Open access
  • Published: 04 May 2024

Efficacy of electronic apex locators in comparison with intraoral radiographs in working length determination- a systematic review and meta-analysis

  • Gurveen Kaur 1 ,
  • Anchu Rachel Thomas 2 ,
  • Renu Sarah Samson 2 ,
  • Eby Varghese 2 ,
  • Ratna Rachel Ponraj 2 ,
  • Sumanth Kumbargere Nagraj 2 ,
  • Deepti Shrivastava   ORCID: orcid.org/0000-0002-1073-9920 3 ,
  • Hmoud Ali Algarni 4 ,
  • Amna Yusuf Siddiqui   ORCID: orcid.org/0000-0003-0817-7129 5 ,
  • Osama S. Alothmani   ORCID: orcid.org/0000-0002-9552-1305 5 &
  • Kumar Chandan Srivastava   ORCID: orcid.org/0000-0002-5969-6810 6 , 7  

BMC Oral Health volume  24 , Article number:  532 ( 2024 ) Cite this article

212 Accesses

Metrics details

Successful endodontic treatment needs accurate determination of working length (WL). Electronic apex locators (EALs) were presented as an alternative to radiographic methods; and since then, they have evolved and gained popularity in the determination of WL. However, there is insufficient evidence on the post-operative pain, adequacy, and accuracy of EALs in determining WL.

The systematic review and meta-analysis aims to gather evidence regarding the effectiveness of EALs for WL determination when compared to different imaging techniques along with postoperative pain associated with WL determination, the number of radiographs taken during the procedure, the time taken, and the adverse effects.

For the review, clinical studies with cross-over and parallel-arm randomized controlled trials (RCTs) were searched in seven electronic databases, followed by cross-referencing of the selected studies and related research synthesis. Risk of bias (RoB) assessment was carried out with Cochrane's RoB tool and a random-effects model was used. The meta-analysis was performed with the RevMan software 5.4.1.

Eleven eligible RCTs were incorporated into the review and eight RCTs into the meta-analysis, of which five had high RoB and the remaining six had unclear RoB. Following meta-analysis, no significant difference in postoperative pain was found among the EAL and radiograph groups (SMD 0.00, CI .29 to .28, 354 participants; P value = 0.98). Radiograph group showed better WL accuracy (SMD 0.55, CI .11 to .99, 254 participants; P value = 0.02), while the EAL group had 10% better WL adequacy (RR 1.10, CI 1.03–1.18, 573 participants; P value = 0.006).

We found very low-certainty evidence to support the efficacy of different types of EAL compared to radiography for the outcomes tested. We were unable to reach any conclusions about the superiority of any type of EAL. Well-planned RCTs need to be conducted by standardizing the outcomes and outcome measurement methods.

Peer Review reports

Introduction

Successful endodontic treatment is highly dependent on efficient debridement, disinfection, and three-dimensional obturation [ 1 , 2 ]. Additionally, precise working length (WL) determination is also considered as a crucial step [ 3 , 4 ]. WL is the measurement from a reference at the coronal portion of the tooth to a specific location where the root canal procedures should conclude [ 5 ].

Root canal instrumentation and obturation are usually terminated at the WL, the best approximation to the apical constriction (AC), regarded as the anatomic reference [ 6 , 7 ]. This prevents damage to the peri-radicular tissue [ 8 , 9 ]. Incomplete obturation could leave infected tissue in the apical region and prevent healing of the periapical region [ 10 , 11 , 12 ]. However, it is challenging to identify the AC clinically or radiographically because it is a highly variable histological reference [ 13 , 14 ].

Radiographic method is the most preferred method for locating the apical end of the roots. However, accurate interpretation is often challenging with two-dimensional radiographs due to the superimposition of anatomical structures [ 15 ]. Concerns regarding radiation exposure, the number of radiographs taken, and the time required to acquire radiographs [ 16 , 17 , 18 ].

Apex locators have been used as an effective alternative for determination of WL compared to the radiographic method. Initially, these devices evaluated electrical resistance and, later, relative impedance within the root canal. They are referred to as "Foramen locators" to clarify their function in the WL determination of the canal. Furthermore, due to their ability to measure relative impedance inside the root canal, these devices can also identify the alteration in the cross-sectional area of the canal near its exit, commonly referred to as the "apical constriction" [ 19 ].

Nevertheless, the reliability of measurements is often compromised by the presence of fluids and metallic restorations. Regardless of the drawbacks, EAL are increasingly used in clinical practice because they reduce the number of radiographs and treatment time. However, a consensus is needed on the comparative accuracy of electronic and radiographic methods, as there is insufficient evidence-based research.

This systematic review and meta-analysis examine the evidence for the efficacy of EALs in assessing postoperative pain, adequacy, and accuracy in determining WL compared with various imaging modalities in patients with permanent dentition. The number of radiographs taken during the procedure, time required, and associated adverse effects were also evaluated.

Materials and methods

This systematic review and meta-analysis was performed and written based on the PRISMA guidelines for reporting research synthesis [ 20 , 21 ]. Protocol was recorded in an international database (PROSPERO- ID: CRD42021254714).

Eligibility criteria

The eligibility criteria were designed according to PICO (Patient/Population, Interventions, Comparison, Outcomes). The patient population included permanent human teeth with closed apices undergoing root canal treatment. The intervention included EALs or Endo motor with integrated apex locator; comparator included 2-dimensional intraoral periapical radiographs and 3-dimensional imaging (CBCT). The primary outcome was postoperative pain, WL accuracy, and adequacy, and secondary outcome included the number of radiographs, time required, and associated adverse effects. The study question was: Is there any difference in postoperative pain and working length determined using apex locators compared to other imaging modalities?

Included studies involved fully developed human permanent teeth; clinical studies; studies that provide comprehensive data about the measured distance between the file tip employed for EAL measurement and the exact location of the apical constriction (AC); manual assessment of the working length using multiple frequency EALs and determination of the working length during rotary file preparation.

Studies done on teeth with open apex, primary teeth, teeth with resorption, perforated, resected teeth; endodontically treated teeth; studies with an observing of the file through the apex; case reports, reviews, and observational studies; histological evaluation of apical anatomy; studies in which the distance of the file tip used for EAL measurement to the AC is not specified or given as a range, first and second generation EAL; identification of landmarks other than the constriction/ minor foramen; foreign language articles without English translation were excluded.

Strategy for search and selection of studies

A computerized literature search in seven databases was undertaken: MEDLINE PubMed, MEDLINE via OVID, LILACS, Embase, Scopus, Google Scholar, Cochrane Library from 1990 until October 2023, using words ‘‘Radiography’’, ‘‘Working length’’, and ‘‘Electronic Apex Locator’’ through PubMed to find the Medical Subject Headings terms for each word (Table  1 ). Additionally, a cross-reference search was done for included studies, other systematic reviews, and meta-analysis through connectedpapers.com.

Among the authors, two of them assessed the research headings and abstracts independently and in duplicate using Rayyan software [ 22 ]. For articles with full text, another two authors screened the studies individually and in duplicate to meet the inclusion and exclusion criteria. Differences in opinion were resolved through discussion with the arbiter to reach a consensus.

Data collection process

Data extraction form was designed with information about the study, purpose, sample, intervention, comparator, outcome measurement, results, adverse events, and author's conclusion (Table  2 ). Excluded studies and grounds for omission are reported in Table  3 .

Risk of bias evaluation

RoB evaluation was carried out by authors in pairs of two employing the RoB 1 tool [ 34 ]. Analysis and data entry was done using Review Manager 5.4.1 software.

Effect measures

Postoperative pain was expressed in the VAS scale as continuous data. If the scales used for postoperative pain were similar, results were described as mean differences (MD) with 95% confidence intervals (CI). If the scales were different, standardized mean difference (SMD) with 95% CI was used to pool data for similar outcomes from different trials. For the data that is continuous, the results were laid out as MD with 95% CI or SMD with 95% CI to incorporate data. The results for the data that is dichotomous, were laid out as risk ratios (RR) with 95% CI. Apart from reporting adverse effects, the rest of the secondary results were represented in the form of MD with 95% CI. Adverse effects were expressed qualitatively.

Synthesis methods

The meta-analysis was carried out with RevMan 5.4.1 which used SMD, random effect model, and inverse difference method. For WL determination mean difference, random effect model, statistical heterogeneity using the visual method, I 2 and Chi-square, and inverse difference method were used. All the results were presented for difference comparison. The I 2 statistic was used to evaluate the percent variation among studies due to heterogeneity.

When faced with uncertainty or incomplete information, the researchers of the studies were reached out to via email. If there was no communication from the authors after 15 days, it was reported as an unclear RoB. We computed the absent information using alternative available data sources, including standard deviations (SDs), P -values, visual representations, and, when necessary, data from other studies. Subsequently, the data was re-analysed in accordance with the intention-to-treat (ITT) principle whenever possible. Nevertheless, none of the studies furnished adequate details to conduct the ITT analysis.

Certainty of evidence

The certainty of evidence for each outcome was assessed using the GRADE framework through GRADEpro GDT [ 21 ].

Study selection

One thousand four hundred forty-three references were found in the electronic database search and 4 additional studies from a non-systematic search of Google Scholar, contacting corresponding authors to attain full texts, relevant systematic reviews, and cross-referencing of included studies. Following the removal of duplicate entries, 1042 references underwent initial screening based on their titles and abstracts, resulting in the exclusion of 1012 references. From 30 full-text articles, 10 studies were excluded (Table  3 ) [ 19 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ]. One ongoing trial [ 44 ]was identified however, from 8 studies required data couldn’t be obtained and hence await classification [ 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 ]. In accordance with the specified inclusion criteria, 11 studies met the requirements for inclusion in the systematic review [ 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ], with 8 of these studies being further incorporated into the meta-analysis [ 23 , 24 , 26 , 27 , 28 , 29 , 31 , 32 , 33 ]. Figure  1 shows the study selection process in detail.

figure 1

Flow diagram showing the literature selection process

Study characteristics

Among the eleven studies considered, ten studies were recorded in scholarly, peer-reviewed publications, while one among them was a dissertation [ 32 ]. Only two studies reported their funding details [ 28 , 32 ]. The rest of the studies neither reported nor disclosed any funding details. All trials were parallel-group randomized controlled trials, with no cross-over trials.

In terms of patients, the least of the sample size was thirty [ 26 ], and the largest sample size was 220 (34). The maximum age of the participants taken was 75 years [ 32 ], and the lowest was 5 years [ 26 ]. All the selected studies had patients with permanent teeth except for one study, which had both primary and permanent dentition [ 26 ]. However, the results of the permanent teeth were only taken for analysis. One study [ 30 ] failed to provide any information about the age group.

Three studies [ 23 , 27 , 33 ] only included teeth with single roots and patent canals; however, another research [ 28 ] included both single-rooted and multirooted teeth. In addition, two research on multirooted teeth alone [ 24 , 30 ] and two investigations on molars [ 25 , 26 ] were also done. Root canals were considered in two studies; however, it was not stated whether the teeth were single or multirooted or whether they were mandibular or maxillary [ 29 , 31 ].

Most included studies recruited healthy participants who were advised for root canal treatment. Patients with cardiac pacemakers, periapical radiolucency, curved roots, incomplete root formation, root resorption, Expectant mothers, and individuals with pre-existing systemic ailments were not considered for inclusion. One study did not mention any exclusion criteria [ 24 ].

Risk of bias in studies

Five studies [ 24 , 25 , 26 , 28 , 32 ] had high susceptibility to bias and the rest of the 6 trials had unclear susceptibility to bias because they each had at least two unclear bias domains [ 23 , 27 , 29 , 30 , 31 , 33 ] (Fig.  2 a and b). Random sequence generation was reported in 7 trials [ 24 , 25 , 27 , 28 , 29 , 32 , 33 ] whereas, only 3 trials provided allocation concealment details [ 24 , 28 , 32 ]. One study provided details on the concealment of information from both participants and personnel [ 32 ] and blinding was reported in 6 studies [ 24 , 25 , 27 , 29 , 31 , 32 ]. For the attrition bias, 3 trials [ 26 , 28 , 32 ] had drop-outs and hence had high susceptibility to bias. Among the entire set of studies, only one had a registered study plan and disclosed all the pre-planned results [ 29 ].

figure 2

a  Risk-of-bias summary. Review authors' judgements about each risk of bias item for each included study. Green colour indicates ‘low risk of bias’, yellow indicates ‘unclear risk of bias’ and red colour indicates ‘high risk of biases. b  Review authors' judgements about each risk of bias item presented as percentages across all included studies. Green indicates ‘low risk of bias’, yellow indicates ‘unclear risk of bias’ and red indicates ‘high risk of biases

No other pertinent prejudices were identified in any of the 11 studies, rendering them at minimal risk of bias. All the incorporated studies had either higher risk or unclear risk of bias. Therefore, sensitivity analysis was not done as planned. Publication bias was not assessed since the meta-analysis didn’t include 10 or more studies.

Results of individual studies

Meta-analysis, postoperative pain.

Three RCTs tested the postoperative pain following working length determination [ 23 , 24 , 33 ]. The inter-quartile range was derived from the graph using Plot Digitizer software for one study [ 23 ] and SMD was calculated according to Sect. 6.5.2.5 of the Cochrane Handbook [ 34 ]. The evidence showed no difference in postoperative pain in the EAL group in comparison with the radiograph group with a pooled effect estimate of SMD 0.00 (CI -0.29, 0.28, 354 participants; P value = 0.98). The confidence intervals intersected the effect line, making the findings' conclusion less robust (Fig.  3 a).

figure 3

a  Forest plot showing post-operative pain (Electronic Apex Locator Vs Radiographs). b  Forest plot showing adequacy in WL determination (Dichotomous data). c  Forest plot showing the accuracy of WL determination (Continuous data)

Working length adequacy – dichotomous data

 Four studies evaluated the adequacy of working length determination between EAL and radiographs [ 27 , 28 , 29 , 31 ]. The evidence showed better adequacy in working length determination in the EAL group in comparison to the group using radiographs with a pooled effect estimate of RR 1.10 (CI 1.03 to 1.18, 573 participants; P value = 0.02) (Fig.  3 b). EAL also gives a 10% increase in working length adequacy compared to the radiographs.

Working length accuracy -continuous data

Four studies evaluated the WL accuracy [ 25 , 26 , 28 , 32 ]. Two studies [ 25 , 26 ] assessed the working length accuracy in individual canals and were not a part of the meta-analysis. The other two studies [ 28 , 32 ] evaluated this comparison. The evidence suggests that working length determination in the EAL group in comparison to the group using radiographs with a pooled effect estimate of SMD 0.55 (CI 0.11 to 0.99, 254 participants; P value = 0.006) (Fig.  3 c).

The certainty of evidence was exceedingly minimal in accordance with the GRADE levels [ 21 ] of certainty for all the primary outcomes (Table  4 ). The downgrading was attributed to concerns about bias, inconsistency, and imprecision (as mentioned in Table  4 ).

Adequate control of the working length during endodontic treatment is expected to impact the treatment results and prevent postoperative pain [ 53 , 54 , 55 ]. Electronic apex locators provide an effective means of locating working lengths for endodontic procedures [ 56 ]. The rationale of this review and meta-synthesis was to determine the existing body of evidence exhibiting the accuracy and adequacy of electronic apex locators and the postoperative pain during WL determination in comparison with radiographic methods which is prevalent in clinical use.

The recommendations of The European Society of Endodontology [ 57 ], suggest the use of an EAL followed by verifying the canal length with a radiograph during the procedure. In some cases, master cone radiograph to confirm the working length is suggested. Since none of the approaches can be considered an exact substitute for the histological method, the radiographic method has been used as the reference standard in this review. Histological methods cannot be a practical option when clinical trials are included.

The efficiency of EALs has been assessed in terms of postoperative pain, accuracy, and adequacy. Accuracy refers to the extent to which measurements deviate from a designated target, such as the apical foramen [ 58 ]. In this review, two studies [ 25 , 26 ] mention the term accuracy in their clinical trials, although accuracy can only be compared using histological landmarks.

Master cone adequacy refers to when the master cone gutta-percha is considered adequate when it is 0–2 mm from the radiographic apex. Ng et al. studied the factors influencing the outcomes of endodontic therapy and concluded that every unistrumented millimeter of the canal, reduces the success rates by 12%, whereas overextended root fillings result in a 62% reduction in success [ 59 ]. Furthermore, in the study done by Meirinhos et al., they stated that periapical lesions were 3.1% more likely to be associated with short root fillings [ 60 ]. These points state the importance of the adequacy of the master cone. Hence, master cone adequacy should be an important outcome in the success of the therapy.

Based on the Cochrane Handbook, outcome measures are not considered criteria for including studies in a review [ 61 ]. Hence, the inclusion criteria for the research synthesis were designed based on the components: population, intervention, and comparator. Postoperative pain is a clinical outcome of inaccurate working length determination and master cone inadequacy; hence it was not included in the eligibility criteria.

A literature search involving seven electronic search engines and a cross-reference search was conducted to identify eligible research comparing the efficiency of EAL to radiographic methods to determine WL during root canal treatment of permanent teeth. Stringent criteria for inclusion and exclusion were employed in the studies to overcome the heterogeneity of data. The search terms were limited to “Radiography’’, ‘‘Working length’’, and ‘‘Electronic Apex Locator’’ to allow the inclusion of a larger number of studies. This systematic review included eleven in vivo studies with cross-over and parallel-arm randomized controlled trials from the above-mentioned period to ensure high-quality evidence. Eight of eleven studies were incorporated into the meta-analysis.

Intervention group incorporated, EALs representing third-generation and higher, and Endo motors with integrated apex locators due to their superior performance in comparison to the first and second-generation models that had been previously reported [ 28 , 62 , 63 ].

Consensus and disparity with other systematic reviews:

The systematic review conducted by Amin et al . concluded that the accuracy of CBCT compared to EAL couldn’t be determined due to significant heterogeneity but suggested using pre-existing CBCT scans for WL determination [ 64 ]. A study concluded that the precision of EAL was comparable to the radiographic method. However, EAL and digital radiographic methods could reduce radiation dose exposure [ 24 ]. Another systematic review by Martins et al . reported inadequate scientific evidence and a considerable risk of bias. They suggested that WL determination using EAL could perform better than radiography alone, reducing patient radiation exposure. However, it was also recommended to perform at least one radiographic assessment to identify potential errors in electronic devices [ 56 ].

In this review, three out of eleven studies assessed the postoperative pain following WL determination using either EAL or radiographs [ 33 , 62 ] which was the primary outcome. Kara-Tuncer & Gerek, Naeem et al ., and Khan et al . 2021 concluded no significant disparity among the EAL and digital radiography groups in the postoperative pain dissipation period [ 23 , 24 , 33 ].

The adequacy and accuracy of WL determination were interpreted as continuous and dichotomous data [ 25 , 26 , 27 , 28 , 29 , 31 , 32 , 33 ]. Studies done by Rathore et al. and Vanitha & Sherwood were a part of the systematic review but weren’t incorporated in the meta-analysis as the readings presented were for individual canals [ 25 , 26 ]. However, both the studies were included qualitatively and the RoB assessment was done. Substantial heterogeneity was observed (Heterogeneity: Tau 2  = 0.06; Chi 2  = 2.06, df = 1 ( P  = 0.15); I 2  = 51%). For the dichotomous data, five studies evaluated the adequacy of working length between EAL and Radiographs [ 27 , 28 , 29 , 30 , 31 ]. However, the study done by Saraf et al. presented data with respect to six different EALs and thus not incorporated in the meta-analysis [ 30 ].

Appropriate utilization of EAL alone could eliminate the requirement for an additional radiograph for diagnosis to determine WL. Patients who don't need to repeatedly be exposed to radiation due to mental, medical, or dental issues may benefit from this procedure. According to previous research, the use of EALs decreased the frequency of taking radiographs, reducing treatment time, effort, and radiation exposure to patients [ 29 , 65 , 66 ]. Furthermore, EALs have the potential to reduce the incidence of overextension of root canal procedures, which may result in postoperative pain and difficulty in maintaining the apical stop. Estimation of WL prior to radiographic verification maintains the correct working length for termination of obturation thus preventing overestimation of root canal length resulting in postoperative pain. The results of a study suggested that the apical foramen was accurately located by apex locators when used correctly and that only one preoperative radiograph was required [ 67 ]. They also help the operator in suspecting root fractures, resorptions, and perforations [ 58 ].

There is significant evidence that shows the point of termination of root canal instrumentation and obturation affects the outcome of endodontic treatment. However, there is insufficient data on the outcome of endodontic therapy in relation to EAL assessment. As a result, the radiographic WL measurement remains crucial for clinical purposes, and the Ideal application of EALs would be to reduce the number of radiographic exposures by estimating the WL accurately before taking any radiographic measurements [ 68 ]. By considering both radiographic and electronic measurements, as well as the accuracy of EALs and the morphology of the root apex, the final working length (FWL), would be determined [ 65 ].

Grade assessment and summary of findings

The meta-analysis suggested that radiographs were more accurate and EALs were more adequate in determining WL. Nonetheless, deriving dependable conclusions wasn’t feasible given the extremely low certainty of evidence owing to concerns about bias, inconsistencies, and imprecision. As a result, the superiority of any intervention over another couldn’t be determined. The findings of the research should be interpreted with caution and further clinical trials are needed to confirm the results.

Since the available evidence is of very low certainty, more randomised controlled trials assessing the effect of working length on postoperative pain, working length accuracy and adequacy need to be conducted. Also, more trials on the effect of integrated apex locators on the working length accuracy and adequacy needs to be performed. Furthermore, well- executed RCTs need to be carried out on different generations of EALs and radiographic methods such as digital radiography and CBCT. Moreover, the studies included in the review did not assess the cost-effectiveness, reduction in radiation exposure, and the number of radiographs except one study [ 29 ].

Due to the unavailability of the full text, eight articles weren’t incorporated in the research synthesis. The studies were searched across libraries, Research Gate, and Google Scholar, apart from efforts made to contact authors by email. The extensiveness of the search was limited since the grey literature was not explored. None of the included studies fulfilled the secondary outcome of adverse events.

To explore the possible effect of losses to follow-up on the effect estimates for the primary outcomes, sensitivity analyses, subgroup analysis and other factors for heterogeneity was planned in the protocol. Nevertheless, only one or two studies were included under most of the comparisons and thus the sensitivity and subgroup analysis could not be conducted.

The publication bias could not be assessed since the meta-analysis didn’t include 10 or more studies. The primary outcome, postoperative pain could have several contributory reasons. However, this review does not include all the reasons since the primary studies did not report the cause of the pain. The results of unpublished data have been included in this meta-analysis, since the overall evidence is of very low quality; we assume it wouldn’t have affected the results.

Future research should focus on:

Population – Well-defined inclusion criteria for participants and clinical trials involving both anterior and posterior teeth with vital and necrotic pulps.

Intervention –Trials focussing on evaluating intervention methods like Endo-Motors with integrated apex locators and the latest generation of EALs.

Control- additional RCTs carried out on different generations of EALs and radiographic methods such as digital radiography and CBCT.

Outcome- Trials focussing on assessing patient-related outcomes, adverse events, and direct outcome measurements like post-operative pain following WL determination.

The current review emphasizes the requirement for well-executed RCTs with trial reports adhering to the guidelines from CONSORT 2010 [ 69 ] and incorporating the results generated through fundamental consortiums like Core Outcome Measures in Effectiveness Trials (COMET) [ 70 ]. This will contribute to the current body of evidence, allowing researchers to formulate more dependable findings.

The review concluded that there was no significant disparity with regard to post-operative pain in the EAL group compared to the radiograph group. Better accuracy with respect to WL using radiographs than EAL and better adequacy in WL using EAL than radiographs. Although there is no gold standard, adequacy is an important outcome in root canal treatment. Hence, we will be valuing the clinician’s judgment even for a short range of 0-2mm for acceptability.

Availability of data and materials

The data will be available on reasonable request from the corresponding author.

Rajamanickam K, Teja KV, Ramesh S, AbuMelha AS, Alkahtany MF, Almadi KH, et al. Comparative Study Assessing the Canal Cleanliness Using Automated Device and Conventional Syringe Needle for Root Canal Irrigation—An Ex-Vivo Study. Materials. 2022;15:6184.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Teja KV, Janani K, Srivastava KC, Shrivastava D, Jose J, Marya A, et al. Comparison of Herbal Agents with Sodium Hypochlorite as Root Canal Irrigant: A Systematic Review of In Vitro Studies. Evidence-Based Complementary and Alternative Medicine. 2021;2021:1–11.

Article   Google Scholar  

Mousa MA, Alshammari AM, Alshammari AZ, Alsharari MM, Baig MN, Srivastava KC, et al. Comparative Evaluation of Adequacy of Root Canal Treatment Performed by Interns and Undergraduate Dental Students—A Cross-Sectional. Retrospective Study Applied Sciences. 2022;12:9079.

Article   CAS   Google Scholar  

Janani K, Teja KV, Srivatsava KC. How much to enlarge? A letter to the editor. Saudi Endodontic Journal. 2023;13:288–9.

Endodontists AA of. Glossary of endodontic terms. American Association of Endodontists; 2003.

Ricucci L. Apical limit of root canal instrumentation and obturation, part 2. A histological study Int Endod J. 1998;31:394–409.

Article   CAS   PubMed   Google Scholar  

Iqbal A, Karobari MI, Alam MK, Khattak O, Alshammari SM, Adil AH, et al. Evaluation of root canal morphology in permanent maxillary and mandibular anterior teeth in Saudi subpopulation using two classification systems: a CBCT study. BMC Oral Health. 2022;22:171.

Article   PubMed   PubMed Central   Google Scholar  

Malkhassian G,Plazas A, Nahmias Y. Electronic Apex Locators and conventional radiographs in working length measurement. 2nd Ed. Edited by Bettina Basrani. New Jersey: Wiley and Blackwell. 2012. p. 218–34.

Teja KV, Janani K, Alqahtani AA, Robaian A, Alhalabi F, Merdad KA, et al. Herbal Agents versus Ethylene Diamine Tetra Acetic Acid on Removal of the Smear Layer—A Systematic Review of In Vitro Studies. Int J Environ Res Public Health. 2022;19:6870.

Schilder H. Filling root canals in three dimensions. Dent Clin North Am. 1967;11:723–44.

Seltzer S, Soltanoff W, Sinai I, Smith J. Biologic aspects of endodontics: IV Periapical tissue reactions to root-filled teeth whose canals had been instrumented short of their apices. Oral Surgery, Oral Medicine, Oral Pathology. 1969;28:724–38.

Teja KV, Janani K, Srivastava KC, Shrivastava D, Natoli V, Di Blasio M, et al. Comparative evaluation of antimicrobial efficacy of different combinations of calcium hydroxide against Enterococcus faecalis. BMC Oral Health. 2023;23:849.

Dummer PMH, Mcginn JH, Rees DG. The position and topography of the apical canal constriction and apical foramen. Int Endod J. 1984;17:192–8.

Jose J, Teja KV, Ranjan M, Mohamed RN, Alam MK, Shrivastava D, et al. Influence of Commonly Used Endodontic Irrigants on the Setting Time and Metal Composition of Various Base Endodontic Sealers. Polymers (Basel). 2021;14:27.

Article   PubMed   Google Scholar  

Srivastava KC, Shrivastava D, Austin RD. Journey towards the 3D dental imaging-the milestones in the advancement of dental imaging. Int J Adv Res. 2016;4:377–82.

Grove CJ. Further evidence that root canals can be filled to the dentinocemental junction. J Am Dent Assoc. 1922;1930(17):1529–35.

Google Scholar  

Kuttler Y. Microscopic investigation of root apexes. J Am Dent Assoc. 1955;50:544–52.

ASushanth A, Srivastava KC, Shrivastava D, Hosni HA, Khan ZA, Al-Johani K, et al. Recommendations, Practices and Infrastructural Model for the Dental Radiology Set-up in Clinical and Academic Institutions in the COVID-19 Era. Biology (Basel). 2020;9:334.

ElAyouti A, Dima E, Ohmer J, Sperl K, von Ohle C, Löst C. Consistency of apex locator function: a clinical study. J Endod. 2009;35:179–81.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700.

Chandler J, Cumpston M, Li T. Cochrane Handbook for Systematic Reviews of Interventions.

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5:1–10.

Kara Tuncer A, Gerek M. Effect of working length measurement by electronic apex locator or digital radiography on postoperative pain: a randomized clinical trial. J Endod. 2014;40(1):38–41.

Naeem F, Abdelaziz S, Ahmed Geraldine M. Evaluation of postoperative pain after working length determination using electronic apex locator versus digital radiography in multirooted teeth. a randomized controlled trial. Int J Adv Res (Indore). 2017;5:799–809.

Vanitha S, Sherwood IA. Comparison of three different apex locators in determining the working length of mandibular first molar teeth with irreversible pulpitis compared with an intraoral periapical radiograph: A block randomized, controlled, clinical trial. J Investig Clin Dent. 2019;10: e12408.

Rathore K, Tandon S, Sharma M, Kalia G, Shekhawat T, Chundawat Y. Comparison of accuracy of apex locator with tactile and conventional radiographic method for working length determination in primary and permanent teeth. Int J Clin Pediatr Dent. 2020;13:235.

Singh D, Tyagi SP, Gupta S, Jain A. Comparative evaluation of adequacy of final working length after using Raypex5 or radiography: An: in vivo: study. Journal of Indian Society of Pedodontics and Preventive Dentistry. 2015;33:208–12.

Jarad FD, Albadri S, Gamble C, Burnside G, Fox K, Ashley JR, et al. Working length determination in general dental practice: a randomised controlled trial. Br Dent J. 2011;211:595–8.

Ravanshad S, Adl A, Anvar J. Effect of working length measurement by electronic apex locator or radiography on the adequacy of final working length: a randomized clinical trial. J Endod. 2010;36:1753–6.

Saraf PA, Ratnakar P, Patil TN, Penukonda R, Kamatagi L, Vanaki SS. A comparative clinical evaluation of accuracy of six apex locators with intraoral periapical radiograph in multirooted teeth: An in vivo study. J Conserv Dent. 2017;20(4):264–8.

Koçak S, Koçak MM, Sağlam BC. Efficiency of 2 electronic apex locators on working length determination: A clinical study. Journal of Conservative Dentistry and Endodontics. 2013;16:229–32.

Joseph M. Clinical success of two working length determination techniques: a randomized controlled trial. Madha Dental College, Department of Conservative Dentistry and Endodontics, Chennai; 2019.

Khan A, Afridi F, Iqbal K. Effect of working length measurement by electronic apex locater and digital radiography on post-operative pain dissipation. dimensions. 2021;5:5.

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Bmj. 2011;343.

Paludo L, de Souza SL, Só MVR, da Rosa RA, Vier-Pelisser FV, Duarte MAH. An in vivo radiographic evaluation of the accuracy of Apex and iPex electronic Apex locators. Braz Dent J. 2012;23:54–8.

Keller ME, Brown CE Jr, Newton CW. A clinical evaluation of the Endocater—an electronic apex locator. J Endod. 1991;17:271–4.

Orosco FA, Bernardineli N, Garcia RB, Bramante CM, Duarte MAH, de Moraes IG. In vivo accuracy of conventional and digital radiographic methods in confirming root canal working length determination by Root ZX. J Appl Oral Sci. 2012;20:522–5.

Thomas AS, Hartwell GR, Moon PC. The accuracy of the Root ZX electronic apex locator using stainless-steel and nickel-titanium files. J Endod. 2003;29:662–3.

Hembrough JH, Weine FS, Pisano JV, Eskoz N. Accuracy of an electronic apex locator: a clinical evaluation in maxillary molars. J Endod. 1993;19:242–6.

Fouad AF. The use of electronic apex locators in endodontic therapy. Int Endod J. 1993;26:13–4.

Cardoso Tarallo A, De souza Matos F, Bresciani E, Paranhos LR, Ribeiro Camargo CH. Influence of Working Length on Post-operative Pain after Single or Two-Visit Endodontic Treatment: A Randomised Clinical Trial. J Clin Diagn Res. 2018;12(10):ZC06–ZC11

Diniz-de-Figueiredo FE, Lima LF, Oliveira LS, Bernardino IM, Paiva SM, Faria-e-Silva AL. The impact of two root canal treatment protocols on the oral health-related quality of life: a randomized controlled pragmatic clinical trial. Int Endod J. 2020;53:1327–38.

Himel VT, Cain C. An evaluation of two electronic apex locators in a dental student clinic. Quintessence Int (Berl). 1993;24.

Boris Saha, Surendra kumar Mishra. Effect of three different methods of working length determination during root canal preparation on post operative pain - An in vivo study. 2019. https://trialsearch.who.int/Trial2.aspx?TrialID=CTRI/2019/07/019960 . Accessed 31 Mar 2024.

Chang-xi LU. Comparison of the accuracy of measuring root canal working length in 3 different ways. Shanghai Journal of Stomatology. 2017;26:535.

Vachey E, Lemagnen G, Grislain L, Miquel JL. Alternatives to radiography for determining root canal length. Odontostomatol Trop. 2003;26:15–8.

CAS   PubMed   Google Scholar  

Dehon P, Van Nieuwenhuysen JP. Electronic determination of the length of the root canal. Rev Belge Med Dent. 1984;1999(54):259–81.

Einbinder AB, Kratchman SI. Apex locators: a systematic approach for usage. Compendium. 1993;14:34–8.

McDonald NJ. Radiographic and electronic diagnostic systems. Alpha Omegan. 1991;84:45–8.

Ott KH, Pioch T, Hoppe W. Study of properties of electrical root canal length measuring instruments. ZWR. 1991;100:313–4.

Dotto RF, Renner D, Barletta FB, Dotto SR, Wagner MH. Avaliação da determinação do comprimento de trabalho através do localizador apical eletrônico NOVAPEX. Rev odontol Univ Cid Sao Paulo. 2005;:263–9.

Vladimirov SB, Filipov IA, Vangelov LS. Electrometric determination of the working length of root canals by means of Apex locator. Folia Med (Plovdiv). 2001;43:42–5.

Chugal NM, Clive JM, Spångberg LSW. Endodontic infection: some biologic and treatment factors associated with outcome. Oral Surgery, Oral Medicine, Oral Pathology, Oral Radiology, and Endodontology. 2003;96:81–90.

Williams CB, Joyce AP, Roberts S. A comparison between in vivo radiographic working length determination and measurement after extraction. J Endod. 2006;32:624–7.

Ganesh VS, Venkatesh KV, Sihivahanan D, Yadalam PK, Shrivastava D, Srivastava KC. Effect of microbubble as local drug delivery system in endodontic management - An In-Vitro study. Saudi Dent J. 2024. https://doi.org/10.1016/j.sdentj.2024.03.010 .

Martins JNR, Marques D, Mata A, Caramês J. Clinical efficacy of electronic apex locators: systematic review. J Endod. 2014;40:759–77.

Endodontology ES, of. Quality guidelines for endodontic treatment: consensus report of the European Society of Endodontology. Int Endod J. 2006;39:921–30.

ElAyouti A, Connert T, Dummer P, Löst C. A critical analysis of research methods and experimental models to study working length determination and the performance of apex locators–A narrative review with recommendations for the future. Int Endod J. 2022;55:281–94.

Yl N. A prospective study of the factors affecting outcomes of nonsurgical root canal treatment: part 1: periapical health. Int Endod J. 2011;44:583–609.

Meirinhos J, Martins JNR, Pereira B, Baruwa A, Gouveia J, Quaresma SA, et al. Prevalence of apical periodontitis and its association with previous root canal treatment, root canal filling length and type of coronal restoration–a cross-sectional study. Int Endod J. 2020;53:573–84.

McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston R V, Thomas J. Defining the criteria for including studies and how they will be grouped for the synthesis. Cochrane handbook for systematic reviews of interventions. 2019;:33–65.

Gordon MPJ, Chandler NP. Electronic apex locators. Int Endod J. 2004;37:425–37.

Fouad AF, Krell KV, McKendry DJ, Koorbusch GF, Olson RA. A clinical evaluation of five electronic root canal length measuring instruments. J Endod. 1990;16:446–9.

Amin J, Lines J, Milosevic MP, Park A, Sholapurkar A. Comparison of accuracy and reliability of working length determination using cone beam computed tomography and electronic apex locator: a systematic review. J Contemp Dent Pract. 2019;20:1118–23.

Fouad AF, Reid LC. Effect of using electronic apex locators on selected endodontic treatment parameters. J Endod. 2000;26:364–7.

Smadi L. Comparison between two methods of working length determination and its effect on radiographic extent of root canal filling: a clinical study [ISRCTN71486641]. BMC Oral Health. 2006;6:1–6.

Vajrabhaya L, Tepmongkol P. Accuracy of apex locator. Dent Traumatol. 1997;13:180–2.

Brunton PA, Abdeen D, Macfarlane TV. The effect of an apex locator on exposure to radiation during endodontic therapy. J Endod. 2002;28:524–6.

Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, CONSORT, et al. explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;2010:340.

Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET handbook: version 1.0. Trials. 2017;18:1–50.

Download references

Acknowledgements

Non-funded.

Author information

Authors and affiliations.

Dept of Conservative Dentistry and Endodontics, National Dental College, Baba Farid University of HEALth Sciences, Faridkot, India

Gurveen Kaur

Department of Conservative Dentistry and Endodontics, Faculty of Dentistry, Manipal University College Malaysia, Jalan Batu Hampar, Bukit Baru, Melaka, Malaysia

Anchu Rachel Thomas, Renu Sarah Samson, Eby Varghese, Ratna Rachel Ponraj & Sumanth Kumbargere Nagraj

Department of Preventive Dentistry, College of Dentistry, Jouf University, 72345, Sakaka, Saudi Arabia

Deepti Shrivastava

Department of Restorative Dental Sciences, College of Dentistry, Jouf University, 72345, Sakaka, Saudi Arabia

Hmoud Ali Algarni

Department of Endodontics, Faculty of Dentistry, King Abdulaziz University, Jeddah, Saudi Arabia

Amna Yusuf Siddiqui & Osama S. Alothmani

Oral Medicine & Radiology, Department of Oral & Maxillofacial Surgery & Diagnostic Sciences, College of Dentistry, Jouf University, 72345, Sakaka, Saudi Arabia

Kumar Chandan Srivastava

Department of Oral Medicine & Radiology, Saveetha Dental College, Saveetha Institute of Medical and Technical Sciences, Saveetha University, Chennai, 602105, India

You can also search for this author in PubMed   Google Scholar

Contributions

“Conceptualization, G.K.; A.R.T.; D.S.; and K.C.S.; methodology, G.K.; A.R.T.; S.K.N.; D.S.; and K.C.S.; software, R.S.S.; E.V.; R.R.P.; S.K.N.; and D.S.; validation, D.S.; H.A.G.; A.Y.S.; O.S.O.; and K.C.S.; formal analysis, R.S.S.; E.V.; R.R.P.; and K.C.S.; writing—original draft preparation, G.K.; A.R.T.; S.K.N.; D.S.; and K.C.S.; writing—review and editing, G.K.; A.R.T.; R.S.S.; E.V.; R.R.P.; S.K.N.; D.S.; H.A.G.; A.Y.S.; O.S.O.; and K.C.S.; Supervision, S.K.N.; Project Administration, K.C.S.; funding acquisition, A.R.T.; and K.C.S. All authors have read and agreed to the published version of the manuscript.

Corresponding authors

Correspondence to Anchu Rachel Thomas , Deepti Shrivastava or Kumar Chandan Srivastava .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary materials 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kaur, G., Thomas, A.R., Samson, R.S. et al. Efficacy of electronic apex locators in comparison with intraoral radiographs in working length determination- a systematic review and meta-analysis. BMC Oral Health 24 , 532 (2024). https://doi.org/10.1186/s12903-024-04259-w

Download citation

Received : 24 December 2023

Accepted : 13 April 2024

Published : 04 May 2024

DOI : https://doi.org/10.1186/s12903-024-04259-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Root canal treatment
  • Working lengt
  • Radiographs
  • Apex locator

BMC Oral Health

ISSN: 1472-6831

systematic review pico tools

  • Search Menu
  • Advance articles
  • Editor's Choice
  • Supplement Archive
  • Article Collection Archive
  • Author Guidelines
  • Submission Site
  • Open Access
  • Call for Papers
  • Why Publish?
  • About Nutrition Reviews
  • About International Life Sciences Institute
  • Editorial Board
  • Early Career Editorial Board
  • Advertising and Corporate Services
  • Journals Career Network
  • Self-Archiving Policy
  • Journals on Oxford Academic
  • Books on Oxford Academic

Article Contents

Introduction, acknowledgments, supporting information, mapping the evidence of novel plant-based foods: a systematic review of nutritional, health, and environmental impacts in high-income countries.

ORCID logo

  • Article contents
  • Figures & tables
  • Supplementary Data

Sarah Nájera Espinosa, Genevieve Hadida, Anne Jelmar Sietsma, Carmelia Alae-Carew, Grace Turner, Rosemary Green, Silvia Pastorino, Roberto Picetti, Pauline Scheelbeek, Mapping the evidence of novel plant-based foods: a systematic review of nutritional, health, and environmental impacts in high-income countries, Nutrition Reviews , 2024;, nuae031, https://doi.org/10.1093/nutrit/nuae031

  • Permissions Icon Permissions

Shifting from current dietary patterns to diets rich in plant-based (PB) foods and lower in animal-based foods (ABFs) is generally regarded as a suitable strategy to improve nutritional health and reduce environmental impacts. Despite the recent growth in supply of and demand for novel plant-based foods (NPBFs), a comprehensive overview is lacking.

This review provides a synthesis of available evidence, highlights challenges, and informs public health and environmental strategies for purposeful political decision-making by systematically searching, analyzing, and summarizing the available literature.

Five peer-reviewed databases and grey literature sources were rigorously searched for publications.

Study characteristics meeting the inclusion criteria regarding NPBF nutrient composition and health and environmental outcomes in high-income countries were extracted.

Fifty-seven peer-reviewed and 36 grey literature sources were identified; these were published in 2016–2022. NPBFs typically have substantially lower environmental impacts than ABFs, but the nutritional contents are complex and vary considerably across brands, product type, and main primary ingredient. In the limited evidence on the health impacts, shifts from ABFs to PB meats were associated with positive health outcomes. However, results were mixed for PB drinks, with links to micronutrient deficiencies.

If carefully selected, certain NPBFs have the potential to be healthier and nutrient-rich alternatives to ABFs and typically have smaller environmental footprints. More disaggregated categorization of various types of NPBFs would be a helpful step in guiding consumers and key stakeholders to make informed decisions. To enable informed policymaking on the inclusion of NPBFs in dietary transitions as part of a wider net-zero and health strategy, future priorities should include nutritional food standards, labelling, and subdivisions or categorizations of NPBFs, as well as short- and long-term health studies evaluating dietary shifts from ABFs to NPBFs and standardized environmental impact assessments, ideally from independent funders.

The fragile interconnection between food systems and the environment is increasingly evident. 1–3 While current agricultural practices are damaging the environment, environmental change is putting food supplies at risk of disruption if timely adaptation strategies are not used. 4–8 This relationship exists at a time when food systems are already struggling to provide healthy diets for all, with many populations experiencing a coexistence of undernutrition and obesity. 1 , 3

Structural changes in food systems are critical to both safeguard people’s health and accomplish the climate adaptation and mitigation commitments mentioned in The United Nations Framework Convention on Climate Change 9 and the United Nations’ Sustainable Development Goals. 10 While production-side strategies can contribute toward climate mitigation, substantial opportunities for further emission reductions and acceleration toward net-zero targets can be achieved through dietary changes and the resulting lower demand for foods with a large environmental footprint.

In food-secure and high-income settings, a shift from “conventional diets” (which typically contain high amounts of animal-based foods [ABFs]) to predominantly plant-based (PB) diets could improve population and planetary health. 2 , 11 Dietary change has many obstacles, with diets influenced by many factors 12 , 13 that act as barriers to increasing consumption of minimally processed PB foods (eg, legumes, vegetables). If common barriers are removed, such as the need for additional cooking skills, major changes in taste and appearance of commonly consumed dishes, and fear of social stigma, 14 , 15 novel plant-based foods (NPBFs), products designed to mimic and replace ABFs to allow easy incorporation into habitual diets (eg, vegan and vegetarian meat and dairy) (see Box 1 ), may offer an easier option to facilitate this shift.

In recent years, the NPBF landscape has expanded rapidly. Several new types of NPBFs (eg, PB drinks, yogurts, eggs, meats) were introduced to the market, and trends showed increasing sales, volume, and investment growth across many countries. 16–21 In 2023, data suggested a possible slowdown, especially for PB meats, with some consumers criticizing their cost and taste, 22 and some NPBF manufacturers reporting net losses. 23 , 24 However, sales of supermarkets’ own-label PB meat alternatives have seen growth, 23 alongside consistent increases in sales of PB dairy and eggs 25 (see Supplementary file 1, section 1.1, in the Supporting Information online for detailed information on costs).

According to a global survey focusing on individuals following vegan or vegetarian diets most or all of the time, 22.0% of consumers reported adhering to a meat-free diet, and there is growing interest in embracing PB eating, with approximately 42.0% of consumers anticipating that PB foods will replace most meat within a decade. 26 With consumption of NPBFs in the United Kingdom doubling between 2008 and 2019, particularly among women and younger generations, and the fact that in 2022, 60.0% of US households purchased at least 1 type of NPBF, verification of any health and sustainability claims in marketed products is of vital importance. 22 , 27 , 28 Currently, various NPBFs are advertised as potential dietary “game changers,” with claims that they would play an important and positive role in sustainability and health, 29 , 30 and, thus, could play a pivotal role in the so-called consumption corridors. 31 However, because of their novelty, some consumers question these positive claims. 32 Although NPBFs are generally regarded as a low-carbon alternative to ABFs, their nutrient and health profiles remain largely unknown and are often criticized. This is primarily related to concerns regarding micronutrient and protein content, along with higher content of saturated fats and sodium in comparison to ABFs, and level of processing. 33 , 34

Previous reviews have primarily focused on single aspects of NPBFs 17 , 19 , 22 , 25 , 29 , 34–46 or ingredients of NPBFs 39 , 47–50 ; a few recent reviews explored the positive health and environmental outcomes of consuming selected NPBFs. 51–53 However, research quantifying the potential impacts of NPBFs is still in its infancy, and an overview that is both systematic and comprehensive, comprising health, nutrient, and environmental outcomes from peer-reviewed and grey literature of different types of NPBFs, does not yet exist, to our knowledge. This lack makes it difficult for policy makers and consumers to assess the trade-offs between nutrient composition and the environmental and health impacts of NPBFs, and hinders the potential inclusion of NPBFs in sustainable and healthy dietary recommendations.

To synthesize available evidence, highlight challenges, inform public health and environmental strategies, and inform purposeful political decision-making, we aimed, in this study, to systematically search, analyze, and summarize the available grey and peer-reviewed literature on the nutrient composition, environmental footprints, and health effects of NPBFs sold and consumed in high-income countries, and to quantify and summarize their reported results.

The full-study protocol we followed is published elsewhere (see Nájera Espinosa et al 54 and Supplementary file 1, section 2, in the Supporting Information online for more details on the methods). Briefly, a systematic search was performed to identify peer-reviewed journal articles and grey literature that contained data on the nutrient composition, health impacts, and environmental impacts of NPBFs. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. 55

Peer-reviewed literature

Five scientific databases were systematically searched (MEDLINE, Embase, Global Health, GreenFILE, and the Web of Science Core Collection) on August 29, 2021; we conducted an updated search on June 29, 2022. The search was limited to articles published and accepted after January 2016 until June 29, 2022, because of the substantial growth in supply and demand of NPBFs in the past 7 years. 16–19 In addition to database searching, citation lists from identified systematic literature reviews were handsearched (see Supplementary file 1, section 2.6, in the Supporting Information online for the full search strategy). After the quality criteria were applied (described in Supplementary file 1 , Table S1 in the Supporting Information online ), titles were manually and triple screened. Abstracts were manually double screened after application of a supervised machine-learning algorithm (ie, a support vector machine 56 ) through Scikit Learn 57 that ranked and highlighted likely relevant articles (ie, conducted priority screening). This approach is described elsewhere in detail (see Supplementary file 1, section 2.1, in the Supporting Information online ). 58 Full texts were manually screened by 2 authors and data were also double extracted.

Grey literature

To capture grey literature in a systematic way, a manual search was conducted on Google (see Supplementary file 1, section 2.3 , and Table S3 in the Supporting Information online ). Text from the webpages was then scraped and a state-of-the-art, pretrained language model from Hugging Face 59 was used to create a summary of each web link. Results were exported into a comma-separated value, or CSV, file. Additionally, a manual search in Google of relevant websites from the top NPBF producers in the United Kingdom and United States was conducted. 60–63 And literature from relevant websites that promote NPBFs, such as the Good Food Institute and Green Queen, were searched and screened manually (see Supplementary file 1, section 2.3 , and Tables S4 and S5 in the Supporting Information online ).

Data analysis, categorization, and key definitions: nutrient, health, and environmental outcomes

The PICO (population, intervention, comparison, and outcome) criteria are defined in Table 1 (see Supplementary file 1, Table S1 in the Supporting Information online for a detailed list of the inclusion and exclusion criteria). Main study characteristics and nutrient, health, and environmental outcomes were extracted (see Supplementary files 1 and 3 in the Supporting Information online for more details).

PICO criteria for inclusion of studies

PB drinks and milk reported in 100 ml of product.

NPBFs and their ABF counterparts were categorized into food groups on the basis of their primary ingredient ( Table 2 ). See Supplementary file 1, sections 2.4 and 2.5, in the Supporting Information online for more details on the selection of nutrients, data analysis assumptions, and ABF baseline comparators). The following terms for each NPBF type are used in this review:

Food groups for novel plant-based foods and animal-based foods and their respective reported main primary ingredient

For the purposes of this review, peanuts were included in the Nuts and Seeds group because they are typically consumed as such.

Blended or mixed products, if reported, the first ingredient was taken as the primary ingredient. For example, soy & almond PB drinks were labelled as legumes.

If a product did not report any ingredients, they were categorised as unknown.

PB meat products or alternatives: include different types of PB meats (eg, PB chicken, sausages, mincemeat), categories (eg, mycoprotein, legumes), and brands

PB drink products or alternatives: include different PB drink categories (eg, legumes, nuts, seeds) and brands

PB yogurt products or alternatives: include different PB yogurt categories (eg, legumes, coconut) and brands

PB cheese products or alternatives: include different types of PB cheese categories (eg, coconut, nuts, seeds) and brands

PB egg products or alternatives: include different types of PB egg categories and brands

Mention of PB products (without further specification) refers to all the listed product subcategories mentioned, except for PB eggs.

Assessment of robustness and relevance

A modified version of the Critical Appraisal Skills Program checklist for randomized controlled trials 64 was adapted to assess robustness and relevance of the studies in the full-text reviewing stage. The modifications involved the exclusion of the randomization, blinding, and cost-effectiveness criteria on the Critical Appraisal Skills Program checklist, and funding source was added as a criterion. Studies were assessed by 4 reviewers (G.H., R.P., S.P., and S.N.E.). Studies were assessed as follows: (1) clear description of the study design, (2) appropriate comparison group, (3) clear description of the methods, (4) rigorous and clearly described analysis, (5) funding source, and (6) precision of measure of effect. Studies with a minimum score of 1 were included, and sensitivity analysis was performed by funding source (see Supplementary file 1, section 2.2, in the Supporting Information online for more details).

Fruit, vegetable, legume, and nut content in novel plant-based foods

In addition to the review component, a cross-sectional analysis was conducted to examine the total fruit, vegetable, legume, and nut content (percentage estimate) of each type of NPBF sold in the United Kingdom. For this, a time-stamped data set of observations from UK supermarkets generated by FoodDB in October 2021 was used. Details are described elsewhere 65 and in Supplementary file 1, section 2.7, in the Supporting Information online . Detailed data at the global level are not available to date; hence, this part of the analysis is limited to the United Kingdom only.

Sensitivity analysis

A common concern about studies on the health impacts and environmental sustainability of NPBFs is that they can be funded by the industry that produces them; hence, we conducted a sensitivity analysis by funding source. Furthermore, given that relative improvements in health and environmental sustainability depend on the baseline comparator used ( Supplementary file 1, section 2.2, in the Supporting Information online ), the sensitivity analysis based on the main primary ingredient of a given NPBF and its respective ABF comparator was also performed. The Wilcoxon test for sensitivity analysis with a significance level set at P  ≤ 0.05 was used.

Systematic search results

A total of 49 563 peer-reviewed and 891 grey literature records were identified from the initial search. After unique literature sources were screened, 57 peer-reviewed articles and 36 grey literature studies met the inclusion criteria ( Figure 1 ). Supplementary 1, section 2, in the Supporting Information online provides further details on the screening process. The study characteristics that were extracted included basic study details (eg, authors, year, type of study, country, number of participants, follow-up period), relevant macro- and micronutrient content (eg, those related to common deficiencies, such as iron, calcium, vitamin B 12 ), health and health proxy data (eg, obesity, micronutrient status, risk factors related to noncommunicable diseases), and environmental variables (eg, carbon, water, and land-use data).

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of systematic review process reporting nutrient composition, and environmental and health outcomes of novel plant-based products in high-income countries. Abbreviations: IPCC, Intergovernmental Panel on Climate Change.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of systematic review process reporting nutrient composition, and environmental and health outcomes of novel plant-based products in high-income countries . Abbreviations : IPCC, Intergovernmental Panel on Climate Change.

Nutrient composition of novel plant-based foods

The nutrient content of NPBFs was the most frequently studied outcome (n = 56 studies). Nutrient data were typically collected through supermarket cross-sectional surveys or manufacturers’ websites. PB meat alternatives (n = 35) and PB drink alternatives (n = 19) were most frequently reported; fewer studies researched PB cheese (n = 5) and yogurt alternatives (n = 4). No studies were found that assessed PB egg alternatives. The nutritional profile of NPBFs varied greatly by manufacturing process, including the main base ingredient (eg, soy, almond); the processing techniques, time, and temperature applied; and the type of product manufactured (ie, PB drinks, PB meats). 39 , 40 , 66 , 67

Energy density, saturated fat, fiber, sugar, sodium, and micronutrient content of plant-based meat alternatives

The 35 publications evaluating PB meat alternatives reported on 508 PB meat products with 66 ABF comparators. Where the median values for meat comparators were reported to be 221.0 kcal/100 g (interquartile range [IQR], 186.6–246.7), 5.7 g/100 g saturated fat (IQR, 3.2–7.1), and very low fiber (<0.1 g/100 g; IQR, 0.0–0.5), most meat-alternative groups were reported to have lower energy density, lower saturated-fat content, and more fiber ( Figure 2 and Supplementary file 2: Table S1 for detailed macronutrient information disaggregated by main ingredient). Mycoprotein-based meat alternatives were reported to be the least energy dense, with a median energy value of 123.0 kcal/100 g (IQR, 94.0–198.5; with ABFs, P value of difference [ P d ] < 0.001), whereas meat alternatives based on cereals and grain had the highest energy density of all PB meats (226.0 kcal/100 g [IQR, 189.8–268.5]; P d < 0.360), with values very similar to those of meat and poultry. Mycoprotein-based meats were also reported to be lowest in saturated fat (0.8 g/100 g [IQR, 0.5–1.3]; P d < 0.001), whereas nut- and seed-based meats had the highest saturated fat content (1.4 g/100 g [IQR, 1.1–1.7]; P d = 0.003) of all PB meats, which still was significantly lower than saturated fat content in meat and poultry. Finally, mycoprotein-based meat was reported to contain the highest fiber content (median, 6.0 g/100 g [IQR, 5.2–7.1]; P d  < 0.001), whereas cereal- and grain-based meats had the lowest fiber content of all PB meats (3.1 g/100 g [IQR, 2.3–3.9]; P d < 0.001), which still was significantly higher than in meat and poultry.

Macronutrient, sodium, and energy content in plant-based meat and drink alternatives in their respective food group based on main primary ingredient  (ie, predominant or core food item on the ingredient list) compared with meat and poultry, and dairy, respectively. Data were limited to raw products only. Abbreviation: M, median of each category.

Macronutrient, sodium, and energy content in plant-based meat and drink alternatives in their respective food group based on main primary ingredient   (ie, predominant or core food item on the ingredient list) compared with meat and poultry, and dairy, respectively . Data were limited to raw products only. Abbreviation : M, median of each category.

Meat and poultry contained a median of 0.5 g/100 g total sugar (IQR, 0.0–0.9) and 426.7 mg/100 g sodium content (IQR, 101.0–672.8). All PB meats contained more total sugar but had similar levels of sodium in comparison with meat and poultry. Mycoprotein-based meats had the lowest total sugar content of all PB meats (median, 0.8 g/100 g [IQR: 0.5–1.8]; P d < 0.001], and nut- and seed-based meats contained the highest total sugar amount (median, 4.2 g/100 g [IQR, 2.3–6.6]; P d = 0.002); both showed strong evidence of being higher in total sugar content than meat and poultry. This is equivalent to 0.4 g and 3.4 g of total sugar/80.0 g serving size, or, if these sugars are considered free, 1.6% and 13.4% of the maximum recommended approximately 25.0 g average daily sugar intake. 68 Finally, the median sodium values for all PB meat groups did not show strong evidence of a difference from meat and poultry, except for legume-based meats (median, 520.0 mg/100 g [IQR, 400.0–636.0]; P d = 0.011). This is equivalent to 416.0 mg of sodium (or 1.0 g of salt) per 80.0 g serving size, or 20.8% of the maximum recommended 5.0 g average daily salt intake. Moreover, there were extreme outliers, with some PB meats reported to contain more than 1400.00 mg sodium (equivalent to 2.8 g salt) per 80.0 g; thus, consumption of 1 portion of this PB meat alternatives is more than half the recommended maximum daily intake of salt. 69

Only a few studies (n = 9) evaluated micronutrient data; these reported on 250 PB meat products and 24 ABF comparators. Micronutrient content ranged vastly across all groups: whereas some products would provide substantial contributions to average daily requirements, others were much less nutritious ( Table 3 and Supplementary file 2: Table S2 ). 69–83 For example, the median iron content for cereal- and grain-based PB meats (5.4 mg/100 g [IQR, 4.2–5.4]) was higher than the median of meat and poultry (1.3 mg/100 g [IQR, 1.1–1.6]). On the contrary, vitamin B 12 levels were lower for PB meat alternatives (medians ranged from 0.1 μg/100 g [IQR: 0.0–0.9] to 0.3 μg/100 g [IQR: 0.3–0.3]) as compared with 1.2 μg/100 g (IQR: 0.6–1.6) in meat and poultry. However, certain individual products had a comparable or higher vitamin B 12 content than their ABF comparator.

Summarized micronutrient values for PB meat and drinks and animal-based foods a

Values are compared with global average daily requirements (see Supplementary file 2 in the Supporting Information online for detailed information containing all disaggregated numbers by main ingredient of each novel plant-based food and animal-based foods). The table only reports micronutrients commonly found in meat and dairy. PB products also provided other micronutrients not commonly found in meat and dairy (ie, calcium in PB meats).

Abbreviations : ADR, average daily requirement; max, maximum; min, minimum; IQR, interquartile range; PB, plant-based.

No studies reported nutrient data from organic products. Although protein levels were not the main focus of this study, protein results are reported in Supplementary file 1: Figure S2 and Supplementary file 2: Table S1 , and show that, particularly, legume- and mycoprotein-based PB meats typically match meat and poultry in protein content.

Energy density, saturated fat, fiber, sugar, sodium, and micronutrient content of plant-based drinks

The 19 studies evaluating PB drinks reported on 397 PB drinks (unflavored and unsweetened) and 52 dairy milk products. Where dairy milk comparators were reported to contain median values of 50.1 kcal/100 mL energy density (IQR, 39.3–63.0), 1.1 g/100 mL saturated fat (IQR, 0.9–2.2), and no fiber (0.0 g/100 mL; IQR, 0.0–0.0), most PB drink groups were reported to have lower energy density, lower saturated fat content, and more fiber ( Figure 2 and Supplementary file 2: Table S1 ). Coconut-based drinks were reported to be the least energy dense (median energy value, 20.0 kcal/100 g [IQR: 19.0–33.7]; P d < 0.001), whereas drinks based on cereals and grains had the highest energy density of all PB drinks (median, 59.0 kcal/100 mL [IQR: 43.0–57.0]; P d = 0.566) but not significantly higher than dairy milks. PB drinks made of cereals and grains, fruits and vegetables, and nuts and seeds were reported to be lowest in saturated fat (median, 0.2 g/100 mL; IQRs, 0.1–0.2, 0.2–0.2, and 0.1–0.3, respectively; P d < 0.001), whereas coconut-based drinks had the highest saturated fat content (median, 1.1 g/100 mL; IQR, 0.9–1.7; P d = 0.952) of all PB drinks, but this was not significantly different than dairy milks. All PB drinks contained more fiber than dairy milks; however, only the drinks based on cereals and grains, legumes, and nuts and seeds were significantly higher in fiber when compared with dairy milks (for cereals and grains, and for legumes: median, 0.5 g/100 mL [IQRs, 0.2–0.8 and 0.2–0.6, respectively]; and for nuts and seeds, 0.3 g/100 mL [IQR, 0.3–0.5]; P d < 0.001).

Dairy milks contained a median of 4.7 g/100 mL total sugar (IQR, 4.3–5.0) and 39.1 mg/100 mL sodium (IQR, 33.6–43.3). Most PB drinks contained less total sugar than did dairy milks, but they had similar levels of sodium. However, the total sugar content was only significantly lower for coconut (median, 1.9 g/100 mL; IQR, 1.5–2.5), legumes (median, 1.9 g/100 mL; IQR, 0.5–2.6), and nut- and seed-based drinks (median, 2.4 g/100 mL; IQR, 0.2–3.3) when compared with dairy milks ( P d < 0.001). This is equivalent to 3.8 g and 4.8 g of total sugar/200.0 mL serving size, or, if these sugars are considered free, 15.2% and 19.2% of the maximum recommended 25.0 g average daily sugar intake. 68 The only PB drink group that was statistically different in sodium content compared with dairy milks was the group based on nuts and seeds (median, 47.2 mg/100 mL [IQR, 34.0–60.0]; P d = 0.032). This is equivalent to 94.4 mg of sodium (0.2 g of salt) per 200.0 mL serving size, or 4.0% of the maximum recommended 5.0 g average daily salt intake. 69 However, there were also some extreme outliers, some of which reported containing more than 3 times this amount of sodium per 200.0 mL, the equivalent of approximately 12.0% of the daily World Health Organization recommendation. 69

A few studies (n = 16) evaluated micronutrient data of PB drinks, reporting on 249 PB alternative products and 37 ABF comparators. Iodine was only reported in PB drinks, not in other types of PB products. Like PB meat alternatives, micronutrient content ranged vastly across all groups: some products contributed to the average daily requirement, whereas others were much less nutritious ( Table 3 and Supplementary file 2: Table S2 ). For example, the median calcium content for all PB drink categories was 120.0 mg/100 mL (IQRs as follows: cereals and grains, 120.0–120.0; coconut, 120.0–120.0; fruits and vegetables, 120.0–120.0; legumes, 120.0–120.0; nuts and seeds, 114.5–120.0) as compared with 116.7 mg/100 mL (IQR, 109.3–124.0) for dairy milks. However, none of the PB products (median, 0.0 μg/100 mL; IQR, 0.0–1.4) matched the iodine content of dairy milks (median, 24.9 μg/100 mL; IQR, 20.0–36.5).

Only 4 studies (evaluating 29 PB drinks and 11 dairy milk products) reported nutrient data from organic PB products. All evaluated different nutrients, hence no further pooling of results was possible for organic products as a subgroup. Protein results are reported in Supplementary file 1: Figure S2 and Supplementary file 2: Table S1 in the Supporting Information online, which show that, particularly, legume-based PB drinks typically match dairy milk in protein content.

Energy density, saturated fat, fiber, sugar, sodium and micronutrient content of plant-based yogurt alternatives

The 4 studies on PB yogurt alternatives evaluated 191 PB yogurt products with 90 dairy-based comparator products (unflavored and unsweetened). The overall nutritional composition of PB yogurts appears to show some variation by main primary ingredient (see Supplementary file 2 in the Supporting Information online ); however, formal disaggregated assessment of PB yogurts by primary ingredient was not possible, because that information was often not reported by authors. At an aggregate level, PB yogurts typically contained less saturated fat and sodium but had a higher energy density and higher total sugar and fiber content.

Only 2 studies evaluated micronutrient data of PB yogurts (excluding sodium) and, therefore, no further pooling of results was possible. No studies reported nutrient data from organic PB yogurts. Protein results are reported in Supplementary file 2: Table S1 in the Supporting Information online . Only the sample of a legume-based PB yogurts came close to matching dairy yogurt in protein content.

Energy density, saturated fat, fiber, sugar, sodium and micronutrient content of plant-based cheese alternatives

The 5 studies evaluating PB cheese alternatives reported on 163 PB cheese products with 143 dairy-based comparator products. PB cheese alternatives were the least nutritionally diverse foods. Where the primary ingredient of PB cheeses was known, this was mostly coconut oil ( Supplementary file 2 in the Supporting Information online ); however, like PB yogurts, the main ingredient was often not reported by authors.

The cheese comparators were reported to contain median values of 284.0 kcal/100 g energy density (IQR, 108.0–330.1), 14.0 g/100 g saturated fat (IQR, 11.0–17.3), and no fiber (0.0 g/100 g; IQR, 0.0–0.0). Most PB cheese subgroups were reported to have higher energy densities and higher saturated fat and fiber content. PB cheese based on nuts and seeds had the highest energy density (328.0 kcal/100 g [IQR, 306.0–328.0]; P d = 0.334]), whereas coconut oil-based cheese had the highest saturated fat content (21.0 g/100 g [IQR, 19.7–22.0]; P d < 0.001]), a significant difference, with 50.0% more than dairy cheese. Unlike PB drinks, PB meat, and PB yogurt alternatives, not all PB cheese contained fiber. Nut- and seed-based cheese had the highest fiber content (median, 2.5 g/100 g [IQR, 2.4–2.7]; P d < 0.001). Although the median fiber content of PB cheese made from coconut oil was 0.0 g/100 g (IQR, 0.0–1.7; P d = 0.011), some products did contain up to 5.9 g/100 g and, therefore, strong evidence was found that both PB cheese based on nuts and seeds and on coconut oil had significantly higher fiber content than did dairy cheese.

Most PB cheese contained less sugar and sodium than did dairy cheese, which had a median of 2.0 g/100 g (IQR, 0.5–5.0) and 720.0 mg/100 g (IQR, 560.0–1000.0), respectively, across the identified studies. In general, PB cheese alternatives had either no or minimal total sugar content. Finally, coconut oil–based cheese had the highest sodium content across all PB cheese (median, 714.0 mg/100 g [IQR, 600.0–880.0]; P d = 0.897), but this was similar to dairy cheese. PB cheese made of nuts and seeds had the lowest median sodium content (240.0 mg/100 g [IQR, 200.0–240.0]; P d = 0.001), which would equal 48.0 mg of sodium (0.1 g of salt) per 20.0 g serving size, or 2.0% of the recommended maximum daily salt intake 69 ; hence, this type of PB cheese had a large reduction in sodium compared with dairy cheese.

The micronutrient content of PB cheese was evaluated by only 2 studies. Only 1 product made of nuts and seeds was fortified with calcium, whereas coconut-based PB cheese was typically fortified with vitamin B 12 (median, 2.5 μg/100 g; IQR, 2.5–2.5). For dairy cheeses, these medians were 815.0 mg/100 g (IQR, 463.0–930.0) for calcium and 2.5 μg/100 g (IQR, 1.8–2.5) for vitamin B 12 .

No studies reported nutrient data from organic products. Protein results are reported in Supplementary file 2: Table S1 in the Supporting Information online. Nut- and seed-based cheese typically had the highest protein content, though it did not match the protein content of dairy cheese.

Health impacts and risk factors of novel plant-based foods

Eleven peer-reviewed studies were included in this review, 9 of which evaluated PB meat alternatives and 3 evaluated PB drinks ( Table 4 ) 84–94 (see Supplementary file 1, section 3.3, in the Supporting Information online for further details on the health outcomes). No health studies were found that evaluated consumption of PB cheese, yogurt, or egg alternatives; links between NPBFs and mental health outcomes; nor any grey literature evaluating any health outcomes.

Summary of the evidence on the health impacts and risks of novel plant-based foods

Health impacts and risk factors of plant-based meat alternatives

Studies of PB meats (n = 9) showed positive health outcomes when individuals switched from consuming ABFs. Three studies on mycoprotein consumption by both healthy and overweight adults found a positive association with lower glycemic markers, 84 reduced energy intake, 84 , 85 and insulin release. 85 Moreover, mycoprotein consumption was hypothesized to have a beneficial impact on the plasma lipidome. 86

Four studies with healthy adults evaluated PB meat alternatives consumption (other than mycoprotein). When considering the same caloric intake, consumption of PB meats was associated with a lower risk of cardiovascular disease than was consumption of ABFs, mostly by reducing fasting serum levels of trimethylamine- N -oxide, and low-density lipoprotein cholesterol concentrations, compared with ABF consumption. 87 Furthermore, consumption of PB meats was associated with a reduction in body weight as compared with meat consumers. 87 , 88 Lysine-enriched PB meat as a substitute for ABFs was reported to increase muscle protein synthesis rates, which is a biological process of building new protein cells via amino acids. 89 Last, the replacement of 4 meat-containing meals per week with PB meat alternatives elicited positive changes in the gut microbiome, with changes in the presence of butyrate-producing pathways and increased taxa. 90

Health impacts and risk factors of plant-based drinks

Studies assessing PB drinks (n = 3) only focused on almond and soy drinks. The main focus and health outcomes of these studies varied. Sun et al 91 researched the reduction in glycemic response in young adults consuming soy drink or bovine milk together with white bread. These authors found that both products had a similar glycemic response through different biological pathways. Dineva et al 92 assessed micronutrient content in PB drinks and found significantly lower iodine intake and urinary iodine concentration in people consuming only PB drinks, 93 highlighting the need for appropriate fortification as more people transition to eat more NPBFs. Finally, Shen et al 93 evaluated the impact of PB drinks on dental health and found that a soy drink with added sugar caused enamel demineralization, compared with dairy milk, which promoted remineralization.

Environmental impacts of novel plant-based foods

A total of 53 studies evaluated at least 1 environmental outcome, using the life cycle assessment method, evaluating 209 PB products and 91 ABFs as comparators. Most studies used life cycle assessment inventories, and some relied on data providers (n = 32) to calculate environmental footprints. System boundaries varied across studies, with the majority evaluating category impacts from cradle-to-retail (see Supplementary file 3 in the Supporting Information online ). Studies mainly assessed the effect of substituting ABFs with NPBFs on greenhouse gas emissions (GHGE) (n = 50), followed by blue-water footprint (WF) (n = 39) and land use (LU) (n = 17) ( Figure 3 and Supplementary file 1: Table S11 in the Supporting Information online ). Although methods, assumptions, and inventory data varied from 1 study to another, most studies consistently reported percentage reductions in GHGE and LU for the production of NPBFs as compared with ABFs. Wider differences were observed in blue WF.

Reduction of environmental impacts by respective funding source. Calculated as a percentage difference between each novel plant-based (PB) product (by product type and food group based on main primary ingredient [ie, predominant or core food item on the ingredient list]) in comparison with their respective reported baseline (eg, dairy milk and cheese, meat and poultry). See Supplementary file 3 in the Supporting Information online for detailed information on the baseline used for each reference. Data were limited to raw products only. Studies reporting data on cooked PB products also found reductions in environmental impacts.

Reduction of environmental impacts by respective funding source . Calculated as a percentage difference between each novel plant-based (PB) product (by product type and food group based on main primary ingredient [ie, predominant or core food item on the ingredient list]) in comparison with their respective reported baseline (eg, dairy milk and cheese, meat and poultry). See Supplementary file 3 in the Supporting Information online for detailed information on the baseline used for each reference. Data were limited to raw products only. Studies reporting data on cooked PB products also found reductions in environmental impacts.

Environmental footprints of plant-based meat alternatives replacing meat and poultry

The 34 publications evaluating PB meat alternatives reported on 135 PB meat products with 53 ABF comparators. The percentage difference showed reductions of more than 70% in GHGE, LU, and WF for most products when shifting from ABFs to PB meat alternatives. GHGE reductions across PB meat groups, based on primary ingredients, were similar, with the largest reduction in GHGE seen for nut- and seed-based meats, with a median value of –94.2% (IQR, –94.4 to –93.4), whereas PB meats based on legumes had the smallest reduction (–86.1%; IQR, –88.6 to –77.5). Only 2 of 134 PB products had higher levels of GHGE than their ABF comparator. For LU, mycoprotein (median, 89.0%; IQR, –92.3 to –76.5) and nut- and seed-based meats (median, 89.5%; IQR, –90.0 to –89.0) had the largest reduction. Alternatively, legume-based meats had the smallest LU reductions (median, –71.2%; IQR, –84.7 to –47.6). Only 3 of 55 products had higher LU than their ABF comparator. Finally, the largest reduction of WF was observed in PB meats made of cereals and grains (median, –92.6%; IQR, –94.1 to –92.0), and the smallest was observed with products made of mycoprotein (median, –73.7%; IQR, –84.4 to –55.2). Nine of 51 products had a higher WF than their respective ABF counterparts. Specifically, when certain individual legume- and mycoprotein-based meats were compared with chicken, PB meat alternatives reported requiring between 2.7% and 339.0% more water, with the largest difference observed in a Swedish chicken comparator to mycoprotein-based meats. This variation was attributed to differences between feed types, rearing systems, and farm efficiency across countries. 74 Comparisons were also made between the upper limit footprint of mycoprotein-based items and the average or lower limit footprint of the ABF. Moreover, there were extreme outliers, with some PB meats reporting a water percentage difference of 8006.9%. The authors attributed this to soybeans’ substantial water demand during processing and lower yield per soybean. 74

Environmental footprints of plant-based drinks alternatives replacing dairy milk

The 21 publications evaluating PB drinks reported on 51 PB drink products with 13 ABF comparators. PB drinks also were associated with reductions in GHGE and LU when shifting from dairy milk to PB drinks. Fruit- and vegetable-based drinks had the largest reduction of GHGE (median, –90.2%; IQR, –90.8 to –90.2]), whereas PB drinks based on cereals and grains had the smallest reduction (median, –76.9%; IQR, –88.8 to –56.0). Only 2 products of 36 had an increase of GHGE when comparing soy- (40.0%) and almond-based (18.9%) drinks with dairy milk (equivalent to 0.3, 0.4, and 0.3 kg CO 2 eq/100 g, respectively). 95 Wider differences were observed on the LU percentage difference; however, reductions were found for all products (n = 13 PB drinks).

Cereal- and grain-based drinks had the largest reduction (median, –86.4%; IQR, –92.7 to –76.0), whereas legume-based drinks had the smallest LU reductions (median, –56.6%; IQR, –75.5 to –38.8). The magnitude of change in the percentage difference for WF varied considerably, although, these data were less frequently reported by authors (n = 11 PB drinks). Cereal- and grain-based drinks had the largest reduction (median, –85.0%; IQR, –88.7 to –71.0), whereas legume-based drinks had the smallest WF reductions (median, –67.6%; IQR, –73.9 to –42.2). Nut- and seed-based drinks presented contradictory evidence. For example, Grant and Hicks 95 observed that almond drinks (9241.9%) required considerably more water than soy (–35.6%) and dairy milks (equivalent to 109.3, 0.8, and 1.2 L/100 g, respectively); whereas Ritchie 96 found that an almond drink required half the amount of water (–40.87%) than dairy milk (equivalent to 37.2 and 62.8 L/100 g, respectively). Data were limited to these 2 products; hence, no further pooling of results was possible.

Environmental footprints of plant-based yogurt alternatives replacing dairy yogurt

The 2 publications evaluating 2 PB yogurt alternatives compared to 2 dairy yogurts. They reported GHGE reductions ranging between –64.7% and –52.9%. Analysis of LU and WF was not possible due to lack of a baseline, differences in methods, and system boundaries.

Environmental footprints of plant-based cheese alternatives replacing dairy cheese

The 2 publications evaluating PB cheese alternatives reported on 21 PB cheese products with 23 ABF comparators. Data on the environmental impacts were particularly from coconut oil–based cheese alternatives (n = 20). All coconut oil–based cheese alternatives had a large reduction in amounts of GHGE and LU (GHGE: median, –75.4% [IQR, –77.4 to –59.3]; LU: median, –83.1% [IQR, –83.8 to –80.6]). A smaller reduction was observed in WF (median, –45.1%; IQR, –52.0 to 38.5), with a higher WF being reported than for the ABF comparator for only 3 products.

Health effects and environmental impacts of novel plant-based foods

Studies that simultaneously assessed both health and environmental outcomes and/or nutrient profiles of NPBFs were pooled ( Figure 4 ). Only 1 study reported environmental outcomes together with diet-related health effects of PB meat alternatives, and this study found that free access to NPBFs was associated with greater weight loss and reduced dietary carbon and LU, as compared with a control arm. 88 From 93 references, 20 studies assessed the environmental outcome and nutrient content of NPBFs; only 6 studies evaluated the health effects and nutrient content of NPBFs (see Supplementary file 1: Table S9 in the Supporting Information online ).

Reduction of environmental outcomes and their associated nutrient outcomes of novel plant-based foods (NPBFs) compared with baseline (eg, dairy milk and cheese, meat and poultry), expressed in percentage difference. The y-axis shows the increase or decrease of the nutrient content (energy, fiber, sodium, and saturated fat) in comparison with baseline; and the x-axis shows the reduction (or increase) of the environmental categories. Three environmental categories are reported: greenhouse gas emissions (circles), land use (triangles), and blue-water use (squares). Three NPBFs are reported: plant-based (PB) cheese alternatives (brown), PB meat alternatives (purple), and PB drinks (orange). PB yogurts were not included due to the limited amount of data. See Supplementary file 2 in the Supporting Information online for detailed information on the baseline used for each reference. Data were limited to raw products only.

Reduction of environmental outcomes and their associated nutrient outcomes of novel plant-based foods (NPBFs) compared with baseline (eg, dairy milk and cheese, meat and poultry), expressed in percentage difference . The y -axis shows the increase or decrease of the nutrient content (energy, fiber, sodium, and saturated fat) in comparison with baseline; and the x -axis shows the reduction (or increase) of the environmental categories. Three environmental categories are reported: greenhouse gas emissions (circles), land use (triangles), and blue-water use (squares). Three NPBFs are reported: plant-based (PB) cheese alternatives (brown), PB meat alternatives (purple), and PB drinks (orange). PB yogurts were not included due to the limited amount of data. See Supplementary file 2 in the Supporting Information online for detailed information on the baseline used for each reference. Data were limited to raw products only.

When compared with ABF counterparts, data suggest NPBFs are overwhelmingly associated with smaller environmental footprints. Data on nutritional profiles of NPBF were mixed: nutritional profiles for some NPBF groups were better aligned with healthy diets, but not for others. Clear co-benefits were observed for fiber intake from NPBFs. However, for the other nutrients, the picture was much more mixed due to the variability in content arising from differences in the main primary ingredients and the type of NPBFs.

Fruit, vegetable, legume, and nut content of novel plant-based foods

The percentage of fruit, vegetable, legume, and nut content in each NPBF in the United Kingdom was estimated as a case study ( Figure 5 ). Most NPBFs had at least 1 fruit, vegetable, legume, or nut, ranging from 0.0% to 100.0% of their weight. Overall, median content was low, with a few exceptions. PB meat alternatives had the highest content of vegetables and legumes, and PB cheese alternatives had the lowest content ( Supplementary file 1: Figure S5 and Supplementary file 2: Table S5 in the Supporting Information online ).

Estimated fruit, vegetable, legume, and nut content (%) in each novel plant-based foods product from time-stamped data from UK supermarkets. Panels show (a) plant-based (PB) drink alternatives; (b) PB meat alternatives; (c) PB cheese alternatives; and (d) PB yogurt alternatives.

Estimated fruit, vegetable, legume, and nut content (%) in each novel plant-based foods product from time-stamped data from UK supermarkets . Panels show (a) plant-based (PB) drink alternatives; (b) PB meat alternatives; (c) PB cheese alternatives; and (d) PB yogurt alternatives.

Assessment of robustness and relevance of the included studies

For results on the assessment of robustness and relevance of the included studies see Supplementary file 1 : Table S12 in the Supporting Information online in section 3.6 .

Sensitivity analysis of funding sources of nutrient composition studies

Almost half of the nutrition studies included (n = 26; 46.4%) were funded by academic funders; 44.6% (n = 25) were fully funded or partially funded by industry; and 10.0% (n = 5) did not state their funding source. NPBF manufacturers were the support for the majority of industry-funded studies (n = 21; 37.5%), followed by the livestock industry (n = 3; 5.4%), and both (n = 1; 1.8%). The sensitivity analysis of the percentage difference for all the nutrients associated with the burden of disease, except total sugar, revealed that studies funded by industry were more likely to find differences than those funded by academia, with the former typically reporting more positive results on lower energy and saturated fat ( Table 5 and see Supplementary file 1: Table S13 in the Supporting Information online for sensitivity analysis on studies partially funded by the industry). However, the direction across all studies was the same: reductions in energy and saturated fat content, and increases in fiber, total sugar, and sodium content.

Sensitivity analysis, based on funding source, of the percentage difference between novel plant-based foods vs animal-based foods in nutrient content and environmental impacts a

The funding source of 6 articles were unknown, so they were excluded from this analysis. The superscript b and c indicate the direction and dimension of the association.

Industry-funded studies show a more positive impact on health and environmental outcomes of their PB products (vs animal sourced foods) as compared with academically funded studies.

Industry-funded studies show a less positive impact on health and environmental outcomes of their PB products (vs animal-sourced foods) as compared with academically funded studies.

Abbreviations : ABF, animal-based food; IQR, interquartile range; NPBF, novel plant-based food.

Sensitivity analysis of funding sources of health studies

Only 2 health studies were funded by academia; the rest of the studies were either partially or wholly funded by industry (n = 9). Most industry-funded studies were from NPBF manufacturers (n = 8); 1 study was partially funded by Dairy Australia.

Sensitivity analysis of funding sources of environmental studies

Compared with nutritional studies, a greater percentage of environmental studies were by industry researchers, particularly from NPBF manufacturers (67.9%). Approximately 71.7% of studies (n = 38) were fully funded or partially funded by industry; 26.4% (n = 14) were supported by academic funders; and 1.9% (n = 1) did not state their funding source. Of the industry-funded studies, only 2 (3.8%) were funded by the livestock industry. The sensitivity analysis revealed that the percentage differences were significantly larger between academic and industry funders in terms of GHGE and LU. Studies funded by industry typically reported more positive results on LU than did studies funded by academic funders, and the opposite was observed for GHGE. Like nutrient studies, the direction (decreases in GHGE, LU, and WF) was the same regardless of the funding source ( Table 5 , and see Supplementary file 1: Table S13 in the Supporting Information online for the sensitivity analysis of studies partially funded by the industry).

Research findings

We reviewed evidence from high-income countries that was published in peer-reviewed and grey literature within the past 7 years on nutrient content, and environmental and health outcomes of consuming NPBFs. Most NPBFs typically have much lower environmental impacts compared with ABFs, particularly with respect to GHGE and, to a lesser extent, to LU and WF. The nutrient content of NPBFs is highly variable in comparison to the nutrient profiles of ABFs. Although several individual NPBFs had positive health and environmental outcomes, co-benefits identified were not universal across all NPBFs and several trade-offs were identified. The main primary ingredient, type of product, processing techniques, and brand were all important determinants of health, and nutritional and environmental outcomes, findings that show the need for further subcategorization of NPBFs to better educate consumers and enable them to take informed decisions regarding the healthiness and sustainability of their diets and (potential) dietary changes.

Research in context

If carefully selected, certain NPBFs (particularly certain PB drinks and meat alternatives) could be an effective part of interventions to achieve net-zero and health targets in high-income countries. By applying a combination of strategies, enhanced uptake of these foods could improve the nutritional quality of diets, improve health, and contribute to tackling climate change impacts.

At the macronutrient level, NPBFs are generally the healthier option, given their higher fiber content and typically lower saturated fat and calorie contents, which could be advantageous in high-income (often obesogenic) settings. Certain types of NPBFs, particularly mycoprotein and legume-based meats, often also contain a substantial amount of fruit, vegetables, legumes, and/or nuts, which are food groups that are typically underconsumed in high-income settings. Composition of legume and fruit and vegetable-based drinks, were also typically consistent with healthier diets in high-income food secure settings, including low energy density, low total sugar, high fiber and low saturated fat content. Caution is recommended in the selection of these products if they were to be part of dietary recommendations, or standard institutional procurement for example, as certain NPBFs can also have higher levels of total sugar, sodium, and saturated fats in comparison to their respective ABF. This is particularly true for certain cereal and grain-based drinks, and coconut-based cheese and yogurts. Although the specific type of oil used in each NPBF product was not analyzed, coconut oil, which is high in saturated fatty acids, is often the ingredient that increases saturated fat levels in NPBFs to levels similar to its ABF counterparts. 51 , 75 Indeed, coconut oil-based cheese had approximately 50% more saturated fat than dairy cheese, and typically contained the least amount of fruit, vegetables, legumes or nuts, with the majority being absent.

In line with other evidence, 39 , 97 , 98 fortified NPBFs, in some cases, can be nutritionally comparable to their respective ABFs. Some individual NPBFs contained even higher concentrations of iron, vitamin B 12 , and calcium, whereas others did not. However, micronutrient assessment was difficult because not all included studies reported micronutrients. This could be because either NPBFs were unfortified or the information simply was not reported. Especially when nutrient information is gathered from supermarket websites for individual studies, micronutrient data are generally not reported.

The highly varying nutrient content across and within all PB products and categories may cause consumer confusion when individuals are looking for healthy and environmentally friendly alternatives to ABFs. Clearer front-of-package labelling of certain nutrients and information campaigns could reduce such confusion and better enable the consumer to make informed decisions about food purchases. 99 Potential development of rules and regulations on the food standards of NPBFs could also be a step forward in having a larger range of “healthy” NPBFs, because such regulations could potentially encourage reformulation of NPBFs, including the reduction of sodium, total sugar, and saturated fat content, and increased micronutrients. From a technological perspective, this is certainly possible. For example, new biotechnological techniques have been developed that enable companies to reduce sugar content and improve palatability, nutrient profile, and digestibility of PB drinks. 67 , 100–103 Some processing techniques can also decrease levels of anti-nutrients and polyphenols, which commonly are associated with low mineral and vitamin bioavailability, 35 , 98 , 101 , 104–107 and increase protein yield. 101 Given that specific raw materials, isolated proteins, processing levels, and fortification methods, often used in NPBFs, as compared with ABF nutrient profiles, are still debated in the scientific community, further research on the nutrient content and health risks related to bioavailability, bioaccessibility, and byproduct formation during industrial processes will reveal whether there are differences in terms of health impacts of “natural” vs more “isolated” nutrients. 30 , 108 , 109 More research into the metabolic profiles of NPBFs is imperative, particularly in light of a recent study identifying differences in the abundance of profiled metabolites between beef and PB burgers, despite their labelled nutritional similarities. 110 Instead of continuing the debate between the superiority of ABFs vs NPBFs, or vice versa, acknowledging and embracing their complementary differences can contribute to a less polarized dietary transition. This is especially relevant because emerging evidence has suggested that people who consume NPBFs also tend to purchase ABFs. 111

From the limited evidence on health, the inclusion of NPBFs into diets appears to typically have beneficial health effects, particularly the consumption of PB meat alternatives. The positive health effects mostly relate to better weight management and associated reduced risk of noncommunicable diseases in high-income (and often obesogenic) countries. This is aligned with a recently published meta-analysis that found positive outcomes on total cholesterol, low-density lipoprotein cholesterol and triglycerides when consuming PB meat alternatives as replacements for meat. 51 Furthermore, a few older studies also found positive health outcomes when assessing consumption of mycoprotein-based foods (eg, drinks, cookies, milkshakes, crisps) 112–115 and soy protein with isoflavones, 50 compared with consumption of dairy milk and/or meat products.

Previous evidence revealed that NPBFs are often regarded as healthier alternatives to ABFs 116 ; hence, it could be hypothesized that people may consume NPBFs in larger quantities than they would otherwise have done when eating ABFs. This may have negative health implications, especially if consumed regularly. Establishing a clear division in PB foods classifications, including ultraprocessed and less processed PB alternative foods, could enable better assessment of short- and long-term health impacts of NPBFs if they were to be consumed at an even larger scale. 116

Ultraprocessed foods have been associated with many diet-related diseases because these foods are generally energy dense and hyperpalatable. 117 , 118 Almost all NPBFs fall, technically, within this category; however, in this review, we found that the nutritional composition of some NPBFs aligns well with healthy dietary recommendations, such as having a high fiber content, low energy density, and low saturated fat content. Additionally, 1 of the included studies 90 also found positive associations with the gut microbiome when substituting meat in certain meals with PB meat alternatives. To get a better overview of the overall effect of NPBFs on health, more information and detailed analyses are needed regarding level of processing and gastrointestinal fate.

Consistent evidence was found regarding environmental outcomes, similar to previous research. 52 , 53 , 108 , 119–121 Most NPBFs had smaller environmental footprints than their ABF counterparts, with median reductions reported of up to 94.3%, 89.5%, and 92.6% for GHGE, LU, and WF, respectively. Nevertheless, some PB products had greater environmental impacts than their ABF counterparts, with some extreme outliers particularly in terms of WF. Although evidence was rather consistent, and the direction of effect appears to be clear, care should be taken not to overinterpret the exact numerical results: environmental impact calculations are notoriously context dependent and sensitive to methodological and data choices. This makes it impossible to come up with a summary figure that is representative for all products, produced in all countries; generally, however, there is a broad body of evidence demonstrating a reduction in GHGE, LU, and WF for a wide range of PB products in a wide variety of contexts compared with their ABF equivalents.

To improve the strategic use of NPBFs to achieve more sustainable food systems, life cycle assessments of these products should incorporate the full range of environmental impact categories, as well as sociocultural, economic, and health impacts with harmonized methods and assumptions across studies.

This study revealed an evidence gap for health impacts of NPBFs, including mental and dental health, and other risks associated with micronutrient deficiencies. There is also a lack of health studies on PB yogurts, PB cheese, and PB egg alternatives. Research on the health effects of PB drinks has been conducted with only certain products, “generally soy and almond drinks,” but there is a gap in knowledge about other PB drinks, such as those made from oat, potato, and hazelnut, among others. Furthermore, some concerns have been raised about the carbohydrate content in some PB drinks. A study by Jeske et al 122 revealed that the presence of β-glucan in many oat-based drinks causes a moderate glycemic index, despite the high carbohydrate content. In fact, Dhankhar 104 associated the consumption of oat drinks with high β-glucan levels with a reduction in cholesterol levels in study participants. However, this evidence needs to be updated to reflect the potential benefits of different types of PB drinks and current market brands. Although dairy products contain naturally occurring sugars from lactose, it is difficult to determine the breakdown of “natural” vs added sugars in NPBFs from the available literature. More research is also required on dental health to assess the potential risks of increased dental cavities due to lower calcium bioavailability, and the effects of free sugar content, pH levels, and buffering capacity in NPBFs.

Additional research is needed to provide more nutrient environmental and health evidence for PB yogurts, cheese, and egg alternatives. Last, although this review assessment focused on 3 environmental outcomes, evidence on other environmental impacts, including biodiversity loss and socioeconomic implications, is scarce. Across the 3 themes assessed in this review, better standardization and clear reporting of results in NPBF studies in the future would facilitate updates of this review.

Relevance for policy and practice

Minimally processed PB foods are still considered the gold standard for healthier and more sustainable diets. However, shifts from ABFs to PB whole foods remain problematic because, despite all the scientific knowledge about healthy eating, dietary change toward minimally processed PB foods has not been achieved. This review revealed that NPBFs can be healthier and more environmentally friendly alternatives to ABF consumption, if carefully selected. Although behavioral aspects are embedded in this transition, NPBFs could offer a convenient, novel, and potentially more realistic option to facilitate dietary transitions at large scale, diversifying diets, and increasing consumption of fruits, vegetables, legumes, and nuts without the need for significant individual dietary habits.

For potential promotion of the inclusion of NPBFs as part of public procurement or embedding them into food-based dietary guidelines, some of the consideration regarding varying healthiness of specific types of NPBFs and the need for further subclassifications needs to be carefully addressed. Furthermore, affordability is a concern because NPBFs often are more expensive than their ABF counterparts. Although comprehensively synthesizing price data was outside of the scope of this study, in the United Kingdom, the Food Foundation found that PB drinks are, on average, 50.0% more expensive than dairy milk. 71

Active promotion of NPBFs would require more detailed analysis of consumer behavior: current consumption of NPBFs is generally higher among younger generations, women, White populations, and those with higher education and incomes. 28 Better understanding of main drivers and barriers of consumption of NPBFs would allow targeted promotion to widen this consumer group. 71 NPBFs could play an additional role in reducing the prevalence of micronutrient deficiencies, especially given their reformulation and fortification potential. For example, in Finland, a mass fortification strategy of vitamin D across dairy and nondairy products has shown positive health outcomes over the past decade. 123 Finally, formalization, standardization, and accountability of environmental labelling could help consumers making informed decisions and avoid misinformation.

Strengths and limitations

To our knowledge, this is the first systematic review assessing the published peer-reviewed and grey literature evidence from studies that evaluated nutrient, and health and environmental impacts or benefits of NPBFs. A strict and comprehensive search string was developed to assess the full breadth of studies and reports, and machine-learning models were used to filter the large number of studies and systematically present all the available evidence on various NPBFs.

This study only covered the past 7 years to assess the current evidence, and an exhaustive cross-check of references was not performed, which possibly introduces reporting bias for missed relevant studies from previous years. However, it was assumed that only a small amount of additional findings had been missed, given the recent emergence of the variety and types of these novel products. Second, only 3 environmental impact categories were examined: carbon footprint, LU, and blue-water consumption. However, the heterogeneity of study designs, from system boundaries to geographical location, agricultural inputs, and methods used to calculate environmental footprints, made the review process too time consuming to expand on other environmental impacts in this particular study. Reliable reporting of environmental impacts of novel ingredients used in NPBFs, including added minerals and vitamins for fortification purposes, are generally missing in many studies. All the data reported by authors were collected and each study was compared individually against its own baseline (ie, the ABF comparator provided by author). Given the large spectrum of methods to determine environmental footprints, this could have introduced some bias; however, the alternative (using a standardized comparator) would equally have its limitations (eg, this would not be representative for all farming systems and products). Third, products and nutrients were assessed individually. Although the nutrient content gives some guidance on probable health risks, in reality, people consume diets in which individual compounds interact, influencing unknown biological pathways. Fourth, several studies that did not specifically report on the proportion and type of NPBF in (self-)reported PB diets had to be excluded. For those studies, it was impossible, therefore, to assess the effect on health and environment of NBPFs alone vs all PB foods together (ie, whole foods, NPBFs, other PB foods such as tofu and tempeh) and complicated any efforts to calculate dietary shifts. Finally, most studies did not report the precision of measures of effect (n = 68), making it difficult to pool and synthesize results across the 3 themes assessed in this review.

Food systems and diets need to change to meet environmental and health targets. This comprehensive systematic review presents a holistic approach to summarize the evidence on the nutrient, health, and environmental impacts of NPBF consumption. Although PB whole foods remain the preferred option on health grounds, some NPBFs have potential for being a useful steppingstone in the process of food system and dietary transformation, functioning as a healthy and environmentally friendly alternative to ABFs, if carefully selected. Reformulation and fortification could further enhance NPBFs as a viable and effective food group that could accelerate the dietary transition toward sustainable and healthy diets. However, given the great variability in nutritional composition of individual NPBFs, widespread promotion of such products should be introduced and addressed with caution. Given that NPBFs are already important in the food system and consumption is expected to continue to increase, a few steps are urgently required to guide consumers and enable them to make informed decisions regarding their diets. These include a further subdivision or categorization of NPBFs, which currently fall mainly in the ultraprocessed (hence, “unhealthy”) food category. Furthermore, standardized and verifiable environmental assessments of NPBFs are needed to compare foods with regard to their environmental footprints. Finally, more research on the short- and longer-term health effects of NPBFs is urgently required to facilitate informed decision-making on the inclusion of NPBFs as part of a wider net-zero and health strategy.

Gratitude is extended to the authors who responded to inquiries and generously shared their individual data. Additionally, sincere appreciation is expressed to the FoodDB team for sharing their time-stamped data set of observations from UK supermarkets to estimate the total fruit, vegetable, legume, and nut content of foods.

Author contributions. S.N.E. contributed to conceptualization of the study, methodology, investigation, data curation, formal analysis, coding and analysis of machine learning, and writing the original draft of the article. G.H. contributed to the formal analysis (screening process), data validation, and review and editing of the manuscript. A.J.S. led the coding and analysis of machine learning, and reviewed and edited the manuscript. C.A.-C. and G.T. contributed to the literature screening process, and reviewed and edited the manuscript. R.G. reviewed and edited the manuscript. S.P. and R.P. contributed to data validation, and reviewed and edited the manuscript. P.S. contributed to conceptualization of the study, methodology, reviewed and edited the manuscript, supervised the work, acquired funding, and contributed to project administration.

Funding. This work was supported by a research grant from the National Institute for Health Research, Health Protection Research Unit PhD Studentship in Environmental Change and Health (grant NIHR200909) and the af Jochnick Foundation.

The funders had no role in the conception, design, performance, and approval of this work.

Declaration of interest. The authors have no relevant interests to declare.

Data availability. Source code for this work is available online (DOI: 10.5281/zenodo.7116157).

The following supporting information is available through the online version of this article at the publisher’s website:

Supplementary file 1 .

Supplementary file 2 .

Supplementary file 3 .

Box 1 Key Definitions

Novel plant-based foods (NPBFs): Acknowledging differences in terminology for NPBFs, for the purpose of this review, the term novel plant-based foods is used to describe plant-based (PB) drinks and PB meat, cheese, eggs, and yogurt alternatives that are of plant or fungal origin and designed to directly replace or mimic animal-based foods. This definition includes fungi-based foods (ie, mycoprotein) that biologically do not belong to the plant kingdom but are typically “designed” similarly to NPBFs as a direct replacement for animal-based foods. Here, the term excludes tofu, tempeh, and seitan because although these might be novel to some high-income settings, they have been part of traditional Asian diets for centuries and, hence, are not subject to the same challenges and evidence gap as NPBFs.

Ultraprocessed: Foods that have undergone a series of industrial techniques and processes

Minimally processed plant-based foods: Plant-based whole foods such as nuts, seeds, cereals, and legumes

Food and Agriculture Organization , UNICEF, World Food Program, World Health Organization . The state of food security and nutrition in the world 2021. Transforming food systems for food security, improved nutrition and affordable healthy diets for all. 2021 . Available at: https://www.fao.org/3/cb4474en/online/cb4474en.html . Accessed February 2024.

Willett W , Rockstrom J , Jonell M , et al.    Food in the Anthropocene: the EAT-Lancet Commission on healthy diets from sustainable food systems . Lancet . 2019 ; 393 : 447 – 492 . doi: 10.1016/S0140-6736%2818%2931788-4 .

Google Scholar

Food and Agriculture Organization , UNICEF, World Food Program, World Health Organization . The state of food security and nutrition in the world 2023. 2023 . Available at: https://www.fao.org/3/CC3017EN/online/CC3017EN.html . Accessed February 2024.

Alae-Carew C , Nicoleau S , Bird FA , et al.    The impact of environmental changes on the yield and nutritional quality of fruits, nuts and seeds: a systematic review . Environ Res Lett . 2020 ; 15 : 023002 . doi: 10.1088/1748-9326/ab5cc0 .

Parajuli R , Thoma G , Matlock MD.   Environmental sustainability of fruit and vegetable production supply chains in the face of climate change: a review . Sci Total Environ . 2019 ; 650 : 2863 – 2879 . doi: 10.1016/j.scitotenv.2018.10.019 .

Wang J , Vanga S , Saxena R , et al.    Effect of climate change on the yield of cereal crops: a review . Climate . 2018 ; 6 : 41 . doi: 10.3390/cli6020041 .

Bisbis MB , Gruda N , Blanke M.   Potential impacts of climate change on vegetable production and product quality—a review . J Clean Prod . 2018 ; 170 : 1602 – 1620 . doi: 10.1016/j.jclepro.2017.09.224 .

Scheelbeek P , Bird FA , Tuomisto HL , et al.    Effect of environmental changes on vegetable and legume yields and nutritional quality . Proc Natl Acad Sci USA . 2018 ; 115 : 6804 – 6809 . doi: 10.1073/pnas.1800442115 .

Intergovernmental Panel on Climate Change ( IPCC) . Climate change 2022: impacts, adaptation and vulnerability. IPCC sixth assessment report. 2022 . Available at: https://www.ipcc.ch/report/ar6/wg2/ . Accessed February 2024.

United Nations . Sustainable development goals. Available at: https://www.un.org/en/sustainable-development-goals . Accessed June 2022.

Mendenhall E , Singer M.   The global syndemic of obesity, undernutrition, and climate change . Lancet . 2019 ; 393 : 741 . doi: 10.1016/s0140-6736(19)30310-1 .

Aggarwal A , Rehm CD , Monsivais P , et al.    Importance of taste, nutrition, cost and convenience in relation to diet quality: evidence of nutrition resilience among US adults using National Health and Nutrition Examination Survey (NHANES) 2007-2010 . Prev Med . 2016 ; 90 : 184 – 192 . doi: 10.1016/j.ypmed.2016.06.030 .

Davis R , Campbell R , Hildon Z , et al.    Theories of behaviour and behaviour change across the social and behavioural sciences: a scoping review . Health Psychol Rev . 2015 ; 9 : 323 – 344 . doi: 10.1080/17437199.2014.941722 .

Markowski KL , Roxburgh S.   "If I became a vegan, my family and friends would hate me”: anticipating vegan stigma as a barrier to plant-based diets . Appetite . 2019 ; 135 : 1 – 9 . doi: 10.1016/j.appet.2018.12.040 .

Lea EJ , Crawford D , Worsley A.   Consumers' readiness to eat a plant-based diet . Eur J Clin Nutr . 2006 ; 60 : 342 – 351 . doi: 10.1038/sj.ejcn.1602320 .

Good Food Institute . U.S. retail market data for the plant-based industry. Available at: https://gfi.org/marketresearch/#comparison-to-animal-based-foods . Accessed June 2022.

Spalding N. The APAC Alternative Protein Industry Report 2021. 2021 . Available at: https://www.greenqueen.com.hk/apac-alt-protein-report-2021/ . Accessed February 2024.

Gaan K. Plant-based meat, eggs and dairy. 2020 The State of the Industry Report. 2020 . Available at: https://gfi.org/resource/plant-based-meat-eggs-and-dairy-state-of-the-industry-report/ . Accessed February 2024.

Smart Protein Project . Plant-based foods in Europe: how big is the market ? Smart Protein Plant-Based Food Sector Report. 2021 . Available at: https://smartproteinproject.eu/plant-based-food-sector-report . Accessed February 2024.

Devlin E. This raises £15m in series B funding round to fuel ongoing rapid growth. Available at: https://www.thegrocer.co.uk/fundraising/this-raises-15m-in-series-b-funding-round-to-fuel-ongoing-rapid-growth/676873.article . Accessed April 2023.

Ettinger J. Heura saw a 260% increase in international vegan meat sales in 2022. Green Queen. https://www.greenqueen.com.hk/heura-2022-international-vegan-meat-sales/ . Accessed March 2023.

Datassential Research . 2023 Food trends. FOODBYTES: trend report. 2022 . Available at: https://offers.datassential.com/2023-trends . Accessed February 2024.

Foster-Collins L. The death of vegan brands and the unlikely saviour of plant-based diets. Available at: https://www.thegrocer.co.uk/plant-based/the-death-of-vegan-brands-and-saviour-of-plant-based-diets/680057.article?utm_source=Weekly%20Shop%20(The%20Grocer)&utm_medium=email&utm_campaign=2023-06-12&c=&cid=DM1080641&bid=132435027 . Accessed June 2023.

Tatum M. Is Beyond Meat beyond hope? And what its slump says about the other plant-based players. The Grocer . July 9, 2022. Available at: https://www.thegrocer.co.uk/plant-based/is-beyond-meat-beyond-hope-and-what-its-slump-says-about-the-other-plant-based-players/669272.article . Accessed March 2023.

Good Food Institute . 2022 State of the Industry Report: plant-based meat, seafood, eggs, and dairy. 2022 . Available at: https://gfi.org/resource/plant-based-meat-eggs-and-dairy-state-of-the-industry-report/ . Accessed February 2024.

EAT, GlobeScan . Grains of Truth 2: a consumer report on healthy and sustainable food system. 2022 . Available at: https://eatforum.org/content/uploads/2022/11/EAT-Forum-Report.pdf . Accessed February 2024.

Good Food Institute . U.S. retail market insights for the plant-based industry. Available at: https://gfi.org/marketresearch/#purchase-dynamics . Accessed October 2023.

Alae-Carew C , Green R , Stewart C , et al.    The role of plant-based alternative foods in sustainable and healthy food systems: consumption trends in the UK . Sci Total Environ . 2021 ; 807 : 151041 . doi: 10.1016/j.scitotenv.2021.151041 .

Frontier F , Lichtenstein T. Plant-based meat: a healthier choice? 2020 . A comprehensive health and nutrition analysis of plant-based meat products in the Australian and New Zealand markets. Available at: https://www.foodfrontier.org/wp-content/uploads/2022/10/Plant-Based_Meat_A_Healthier_Choice-1.pdf . Accessed February 2024.

World Economic Forum . Meat: The Future series. Alternative Proteins. 2019 . Available at: https://www3.weforum.org/docs/WEF_White_Paper_Alternative_Proteins.pdf . Accessed February 2024.

Sahakian M , Fuchs D , Lorek S , et al.    Advancing the concept of consumption corridors and exploring its implications . Sustainability . 2021 ; 17 : 305 – 315 . doi: 10.1080/15487733.2021.1919437 .

Noguerol AT , Pagan MJ , Garcia-Segovia P , et al.    Green or clean? Perception of clean label plant-based products by omnivorous, vegan, vegetarian and flexitarian consumers . Food Res Int . 2021 ; 149 : 110652 . doi: 10.1016/j.foodres.2021.110652 .

Mertens E , Biesbroek S , Dofkova M , et al.    Potential impact of meat replacers on nutrient quality and greenhouse gas emissions of diets in four European countries . Sustainability . 2020 ; 12 : 6838 .

Edge MS , Garrett JL.   The nutrition limitations of mimicking meat . Cereal Foods World . 2020 ; 65 : 3 . doi: 10.1094/CFW-65-4-0045 .

Paul AA , Kumar S , Kumar V , et al.    Milk analog: plant based alternatives to conventional milk, production, potential and health concerns . Crit Rev Food Sci Nutr . 2020 ; 60 : 3005 – 3023 .

Verduci E , D'Elios S , Cerrato L , et al.    Cow's milk substitutes for children: nutritional aspects of milk from different mammalian species, special formula and plant-based beverages . Nutrients   2019 ; 11 : 1739 .

Chalupa-Krebzdak S , Long CJ , Bohrer BM.   Nutrient density and nutritional value of milk and plant-based milk alternatives . Int Dairy J . 2018 ; 87 : 84 – 92 .

Kazir M , Livney YD.   Plant-based seafood analogs . Molecules . 2021 ; 26 : 1559 .

Silva ARA , Silva MMN , Ribeiro BD.   Health issues and technological aspects of plant-based alternative milk . Food Res Int . 2020 ; 131 : 108972 .

McClements DJ , Newman E , McClements IF.   Plant-based milks: a review of the science underpinning their design, fabrication, and performance . Compr Rev Food Sci Food Saf . 2019 ; 18 : 2047 – 2067 .

Hadi J , Brightwell G.   Safety of alternative proteins: technological, environmental and regulatory aspects of cultured meat, plant-based meat, insect protein and single-cell protein . Foods . 2021 ; 10 : 1226 .

Liu S , Sha L.   Nutrition and safety assessment of plant protein-based meat alternative products . Food Ferment Industr . 2021 ; 47 : 297 – 303 .

Kołodziejczak K , Onopiuk A , Szpicer A , et al.    Meat analogues in the perspective of recent scientific research: a review . Foods . 2021 ; 11 : 105 . doi: 10.3390/foods11010105 .

Lonkila A , Kaljonen M.   Promises of meat and milk alternatives: an integrative literature review on emergent research themes . Agric Hum Values . 2021 ; 38 : 625 – 639 .

Boukid F.   Plant-based meat analogues: from niche to mainstream . Eur Food Res Technol . 2020 ; 247 : 297 – 308 .

Good Food Institute . Plant-Based Meat for a Growing World. 2019 . Available at: https://gfi.org/wp-content/uploads/2021/02/GFI-Plant-Based-Meat-Fact-Sheet_Environmental-Comparison.pdf . Accessed February 2023.

Nawaz MA , Tan MV , Oiseth S , et al.    An emerging segment of functional legume-based beverages: a review . Food Rev Int . 2020 ; 38 : 1064 – 1102 .

Boukid F , Rosell CM , Rosene S , et al.    Non-animal proteins as cutting-edge ingredients to reformulate animal-free foodstuffs: present status and future perspectives . Crit Rev Food Sci Nutr . 2022 ; 62 : 6390 – 6420 .

Vatansever S , Tulbek MC , Riaz MN.   Low- and high-moisture extrusion of pulse proteins as plant-based meat ingredients: a review . Cereal Foods World . 2020 ; 65 : 12 – 14 .

Chalvon-Demersay T , Azzout-Marniche D , Arfsten J , et al.    A systematic review of the effects of plant compared with animal protein sources on features of metabolic syndrome . J Nutr . 2017 2017; 147 : 281 – 292 .

Gibbs J , Leung G-K.   The effect of plant-based and mycoprotein-based meat substitute consumption on cardiometabolic risk factors: a systematic review and meta-analysis of controlled intervention trials . Dietetics . 2023 ; 2 : 104 – 122 . doi: 10.3390/dietetics2010009 .

Silva BQ , Smetana S.   Review on milk substitutes from an environmental and nutritional point of view . Appl Food Res . 2022 ; 2 : 100105 . doi: 10.1016/j.afres.2022.100105 .

Bryant CJ.   Plant-based animal product alternatives are healthier and more environmentally sustainable than animal products . Future Foods . 2022 ; 6 : 100174 . doi: 10.1016/j.fufo.2022.100174 .

Nájera Espinosa S , Pastorino S , Scheelbeek P. Mapping the evidence of plant-based alternative foods: a systematic literature review protocol on nutritional, health, and environmental impacts. 2022 . Available at: https://datacompass.lshtm.ac.uk/id/eprint/3015/1/Lit_Review_Protocol.pdf . Accessed December 2024.

Page MJ , Moher D , Bossuyt PM , et al.    PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews . BMJ . 2021 ; 372 : n160 . doi: 10.1136/bmj.n160 .

Chang C-C , Lin C-J.   LIBSVM: a library for support vector machines . ACM Trans Intell Syst Technol . 2011 ; 2 : 1 – 27 . doi: 10.1145/1961189.1961199 .

Pedregosa F , Varoquaux G , Gramfort A , et al.    Scikit-learn: machine learning in Python . J Mach Learn Res . 2011 ; 12 : 2825 – 2830 .

Berrang-Ford L , Sietsma AJ , Callaghan M , et al.    Systematic mapping of global research on climate and health: a machine learning review . Lancet Planet Health . 2021 ; 5 : e514 – 25 . doi: 10.1016/s2542-5196(21)00179-0 .

Renotte N. nicknochnack/Longform-Summarization-with-Hugging-Face. Available at: https://github.com/nicknochnack/Longform-Summarization-with-Hugging-Face/blob/main/LongSummarization.ipynb . Accessed September 2022.

Pistilli M. Plant-based food stocks: 10 biggest companies in 2023. Available at: https://investingnews.com/daily/life-science-investing/top-plant-based-food-stocks/ . Accessed May 2023.

Team V . The big list of vegan food companies. Available at: https://www.veganrecipeclub.org.uk/articles/big-list-vegan-food-companies/ . Accessed May 2023.

Meticulous Research . Top 10 companies in plant based food market. Available at: https://meticulousblog.org/top-10-companies-in-plant-based-food-market/ . Accessed May 2023.

Enerva K. Top plant-based food companies in the UK. Available at: https://viable.earth/plant-based-food/top-plant-based-food-companies-in-the-uk/ . Accessed May 2023.

Critical Appraisal Skills Programme . CASP checklist: CASP randomised controlled trial checklist. Available at: https://casp-uk.net/casp-tools-checklists/ . Accessed March 2023.

Harrington RA , Adhikari V , Rayner M , et al.    Nutrient composition databases in the age of big data: FoodDB, a comprehensive, real-time database infrastructure . BMJ Open . 2019 ; 9 : e026652 . doi: 10.1136/bmjopen-2018-026652 .

Reyes-Jurado F , Soto-Reyes N , Davila-Rodriguez M , et al.    Plant-based milk alternatives: types, processes, benefits, and characteristics . Food Rev Int . 2021 ; 39 : 2320 – 2351 .

Tangyu M , Muller J , Bolten CJ , et al.    Fermentation of plant-based milk alternatives for improved flavour and nutritional value . Appl Microbiol Biotechnol . 2019 ; 103 : 9263 – 9275 .

World Health Organization . Guideline: sugars intake for adults and children. World Health Organization;   2015 . Available at: https://www.who.int/publications/i/item/9789241549028 . Accessed February 2024.

World Health Organization . Salt intake. World Health Organization. Available at: https://www.who.int/data/gho/indicator-metadata-registry/imr-details/3082 . Accessed October 2022.

Sumner O , Burbridge L.   Plant-based milks: the dental perspective . Br Dent J . 2020 ; 8 : 16 – 23 .

Goudie S , Hughes I. The Broken Plate Report 2022. 2022 . Available at: https://foodfoundation.org.uk/publication/broken-plate-2022 . Accessed February 2024.

Collard KM , McCormick DP.   A nutritional comparison of cow's milk and alternative milk products . Acad Pediatr . 2020 ; 21 : 1067 – 1069 .

Marques M , Correig E , Capdevila E , et al.    Essential and non-essential trace elements in milks and plant-based drinks . Biol Trace Elem Res . 2021 ; 200 : 4524 – 4533 . doi: 10.1007/s12011-021-03021-5 .

Kazer J , Orfanos G , Gallop C. Quorn Footprint Comparison Report. Carbon Trust; 2022 . Available at: https://www.quorn.co.uk/assets/files/content/Carbon-Trust-Comparison-Report-2022.pdf .

De Marchi M , Costa A , Pozza M , et al.    Detailed characterization of plant-based burgers . Sci Rep . 2021 ; 11 : 2049 .

Zhang YY , Hughes J , Grafenauer S.   Got Mylk? The emerging role of Australian plant-based milk alternatives as a cow's milk substitute . Nutrients . 2020 ; 12 : 1254 .

Isidro V.   The nutritional limitations of plant-based beverages in infancy and childhood . Nutr Hosp . 2017 ; 34 : 1205 – 1214 .

Farsi DN , Uthumange D , Munoz Munoz J , et al.    The nutritional impact of replacing dietary meat with meat alternatives in the UK: a modelling analysis using nationally representative data . Br J Nutr . 2021 ; 127 : 1 – 31 .

Bryngelsson S , Moshtaghian H , Bianchi M , et al.    Nutritional assessment of plant-based meat analogues on the Swedish market . Int J Food Sci Nutr . 2022 ; 73 : 889 – 901 . doi: 10.1080/09637486.2022.2078286 .

Seves SM , Verkaik-Kloosterman J , Biesbroek S , et al.    Are more environmentally sustainable diets with less meat and dairy nutritionally adequate?   Public Health Nutr . 2017 ; 20 : 2050 – 2062 .

Goldstein B , Moses R , Sammons N , et al.    Potential to curb the environmental burdens of American beef consumption using a novel plant-based beef substitute . PLoS One . 2017 ; 12 : E 0189029 .

Kazer J , Orfanos G , Gallop C. Quorn Footprint Comparison Report. Carbon Trust; 2021 . Available at: https://www.quorn.co.uk/assets/files/content/Carbon-Trust-Comparison-Report-2021.pdf .

Allen LH , Carriquiry AL , Murphy SP.   Perspective: proposed harmonized nutrient reference values for populations . Adv Nutr . 2020 ; 11 : 469 – 483 . doi: 10.1093/advances/nmz096 .

Cherta-Murillo A , Frost GS.   The association of mycoprotein-based food consumption with diet quality, energy intake and non-communicable diseases' risk in the UK adult population using the National Diet and Nutrition Survey (NDNS) years 2008/2009-2016/2017: a cross-sectional study . Br J Nutr . 2021 ; 127 : 1685 – 1694 .

Bottin JH , Swann JR , Cropp E , et al.    Mycoprotein reduces energy intake and postprandial insulin release without altering glucagon-like peptide-1 and peptide tyrosine-tyrosine concentrations in healthy overweight and obese adults: a randomised-controlled trial . Br J Nutr . 2016 ; 116 : 360 – 374 .

Coelho MOC , Monteyne AJ , Dirks ML , et al.    Daily mycoprotein consumption for 1 week does not affect insulin sensitivity or glycaemic control but modulates the plasma lipidome in healthy adults: a randomised controlled trial . Br J Nutr . 2021 ; 125 : 147 – 160 .

Crimarco A , Springfield S , Petlura C , et al.    A randomized crossover trial on the effect of plant-based compared with animal-based meat on trimethylamine- N -oxide and cardiovascular disease risk factors in generally healthy adults: Study With Appetizing Plantfood-Meat Eating Alternative Trial (SWAP-MEAT) . Am J Clin Nutr . 2020 ; 112 : 1188 – 1199 .

Bianchi F , Stewart C , Astbury NM , et al.    Replacing meat with alternative plant-based products (RE-MAP): a randomized controlled trial of a multicomponent behavioral intervention to reduce meat consumption . Am J Clin Nutr . 2021 ; 115 : 1357 – 1366 . doi: 10.1093/ajcn/nqab414 .

Kouw IWK , Pinckaers PJM , Le Bourgot C , et al.    Ingestion of an ample amount of meat substitute based on a lysine-enriched, plant-based protein blend stimulates postprandial muscle protein synthesis to a similar extent as an isonitrogenous amount of chicken in healthy, young men . Br J Nutr . 2021 ; 128 : 1 – 1965 . doi: 10.1017/S0007114521004906 .

Toribio-Mateas MA , Bester A , Klimenko N.   Impact of plant-based meat alternatives on the gut microbiota of consumers: a real-world study . Foods . 2021 ; 10 : 2040 . doi: 10.3390/foods10092040 .

Sun LJ , Tan KWJ , Siow PC , et al.    Soya milk exerts different effects on plasma amino acid responses and incretin hormone secretion compared with cows' milk in healthy, young men . Br J Nutr . 2016 ; 116 : 1216 – 1221 .

Dineva M , Rayman MP , Bath SC.   Iodine status of consumers of milk-alternative drinks v. cows' milk: data from the UK National Diet and Nutrition Survey . Br J Nutr . 2021 ; 126 : 28 – 36 .

Shen P , Walker GD , Yuan Y , et al.    Effects of soy and bovine milk beverages on enamel mineral content in a randomized, double-blind in situ clinical study . J Dent . 2019 ; 88 : 103160 . doi: 10.1016/j.jdent.2019.06.007 .

Farsi DN , Gallegos JL , Koutsidis G , et al.    Substituting meat for mycoprotein reduces genotoxicity and increases the abundance of beneficial microbes in the gut: Mycomeat, a randomised crossover control trial . Eur J Nutr . 2023 ; 62 : 1479 – 1492 . doi: 10.1007/s00394-023-03088-x .

Grant CA , Hicks AL.   Comparative life cycle assessment of milk and plant-based alternatives . Environ Eng Sci . 2018 ; 35 : 1235 – 1247 .

Ritchie H. Dairy vs. plant-based milk: what are the environmental impacts? Available at: https://ourworldindata.org/environmental-impact-milks . Accessed May 2022.

Saget S , Costa M , Santos CS , et al.    Substitution of beef with pea protein reduces the environmental footprint of meat balls whilst supporting health and climate stabilisation goals . J Clean Prod . 2021 ; 297 : 126447 .

Rubio NR , Xiang N , Kaplan DL.   Plant-based and cell-based approaches to meat production . Nat Commun . 2020 ; 11 : 6276 .

Julia C , Fialon M , Galan P , et al.    Are foods ‘healthy’ or ‘healthier’? Front-of-pack labelling and the concept of healthiness applied to foods . Br J Nutr . 2022 ; 127 : 948 – 952 . doi: 10.1017/S0007114521001458 .

Boeck T , Zannini E , Sahin AW , et al.    Nutritional and rheological features of lentil protein isolate for yoghurt-like application . Foods . 2021 ; 10 : 1692 .

Han H , Choi JK , Park J , et al.    Recent innovations in processing technologies for improvement of nutritional quality of soymilk . CyTA-J Food . 2021 ; 19 : 287 – 303 .

Kumari M , Kokkiligadda A , Dasriya V , et al.    Functional relevance and health benefits of soymilk fermented by lactic acid bacteria . J Appl Microbiol . 2021 ; 133 : 104 – 119 . doi: 10.1111/jam.15342 .

Jeske S , Zannini E , Lynch KM , et al.    Polyol-producing lactic acid bacteria isolated from sourdough and their application to reduce sugar in a quinoa-based milk substitute . Int J Food Microbiol . 2018 ; 286 : 31 – 36 .

Dhankhar J.   Perspective on the pros and cons, manufacturing aspects, and recent advances in non-dairy milk alternatives . J Microb Biotech Food Sci . 2023 ; 12 : e9543 . doi: 10.55251/jmbfs.9543 .

Boeck T , Sahin AW , Zannini E , et al.    Nutritional properties and health aspects of pulses and their use in plant-based yogurt alternatives . Compr Rev Food Sci Food Saf . 2021 ; 20 : 3858 – 3880 .

Aydar EF , Tutuncu S , Ozcelik B.   Plant-based milk substitutes: bioactive compounds, conventional and novel processes, bioavailability studies, and health effects . J Funct Foods . 2020 ; 70 : 103975 .

Dahdouh S , Grande F , Espinosa SN , et al.    Development of the FAO/INFOODS/IZINCG global food composition database for phytate . J Food Compost Anal . 2019 ; 78 : 42 – 48 . doi: 10.1016/j.jfca.2019.01.023 .

van Vliet S , Kronberg SL , Provenza FD.   Plant-based meats, human health, and climate change . Front Sustain Food Syst . 2020 ; 4 : 128 .

WHO European Office for the Prevention and Control of Noncommunicable Diseases . Plant-based diets and their impact on health, sustainability and the environment: a review of the evidence. WHO Regional Office for Europe ; 2021 . Available at: https://www.who.int/europe/publications/i/item/WHO-EURO-2021-4007-43766-61591 . Accessed February 2024.

van Vliet S , Bain JR , Muehlbauer MJ , et al.    A metabolomics comparison of plant-based meat and grass-fed meat indicates large nutritional differences despite comparable Nutrition Facts panels . Sci Rep . 2021 ; 11 : 13828 .

Neuhofer ZT , Lusk JL.   Most plant-based meat alternative buyers also buy meat: an analysis of household demographics, habit formation, and buying behavior among meat alternative buyers . Sci Rep . 2022 ; 12 : 13062 . doi: 10.1038/s41598-022-16996-5 .

Coelho MOC , Monteyne AJ , Dunlop M , et al.    Mycoprotein as a possible alternative source of dietary protein to support muscle and metabolic health . Nutr Rev . 2020 ; 78 : 486 – 497 .

Derbyshire E , Ayoob KT.   Mycoprotein: nutritional and health properties . Nutr Today . 2019 ; 54 : 7 – 15 .

Souza Filho PF , Andersson D , Ferreira JA , et al.    Mycoprotein: environmental impact and health aspects . World J Microbiol Biotechnol . 2019 ; 35 : 147 .

Dunlop MV , Kilroe SP , Bowtell JL , et al.    Mycoprotein represents a bioavailable and insulinotropic non-animal-derived dietary protein source: a dose-response study . Br J Nutr . 2017 ; 118 : 673 – 685 .

Otis BO , Hu FB , McCarthy G.   Can plant-based meat alternatives be part of a healthy and sustainable diet?   J Am Med Assoc . 2019 ; 322 : 1547 – 1548 .

Monteiro CA , Cannon G , Levy RB , et al.    Ultra-processed foods: what they are and how to identify them . Public Health Nutr . 2019 ; 22 : 936 – 941 . doi: 10.1017/S1368980018003762 .

Pagliai G , Dinu M , Madarena MP , et al.    Consumption of ultra-processed foods and health status: a systematic review and meta-analysis . Br J Nutr . 2021 ; 125 : 308 – 318 . doi: 10.1017/S0007114520002688 .

Carlsson Kanyama A , Hedin B , Katzeff C.   Differences in environmental impact between plant-based alternatives to dairy and dairy products: a systematic literature review . Sustainability . 2021 ; 13 : 12599 . doi: 10.3390/su132212599 .

Li Y.   Feeding the future: plant-based meat for global food security and environmental sustainability . Cereal Foods World . 2020 ; 65 : 8 – 11 .

Santo RE , Kim BF , Goldman SE , et al.    Considering plant-based meat substitutes and cell-based meats: a public health and food systems perspective . Front Sustain Food Syst . 2020 ; 4 : 134 .

Jeske S , Zannini E , Arendt EK.   Evaluation of physicochemical and glycaemic properties of commercial plant-based milk substitutes . Plant Foods Hum Nutr . 2017 ; 72 : 26 – 33 .

Jaaskelainen T , Itkonen ST , Lundqvist A , et al.    The positive impact of general vitamin D food fortification policy on vitamin D status in a representative adult Finnish population: evidence from an 11-y follow-up based on standardized 25-hydroxyvitamin D data . Am J Clin Nutr . 2017 ; 105 : 1512 – 1520 . doi: 10.3945/ajcn.116.151415 .

  • alcoholic beverages
  • micronutrients
  • meat substitutes

Supplementary data

Email alerts, citing articles via.

  • Recommend to your Library

Affiliations

  • Online ISSN 1753-4887
  • Print ISSN 0029-6643
  • Copyright © 2024 International Life Sciences Institute
  • About Oxford Academic
  • Publish journals with us
  • University press partners
  • What we publish
  • New features  
  • Open access
  • Institutional account management
  • Rights and permissions
  • Get help with access
  • Accessibility
  • Advertising
  • Media enquiries
  • Oxford University Press
  • Oxford Languages
  • University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

  • Copyright © 2024 Oxford University Press
  • Cookie settings
  • Cookie policy
  • Privacy policy
  • Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

IMAGES

  1. Defining your review question

    systematic review pico tools

  2. The PICO framework for framing systematic review research questions

    systematic review pico tools

  3. The PICO framework for framing systematic review research questions

    systematic review pico tools

  4. PICO framework for framing systematic review research questions

    systematic review pico tools

  5. PICO chart detailing inclusion in systematic review. Note: PICO

    systematic review pico tools

  6. Master the PICO Framework for Systematic Review Research

    systematic review pico tools

VIDEO

  1. Discover Pico8: The Ultimate Fantasy Console on RGB30

  2. Vlogweek ที่เกาหลีตามหาหิมะยาวๆ2ชั่วโมงแบบกังนัมสไตล์ด้วย pico laser

  3. Pico unlocker Tool. with vd removed

  4. Systematic review_01

  5. How to do a Systematic Review

  6. 1-Introduction, Study designs (Randomized Controlled Trial

COMMENTS

  1. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review

    The PICO model is also frequently used as a tool for structuring clinical research questions in connection with evidence syntheses (e.g., systematic reviews). The Cochrane Handbook for Systematic Reviews of Interventions specifies using PICO as a model for developing a review question, thus ensuring that the relevant components of the question ...

  2. Cochrane PICO search

    PICO search is a trial initiative. Cochrane PICO search is made available in the Cochrane Library as a pre-release for large-scale testing. Our aim is to learn from this to prioritize future developments and enhancements of PICO search. A feedback form is available at PICO search feedback. We are keen to hear your comments.

  3. Chapter 3: Defining the criteria for including studies and ...

    Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. ... In Chapter 2, Section 2.3, we introduced the ideas of a review PICO (on which eligibility of studies is based), ... a tool for describing interventions - outlines the characteristics across which an intervention might differ ...

  4. Using the full PICO model as a search tool for systematic reviews

    The use of the four-part PICO model to facilitate search strategy development for a precise answer is recommended for structuring searches for systematic reviews. Existing guidelines generally recommend that a search strategy should include the population, intervention(s), and types of study design. Consequently, comparison and outcome are not recommended as a part of the search strategy.

  5. Using the full PICO model as a search tool for systematic reviews

    1. Introduction. The literature search forms the underlying basis of systematic reviews and thus the quality of the literature search is of crucial importance to the overall quality of the systematic review [1].The use of the four-part PICO model to facilitate searching for a precise answer is recommended [2] stating that a clinical question must be focused and well-articulated for all four ...

  6. Cleveland Clinic Florida

    The PICO model is a tool that can help you formulate a good clinical question. Sometimes it's referred to as PICO-T, containing an optional 5th factor. P - Patient, ... Subjects: Library & Information Science, Library Resources, Systematic Reviews & Meta-Analyses. Tags: ...

  7. PICO

    A research question framework can help structure your systematic review question. PICO/T is an acronym which stands for. Each PICO includes at least a P, I, and an O, and some include a C or a T. Below are some sample PICO/T questions to help you use the framework to your advantage. For an intervention/therapy.

  8. Using the full PICO model as a search tool for systematic reviews

    search strategy for a systematic review. Existing guidelines generally recommend that a search strategy should include the population, in-tervention(s), and types of study design. This study substantiates the assumption that when using PICO as a search strategy tool, some of the PICO elements will result in a considerably lower recall.

  9. PICO, PICOS and SPIDER: a comparison study of specificity and

    Qualitative systematic reviews are increasing in popularity in evidence based health care. Difficulties have been reported in conducting literature searches of qualitative research using the PICO search tool. An alternative search tool, entitled SPIDER, was recently developed for more effective searching of qualitative research, but remained untested beyond its development team.

  10. The impact of patient, intervention, comparison, outcome (PICO) as a

    Objective: This review aimed to determine if the use of the patient, intervention, comparison, outcome (PICO) model as a search strategy tool affects the quality of a literature search. Methods: A comprehensive literature search was conducted in PubMed, Embase, CINAHL, PsycINFO, Cochrane Library, Web of Science, Library and Information Science Abstracts (LISA), Scopus, and the National Library ...

  11. Cochrane Library About PICO

    There are three different sorts of PICOs within Cochrane Reviews. PICO stands for four different potential components of a health question used in Cochrane Review research: Patient, Population or Problem; Intervention; Comparison; Outcome. These components give you the specific who, what, when, where and how, of an evidence-based health-care research question.

  12. Systematic Search for Systematic Review

    A systematic review aims to answer a specific research (clinical) question. A well-formulated question will guide many aspects of the review process, including determining eligibility criteria, searching for studies, collecting data from included studies, and presenting findings (Cochrane Handbook, Sec. 5.1.1).To define a researchable question, the most commonly used structure is PICO, which ...

  13. Using the full PICO model as a search tool for systematic reviews

    Objective The use of the four-part PICO model to facilitate search strategy development for a precise answer is recommended for structuring searches for systematic reviews.

  14. PICO Portal

    PICO Portal is an enterprise literature review and systematic review platform that leverages machine learning and AI to accelerate research and innovation allowing for more efficient and accurate evidence synthesis than ever before. ... Reduce review timeline by 40-60% compared to traditional tools with the most accurate prediction and ...

  15. Systematic Reviews and Other Expert Reviews

    The PICO (Patient or Problem, Intervention, Comparison, Outcome) framework is a commonly used tool for formulating research questions. ... EndNote is the preferred tool for large systematic reviews because of the following features: Bulk import of large files; Advanced options for removing duplicate records;

  16. Using the full PICO model as a search tool for systematic reviews

    Objectives. The use of the four-part PICO model to facilitate search strategy development for a precise answer is recommended for structuring searches for systematic reviews.Existing guidelines generally recommend that a search strategy should include the population, intervention(s), and types of study design.

  17. BeckerGuides: Systematic Review Guide: PICO format

    Here is what PICO stands for: P = Problem/Population. I = Intervention (or the experimental variable) C = Comparison (or the control variable) [Optional] O = Outcome. If your research question does not fit neatly into PICO that is okay. Just try to include the elements of your question as closely as possible into the format.

  18. 5 software tools to support your systematic review processes

    The systematic review Toolbox is a web-based catalogue of various tools, including software packages which can assist with single or multiple tasks within the evidence synthesis process. Researchers can run a quick search or tailor a more sophisticated search by choosing their approach, budget, discipline, and preferred support features, to ...

  19. Data visualisation in scoping reviews and evidence maps on health

    Scoping reviews and evidence maps are forms of evidence synthesis that aim to map the available literature on a topic and are well-suited to visual presentation of results. A range of data visualisation methods and interactive data visualisation tools exist that may make scoping reviews more useful to knowledge users. The aim of this study was to explore the use of data visualisation in a ...

  20. Cancers

    Methodology: For this systematic review, the PICO model was prepared to construct and outline the exact questions of the study, a PRISMA flow diagram was prepared and quality assessment was conducted using the Newcastle Ottawa Scale (NOS) for Cohort Studies, the NIH Quality Assessment Tool-Criteria for Case Reports and the Cochrane quality ...

  21. Efficacy of electronic apex locators in comparison with intraoral

    The systematic review and meta-analysis aims to gather evidence regarding the effectiveness of EALs for WL determination when compared to different imaging techniques along with postoperative pain associated with WL determination, the number of radiographs taken during the procedure, the time taken, and the adverse effects.

  22. Prevalence of experiencing public humiliation and its effects on

    The current systematic review and meta-analysis aimed to investigate the prevalence of experiencing public humiliation and the effects on victims' mental health. A total of 33 studies with 40,468 participants were included in the systematic review and 19 studies were included in the prevalence meta-analysis.

  23. Mapping the evidence of novel plant-based foods: a systematic review of

    Briefly, a systematic search was performed to identify peer-reviewed journal articles and grey literature that contained data on the nutrient composition, health impacts, and environmental impacts of NPBFs. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. 55. Peer-reviewed literature