service design literature review

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

Service design activities in health services: A systematic literature review based on ecosystem perspective and transformative approach

Affiliations.

1 Federal University of Rio de Janeiro, UFRJ, COPPEAD Institute Pascoal Lemme st., Rio de Janeiro, Brazil.
2 Federal University of Rio de Janeiro, UFRJ Pascoal Lemme st., Rio de Janeiro, Brazil.
3 São Paulo School of Business Administration, EAESP-FGV Avenida 9 de julho, São Paulo, Brazil.
PMID: 37547988
DOI: 10.1002/hpm.3682

Service Design (SD) represents a breakthrough in searching for solutions to health systems challenges, but the activities that support these solutions remain underexplored. This research investigates how SD has been applied in the healthcare sector based on two conceptual models: multilevel ecosystem perspective and SD transformative approach. First, we conducted a systematic literature review in eight comprehensive databases in March 2021. Eligibility criteria returned 990 articles filtered by a search protocol, resulting in 47 studies. After this, we identified 23 studies (49%) with a transformative approach through a thematic analysis. Also, the analysis of these 23 studies allowed the identification of five key aspects necessary for enabling a transformative character of SD initiatives: (1) identification of all the actors that make up the provision of healthcare services, (2) identification of users by ecosystem level, (3) knowledge about the SD tools arsenal, (4) use of technology, and (5) applying the Experience-Based Design and Co-Design (EBD/EBCD) approach. The study underlines the role of management for the success of SD in the health sector and suggests an instrument (checklist) to help managers implement SD initiatives successfully.

Keywords: ecosystem; healthcare; service design; transformative approach.

PubMed Disclaimer

Publication types

Search in MeSH

Related information

Linkout - more resources, full text sources.

Ovid Technologies, Inc.

Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Reviews / Why join our community?
For companies
Frequently asked questions

Service Design

What is service design.

Service design is a process where designers create sustainable solutions and optimal experiences for both customers in unique contexts and any service providers involved. Designers break services into sections and adapt fine-tuned solutions to suit all users’ needs in context—based on actors, location and other factors.

“When you have two coffee shops right next to each other, and each sells the exact same coffee at the exact same price, service design is what makes you walk into one and not the other.” — 31Volts Service Design Studio

See how effective service design can result in more delightful experiences.

Transcript loading…

Service Design is about Designing for the Biggest Picture

Users don’t access brands in a vacuum, but within complex chains of interactions. For example, a car is a product, but in service design terms it’s a tool when an elderly customer wants to book an Uber ride to visit a friend in hospital. There’s much to consider in such contexts. This user might be accessing Uber on a smartphone, which she’s still learning to use. Perhaps she’s infirm, too, lives in an assisted living facility and must inform the driver about her specific needs. Also, she’s not the only user involved here. Other users are any service providers attached to her user experience. For example, the driver that customer books also uses Uber—but experiences a different aspect of it. To cater to various users’ and customers’ contexts as a designer, you must understand these sorts of relations between service receivers and service providers and the far-reaching aspects of their contexts from start to finish. Only then can you ideate towards solutions for these users’/customers’ specific ecosystems while you ensure brands can deliver on expectations optimally and sustainably .

In service design, you work within a broad scope including user experience (UX) design and customer experience (CX) design . To design for everyone concerned, you must appreciate the macro- and micro-level factors that affect their realities.

A service design experience often involves multiple channels, contexts and products.

Marc Stickdorn and Jakob Schneider, authors of This is Service Design Thinking , identify five key principles—for service design to be:

User-centered – Use qualitative research to design focusing on all users.

Co-creative – Include all relevant stakeholders in the design process.

Sequencing – Break a complex service into separate processes and user journey sections.

Evidencing – Envision service experiences to make them tangible for users to understand and trust brands.

Holistic – Design for all touchpoints throughout experiences, across networks of users and interactions.

Designers increasingly work more around services than around physical products—e.g., SaaS (software as a service). Meanwhile, with advances in digital technology continually redefining what users can expect whenever they proceed towards goals, brands focus on maximizing convenience and removing barriers for their users . A digital example is Square, which unbundles point-of-sale systems from cash registers and rebundles smartphones as potential point-of-sale systems.

How to Do Service Design Best

First, identify these vital parts of any service encounter:

Actors (e.g., employees delivering the service)

Location (e.g., a virtual environment where customers receive the service)

Props (e.g., objects used during service delivery)

Associates (other organizations involved in providing the service – e.g., logistics)

Processes (e.g., workflows used to deliver the service)

You’ll need to define problems, iterate and address all dimensions of the customers’, users’ and business needs best in a holistic design . To begin, you must empathize with all relevant users/customers. These are some of the most common tools:

Customer journey maps (to find the customers’ touchpoints, barriers and critical moments)

Personas (to help envision target users)

Service blueprints (elevated forms of customer journey maps that help reveal the full spectrum of situations where users/customers can interact with brands)

You should use these to help leverage insights to account for such vital areas as accessibility and customer reengagement.

Service blueprints are an important tool in the service design process.

Do Service Design for the Complete Experience

Remember to design for the complete experience. That means you should accommodate your users’/customers’ environment/s and the various barriers, motivations and feelings they’ll have. Here are some core considerations:

Understand your brand’s purpose, the demand for it and the ability of all associated service providers to deliver on promises.

The customers’ needs come ahead of the brand’s internal ones .

Focus on delivering unified and efficient services holistically —as opposed to taking a component-by-component approach.

Include input from users .

Streamline work processes to maximize efficiency .

Co-creation sessions are vital to prototyping .

Eliminate anything (e.g., features, work processes) that fails to add value for customers.

Use agile development to adapt to ever-changing customer needs.

Service design applies both to not-so-tangible areas (e.g., riders buying a single Uber trip) and tangible ones (e.g., iPhone owners visiting Apple Store for assistance/repairs). Overall, service design is a conversation where you should leave your users and customers satisfied at all touchpoints, delighted to have encountered your brand.

Learn More about Service Design

Learn all about service design by taking our course: Service Design: How to Design Integrated Service Experiences .

Read this insightful piece, Service Design: What Is It, What Does It Involve, And Should You Care?

Discover more about service blueprinting in Service Design 101

Read this eye-opening piece exploring Service Design Thinking

Examine Uber’s service design in Uber Service Design Teardown

Questions related to Service Design

A service design diagram is a visual representation of the overall structure and components of a service, including the interactions between different elements. It provides an overview of the service and helps stakeholders understand how different parts of the service fit together. It may include information such as user interfaces, system components, data flows, and more.

Actors/Roles: Entities bringing the experience to the customer.

Information Flow: Details of data shared, required, or used.

Interactions: Between people, systems, and services.

Devices & Channels: Tools and mediums of communication.

The diagram is essential for understanding the current state of a service, emphasizing the intricacies and interdependencies, guiding service blueprint creation, and identifying potential breakpoints or areas for enhancement.

In the context of service design, frontstage refers to the actions performed by employees that are visible to the customer. It includes interactions such as customer service, product demonstrations, and any other activities that customers can directly observe.

On the other hand, backstage actions are performed by employees that are not visible to the customer. These actions support the service delivery and may include tasks such as inventory management, quality control, and other behind-the-scenes operations.

Good service design is a holistic approach that prioritizes every user interaction, both in digital and real-life contexts. Jonas Piet, Director and Service Design Lead at Inwithforward shares the example of Kudoz, a learning platform to demonstrate backstage service design.

While the digital platform is a crucial component, the user's journey begins long before they interact with the app. It might start with discovering the service at a community event or through a promotional video. Service designers ensure that every touchpoint, from community events to the digital interface, provides a coherent and positive experience. They focus on the intricate details, be it designing the role of an 'Experience Curator', crafting a compelling story, or ensuring safety checks. In essence, good service design intertwines various interactions, ensuring they align perfectly.

Discover the principles of human-centered design through Interaction Design Foundation's in-depth courses: Design for the 21st Century with Don Norman offers a contemporary perspective on design thinking, while Design for a Better World with Don Norman emphasizes designing for positive global impact. To deepen your understanding, Don Norman's seminal book, " Design for a Better World: Meaningful, Sustainable, Humanity Centered ," from MIT Press, is an invaluable resource.

Developing service design begins with

In-depth user research, often ethnographic field studies, forming personas and journey maps.

Engage stakeholders early and consistently.

Utilize tools like the business model and value proposition canvases for a strategic foundation.

Transition from journey maps to service blueprints, mapping out the entire service ecosystem.

Embrace prototyping, iteratively refining with stakeholder input.

Thoroughly test prototypes, launch the finalized service, and continuously measure its impact.

Learn more from the video below:

Service design starts by understanding all pieces of an activity, centered on a user's need.

It involves figuring out systems from the ground up to support the experience, considering digital, physical, and social contexts. In-depth user research, stakeholder engagement, and aligning organizational resources, user needs, and outcomes are vital.

Service design, as discussed in our video, encompasses both the visible interactions a customer experiences and the underlying processes staff engage with. It deals with a complex web of interconnectivity, from front-end interactions to back-end systems and distribution. However, the challenge isn't just about designing services. The organizational culture must be receptive. Even if service designers identify areas of improvement, if the organization isn't prepared or faces legislative and technological barriers, change becomes arduous. Despite having dedicated individuals wanting change, they can often be constrained by larger, intricate issues. Service design requires a holistic approach, and while it can pinpoint problems, actual implementation might be held back by factors beyond the design realm.

UX (User Experience) design centers on the digital experience of users, focusing on specific touchpoints (which are often screen-based interactions). CX (Customer Experience) is broader, encompassing every touchpoint a customer has with a brand, from digital to in-store.

Service design has the highest scope of the three concepts, factoring in business processes, systems, and other back-end elements that the customer does not interact with. While UX zooms in on digital interactions, service design steps back, integrating everything for a seamless journey. All three disciplines aim to enhance the user's or customer's experience but operate at different scales and depths.

Absolutely! As businesses increasingly recognize the value of delivering exceptional customer experiences, service design has become a pivotal discipline. It ensures seamless and holistic services that cater to both customer needs and business goals.

Copyright holder: Matthew Yohe. Appearance time: 0:06 - 0:08 Copyright license and terms: CC-BY-SA-3.0 . Modified: No. Link: https://commons.wikimedia.org/wiki/File:Steve Jobs Headshot 2010-CROP (cropped_2).jpg

The demand for professionals with expertise in service design is growing across various industries, from tech to hospitality. In order to stay competitive and satisfy the current demand, many individuals are looking to improve their skills. For those keen on mastering this domain, Interaction Design Foundation's course on Service Design provides an in-depth understanding and hands-on learning. It's a great way to get started or deepen your expertise!

Answer a Short Quiz to Earn a Gift

What is the primary goal of service design?

To create visually appealing service interfaces
To ensure financial growth for service providers
To optimize experiences for both customers and service providers

Which principle of service design involves breaking complex services into manageable parts?

Co-creative

Which tool is an elevated form of a customer journey map to reveal a full spectrum of user interactions with a brand?

Service blueprints
User scenarios

What should designers focus on for a complete service experience?

They should concentrate solely on the technological aspects.
They should prioritize brand needs over customer needs.
They should understand and accommodate user environments and barriers.

What is an intended outcome of employing service design in business practices?

To decrease the overall user base to manage expectations better
To increase service complexity to enhance user engagement
To leave users satisfied at all touchpoints with the brand

Better luck next time!

Do you want to improve your UX / UI Design skills? Join us now

Congratulations! You did amazing

You earned your gift with a perfect score! Let us send it to you.

Check Your Inbox

We’ve emailed your gift to [email protected] .

Literature on Service Design

Here’s the entire UX literature on Service Design by the Interaction Design Foundation, collated in one place:

Learn more about Service Design

Take a deep dive into Service Design with our course Service Design: How to Design Integrated Service Experiences .

Services are everywhere! When you get a new passport, order a pizza or make a reservation on AirBnB, you're engaging with services. How those services are designed is crucial to whether they provide a pleasant experience or an exasperating one. The experience of a service is essential to its success or failure no matter if your goal is to gain and retain customers for your app or to design an efficient waiting system for a doctor’s office.

In a service design process, you use an in-depth understanding of the business and its customers to ensure that all the touchpoints of your service are perfect and, just as importantly, that your organization can deliver a great service experience every time . It’s not just about designing the customer interactions; you also need to design the entire ecosystem surrounding those interactions.

In this course, you’ll learn how to go through a robust service design process and which methods to use at each step along the way. You’ll also learn how to create a service design culture in your organization and set up a service design team . We’ll provide you with lots of case studies to learn from as well as interviews with top designers in the field. For each practical method, you’ll get downloadable templates that guide you on how to use the methods in your own work.

This course contains a series of practical exercises that build on one another to create a complete service design project . The exercises are optional, but you’ll get invaluable hands-on experience with the methods you encounter in this course if you complete them, because they will teach you to take your first steps as a service designer. What’s equally important is that you can use your work as a case study for your portfolio to showcase your abilities to future employers! A portfolio is essential if you want to step into or move ahead in a career in service design.

Your primary instructor in the course is Frank Spillers . Frank is CXO of award-winning design agency Experience Dynamics and a service design expert who has consulted with companies all over the world. Much of the written learning material also comes from John Zimmerman and Jodi Forlizzi , both Professors in Human-Computer Interaction at Carnegie Mellon University and highly influential in establishing design research as we know it today.

You’ll earn a verifiable and industry-trusted Course Certificate once you complete the course. You can highlight it on your resume, CV, LinkedIn profile or on your website.

All open-source articles on Service Design

The principles of service design thinking - building better services.

1.3k shares

Service Design - Design is Not Just for Products

The Moment of Truth: Build Desirable Relationships with Users and Customers

Product-Service Hybrids – When Products and Services Become One

3 years ago

10 Ideas to Help You Sell UX Work

Making sense of new UX words: A first dictionary for UX Ecosystem Design

8 years ago

Understand the Service Design Process

2 years ago

Keep These Goals in Mind to Create Successful Service Designs

Learn the Language of Service Design

Top Service Blueprint Templates

Start Your UX Journey: Essential Insights for Success

Open Access—Link to us!

We believe in Open Access and the democratization of knowledge . Unfortunately, world-class educational materials such as this page are normally hidden behind paywalls or in expensive textbooks.

If you want this to change , cite this page , link to us, or join us to help us democratize design knowledge !

Privacy Settings

Our digital services use necessary tracking technologies, including third-party cookies, for security, functionality, and to uphold user rights. Optional cookies offer enhanced features, and analytics.

Experience the full potential of our site that remembers your preferences and supports secure sign-in.

Governs the storage of data necessary for maintaining website security, user authentication, and fraud prevention mechanisms.

Enhanced Functionality

Saves your settings and preferences, like your location, for a more personalized experience.

Referral Program

We use cookies to enable our referral program, giving you and your friends discounts.

Error Reporting

We share user ID with Bugsnag and NewRelic to help us track errors and fix issues.

Optimize your experience by allowing us to monitor site usage. You’ll enjoy a smoother, more personalized journey without compromising your privacy.

Analytics Storage

Collects anonymous data on how you navigate and interact, helping us make informed improvements.

Differentiates real visitors from automated bots, ensuring accurate usage data and improving your website experience.

Lets us tailor your digital ads to match your interests, making them more relevant and useful to you.

Advertising Storage

Stores information for better-targeted advertising, enhancing your online ad experience.

Personalization Storage

Permits storing data to personalize content and ads across Google services based on user behavior, enhancing overall user experience.

Advertising Personalization

Allows for content and ad personalization across Google services based on user behavior. This consent enhances user experiences.

Enables personalizing ads based on user data and interactions, allowing for more relevant advertising experiences across Google services.

Receive more relevant advertisements by sharing your interests and behavior with our trusted advertising partners.

Enables better ad targeting and measurement on Meta platforms, making ads you see more relevant.

Allows for improved ad effectiveness and measurement through Meta’s Conversions API, ensuring privacy-compliant data sharing.

LinkedIn Insights

Tracks conversions, retargeting, and web analytics for LinkedIn ad campaigns, enhancing ad relevance and performance.

LinkedIn CAPI

Enhances LinkedIn advertising through server-side event tracking, offering more accurate measurement and personalization.

Google Ads Tag

Tracks ad performance and user engagement, helping deliver ads that are most useful to you.

Share Knowledge, Get Respect!

or copy link

Cite according to academic standards

Simply copy and paste the text below into your bibliographic reference list, onto your blog, or anywhere else. You can also just hyperlink to this page.

New to UX Design? We’re Giving You a Free ebook!

Download our free ebook The Basics of User Experience Design to learn about core concepts of UX design.

In 9 chapters, we’ll cover: conducting user interviews, design thinking, interaction design, mobile UX design, usability, UX research, and many more!

DOI: 10.1386/adch_00025_1
Corpus ID: 229519960

Service design approaches and applications in higher education: A thematic literature review

Marjo Joshi , M. Alavaikko
Published 1 October 2020
Art, Design and Communication in Higher Education

4 Citations

A review of service design pedagogy to identify potential added value to product innovation in higher education, digital design in artistic education: an overview of research in the university setting, bibliometric analysis of review on curriculum reform of design history using cnki database, disciplines go digital - developing transdisciplinary study programs that integrate ai with non-it disciplines, 24 references, project-based learning course design: a service design approach, designing a cultural event as an inclusive educational activity, designing for diverse learning: case study of place-based learning in design and technologies pre-service teacher education, educational design and networked learning: patterns, pattern languages and design practice, experiential learning through community co‐design in interior design pedagogy, measuring the value added from service learning in project-based engineering education, student-staff co-creation in higher education: an evidence-informed model to support future design and implementation, designing for service as one way of designing services, service user and carer involvement in students’ classroom learning in higher education, reimagining digital health education: reflections on the possibilities of the storyboarding method, related papers.

Showing 1 through 3 of 0 Related Papers

Systematic review
Open access
Published: 24 June 2024

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

Laura Ellen Ashcraft ORCID: orcid.org/0000-0001-9957-0617 1 , 2 ,
David E. Goodrich 3 , 4 , 5 ,
Joachim Hero 6 ,
Angela Phares 3 ,
Rachel L. Bachrach 7 , 8 ,
Deirdre A. Quinn 3 , 4 ,
Nabeel Qureshi 6 ,
Natalie C. Ernecoff 6 ,
Lisa G. Lederer 5 ,
Leslie Page Scheunemann 9 , 10 ,
Shari S. Rogal 3 , 11 na1 &
Matthew J. Chinman 3 , 4 , 6 na1

Implementation Science volume 19 , Article number: 43 ( 2024 ) Cite this article

1958 Accesses

18 Altmetric

Metrics details

Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies and the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework to classify outcomes.

We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592). We searched databases using terms “implementation strategy”, “intervention”, “bundle”, “support”, and their variants. We also solicited study recommendations from implementation science experts and mined existing systematic reviews. We included studies that quantitatively assessed the impact of at least one implementation strategy to improve health or health care using an outcome that could be mapped to the five evaluation dimensions of RE-AIM. Only studies meeting prespecified methodologic standards were included. We described the characteristics of studies and frequency of implementation strategy use across study arms. We also examined common strategy pairings and cooccurrence with significant outcomes.

Our search resulted in 16,605 studies; 129 met inclusion criteria. Studies tested an average of 6.73 strategies (0-20 range). The most assessed outcomes were Effectiveness ( n =82; 64%) and Implementation ( n =73; 56%). The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials ( n =99), Conduct Educational Meetings ( n =96), Audit and Provide Feedback ( n =76), and External Facilitation ( n =59). These strategies were often used in combination. Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes. However, many strategies were not tested sufficiently to draw conclusions.

This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings. We present recommendations for improving future similar efforts.

Peer Review reports

Contributions to the literature

While many implementation strategies exist, it has been challenging to compare their effectiveness across a wide range of trial designs and practice settings

This systematic review provides a transdisciplinary evaluation of implementation strategies across population, practice setting, and evidence-based interventions using a standardized taxonomy of strategies and outcomes.

Educational strategies were employed ubiquitously; nineteen other commonly used implementation strategies, including External Facilitation and Audit and Provide Feedback, were associated with positive outcomes in these experimental trials.

This review offers guidance for scholars and practitioners alike in selecting implementation strategies and suggests a roadmap for future evidence generation.

Implementation strategies are “methods or techniques used to enhance the adoption, implementation, and sustainment of evidence-based practices or programs” (EBPs) [ 1 ]. In 2015, the Expert Recommendations for Implementing Change (ERIC) study organized a panel of implementation scientists to compile a standardized set of implementation strategy terms and definitions [ 2 , 3 , 4 ]. These 73 strategies were then organized into nine “clusters” [ 5 ]. The ERIC taxonomy has been widely adopted and further refined [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ]. However, much of the evidence for individual or groups of ERIC strategies remains narrowly focused. Prior systematic reviews and meta-analyses have assessed strategy effectiveness, but have generally focused on a specific strategy, (e.g., Audit and Provide Feedback) [ 14 , 15 , 16 ], subpopulation, disease (e.g., individuals living with dementia) [ 16 ], outcome [ 15 ], service setting (e.g., primary care clinics) [ 17 , 18 , 19 ] or geography [ 20 ]. Given that these strategies are intended to have broad applicability, there remains a need to understand how well implementation strategies work across EBPs and settings and the extent to which implementation knowledge is generalizable.

There are challenges in assessing the evidence of implementation strategies across many EBPs, populations, and settings. Heterogeneity in population characteristics, study designs, methods, and outcomes have made it difficult to quantitatively compare which strategies work and under which conditions [ 21 ]. Moreover, there remains significant variability in how researchers operationalize, apply, and report strategies (individually or in combination) and outcomes [ 21 , 22 ]. Still, synthesizing data related to using individual strategies would help researchers replicate findings and better understand possible mediating factors including the cost, timing, and delivery by specific types of health providers or key partners [ 23 , 24 , 25 ]. Such an evidence base would also aid practitioners with implementation planning such as when and how to deploy a strategy for optimal impact.

Building upon previous efforts, we therefore conducted a systematic review to evaluate the level of evidence supporting the ERIC implementation strategies across a broad array of health and human service settings and outcomes, as organized by the evaluation framework, RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) [ 26 , 27 , 28 ]. A secondary aim of this work was to identify patterns in scientific reporting of strategy use that could not only inform reporting standards for strategies but also the methods employed in future. The current study was guided by the following research questions Footnote 1 :

What implementation strategies have been most commonly and rigorously tested in health and human service settings?

Which implementation strategies were commonly paired?

What is the evidence supporting commonly tested implementation strategies?

We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA-P) model [ 29 , 30 , 31 ] to develop and report on the methods for this systematic review (Additional File 1). This study was considered to be non-human subjects research by the RAND institutional review board.

Registration

The protocol was registered with PROSPERO (PROSPERO 2021 CRD42021235592).

Eligibility criteria

This review sought to synthesize evidence for implementation strategies from research studies conducted across a wide range of health-related settings and populations. Inclusion criteria required studies to: 1) available in English; 2) published between January 1, 2010 and September 20, 2022; 3) based on experimental research (excluded protocols, commentaries, conference abstracts, or proposed frameworks); 4) set in a health or human service context (described below); 5) tested at least one quantitative outcome that could be mapped to the RE-AIM evaluation framework [ 26 , 27 , 28 ]; and 6) evaluated the impact of an implementation strategy that could be classified using the ERIC taxonomy [ 2 , 32 ]. We defined health and human service setting broadly, including inpatient and outpatient healthcare settings, specialty clinics, mental health treatment centers, long-term care facilities, group homes, correctional facilities, child welfare or youth services, aging services, and schools, and required that the focus be on a health outcome. We excluded hybrid type I trials that primarily focused on establishing EBP effectiveness, qualitative studies, studies that described implementation barriers and facilitators without assessing implementation strategy impact on an outcome, and studies not meeting standardized rigor criteria defined below.

Information sources

Our three-pronged search strategy included searching academic databases (i.e., CINAHL, PubMed, and Web of Science for replicability and transparency), seeking recommendations from expert implementation scientists, and assessing existing, relevant systematic reviews and meta-analyses.

Search strategy

Search terms included “implementation strateg*” OR “implementation intervention*” OR “implementation bundl*” OR “implementation support*.” The search, conducted on September 20, 2022, was limited to English language and publication between 2010 and 2022, similar to other recent implementation science reviews [ 22 ]. This timeframe was selected to coincide with the advent of Implementation Science and when the term “implementation strategy” became conventionally used [ 2 , 4 , 33 ]. A full search strategy can be found in Additional File 2.

Title and abstract screening process

Each study’s title and abstract were read by two reviewers, who dichotomously scored studies on each of the six eligibility criteria described above as yes=1 or no=0, resulting in a score ranging from 1 to 6. Abstracts receiving a six from both reviewers were included in the full text review. Those with only one score of six were adjudicated by a senior member of the team (MJC, SSR, DEG). The study team held weekly meetings to troubleshoot and resolve any ongoing issues noted through the abstract screening process.

Full text screening

During the full text screening process, we reviewed, in pairs, each article that had progressed through abstract screening. Conflicts between reviewers were adjudicated by a senior member of the team for a final inclusion decision (MJC, SSR, DEG).

Review of study rigor

After reviewing published rigor screening tools [ 34 , 35 , 36 ], we developed an assessment of study rigor that was appropriate for the broad range of reviewed implementation studies. Reviewers evaluated studies on the following: 1) presence of a concurrent comparison or control group (=2 for traditional randomized controlled trial or stepped wedge cluster randomized trial and =1 for pseudo-randomized and other studies with concurrent control); 2) EBP standardization by protocol or manual (=1 if present); 3) EBP fidelity tracking (=1 if present); 4) implementation strategy standardization by operational description, standard training, or manual (=1 if present); 5) length of follow-up from full implementation of intervention (=2 for twelve months or longer, =1 for six to eleven months, or =0 for less than six months); and 6) number of sites (=1 for more than one site). Rigor scores ranged from 0 to 8, with 8 indicating the most rigorous. Articles were included if they 1) included a concurrent control group, 2) had an experimental design, and 3) received a score of 7 or 8 from two independent reviewers.

Outside expert consultation

We contacted 37 global implementation science experts who were recognized by our study team as leaders in the field or who were commonly represented among first or senior authors in the included abstracts. We asked each expert for recommendations of publications meeting study inclusion criteria (i.e., quantitatively evaluating the effectiveness of an implementation strategy). Recommendations were recorded and compared to the full abstract list.

Systematic reviews

Eighty-four systematic reviews were identified through the initial search strategy (See Additional File 3). Systematic reviews that examined the effectiveness of implementation strategies were reviewed in pairs for studies that were not found through our initial literature search.

Data abstraction and coding

Data from the full text review were abstracted in pairs, with conflicts resolved by senior team members (DEG, MJC) using a standard Qualtrics abstraction form. The form captured the setting, number of sites and participants studied, evidence-based practice/program of focus, outcomes assessed (based on RE-AIM), strategies used in each study arm, whether the study took place in the U.S. or outside of the U.S., and the findings (i.e., was there significant improvement in the outcome(s)?). We coded implementation strategies used in the Control and Experimental Arms. We defined the Control Arm as receiving the lowest number of strategies (which could mean zero strategies or care as usual) and the Experimental Arm as the most intensive arm (i.e., receiving the highest number of strategies). When studies included multiple Experimental Arms, the Experimental Arm with the least intensive implementation strategy(ies) was classified as “Control” and the Experimental Arm with the most intensive implementation strategy(ies) was classified as the “Experimental” Arm.

Implementation strategies were classified using standard definitions (MJC, SSR, DEG), based on minor modifications to the ERIC taxonomy [ 2 , 3 , 4 ]. Modifications resulted in 70 named strategies and were made to decrease redundancy and improve clarity. These modifications were based on input from experts, cognitive interview data, and team consensus [ 37 ] (See Additional File 4). Outcomes were then coded into RE-AIM outcome domains following best practices as recommended by framework experts [ 26 , 27 , 28 ]. We coded the RE-AIM domain of Effectiveness as either an assessment of the effectiveness of the EBP or the implementation strategy. We did not assess implementation strategy fidelity or effects on health disparities as these are recently adopted reporting standards [ 27 , 28 ] and not yet widely implemented in current publications. Further, we did not include implementation costs as an outcome because reporting guidelines have not been standardized [ 38 , 39 ].

Assessment and minimization of bias

Assessment and minimization of bias is an important component of high-quality systematic reviews. The Cochrane Collaboration guidance for conducting high-quality systematic reviews recommends including a specific assessment of bias for individual studies by assessing the domains of randomization, deviations of intended intervention, missing data, measurement of the outcome, and selection of the reported results (e.g., following a pre-specified analysis plan) [ 40 , 41 ]. One way we addressed bias was by consolidating multiple publications from the same study into a single finding (i.e., N =1), so-as to avoid inflating estimates due to multiple publications on different aspects of a single trial. We also included high-quality studies only, as described above. However, it was not feasible to consistently apply an assessment of bias tool due to implementation science’s broad scope and the heterogeneity of study design, context, outcomes, and variable measurement, etc. For example, most implementation studies reviewed had many outcomes across the RE-AIM framework, with no one outcome designated as primary, precluding assignment of a single score across studies.

We used descriptive statistics to present the distribution of health or healthcare area, settings, outcomes, and the median number of included patients and sites per study, overall and by country (classified as U.S. vs. non-U.S.). Implementation strategies were described individually, using descriptive statistics to summarize the frequency of strategy use “overall” (in any study arm), and the mean number of strategies reported in the Control and Experimental Arms. We additionally described the strategies that were only in the experimental (and not control) arm, defining these as strategies that were “tested” and may have accounted for differences in outcomes between arms.

We described frequencies of pair-wise combinations of implementation strategies in the Experimental Arm. To assess the strength of the evidence supporting implementation strategies that were used in the Experimental Arm, study outcomes were categorized by RE-AIM and coded based on whether the association between use of the strategies resulted in a significantly positive effect (yes=1; no=0). We then created an indicator variable if at least one RE-AIM outcome in the study was significantly positive (yes=1; no=0). We plotted strategies on a graph with quadrants based on the combination of median number of studies in which a strategy appears and the median percent of studies in which a strategy was associated with at least one positive RE-AIM outcome. The upper right quadrant—higher number of studies overall and higher percent of studies with a significant RE-AIM outcome—represents a superior level of evidence. For implementation strategies in the upper right quadrant, we describe each RE-AIM outcome and the proportion of studies which have a significant outcome.

Search results

We identified 14,646 articles through the initial literature search, 17 articles through expert recommendation (three of which were not included in the initial search), and 1,942 articles through reviewing prior systematic reviews (Fig. 1 ). After removing duplicates, 9,399 articles were included in the initial abstract screening. Of those, 48% ( n =4,075) abstracts were reviewed in pairs for inclusion. Articles with a score of five or six were reviewed a second time ( n =2,859). One quarter of abstracts that scored lower than five were reviewed for a second time at random. We screened the full text of 1,426 articles in pairs. Common reasons for exclusion were 1) study rigor, including no clear delineation between the EBP and implementation strategy, 2) not testing an implementation strategy, and 3) article type that did not meet inclusion criteria (e.g., commentary, protocol, etc.). Six hundred seventeen articles were reviewed for study rigor with 385 excluded for reasons related to study design and rigor, and 86 removed for other reasons (e.g., not a research article). Among the three additional expert-recommended articles, one met inclusion criteria and was added to the analysis. The final number of studies abstracted was 129 representing 143 publications.

Expanded PRISMA Flow Diagram

The expanded PRISMA flow diagram provides a description of each step in the review and abstraction process for the systematic review

Descriptive results

Of 129 included studies (Table 1 ; see also Additional File 5 for Summary of Included Studies), 103 (79%) were conducted in a healthcare setting. EBP health care setting varied and included primary care ( n =46; 36%), specialty care ( n =27; 21%), mental health ( n =11; 9%), and public health ( n =30; 23%), with 64 studies (50%) occurring in an outpatient health care setting. Studies included a median of 29 sites and 1,419 target population (e.g., patients or students). The number of strategies varied widely across studies, with Control Arms averaging approximately two strategies (Range = 0-20, including studies with no strategy in the comparison group) and Experimental Arms averaging eight strategies (Range = 1-21). Non-US studies ( n =73) included more sites and target population on average, with an overall median of 32 sites and 1,531 patients assessed in each study.

Organized by RE-AIM, the most evaluated outcomes were Effectiveness ( n = 82, 64%) and Implementation ( n = 73, 56%); followed by Maintenance ( n =40; 31%), Adoption ( n =33; 26%), and Reach ( n =31; 24%). Most studies ( n = 98, 76%) reported at least one significantly positive outcome. Adoption and Implementation outcomes showed positive change in three-quarters of studies ( n =78), while Reach ( n =18; 58%), Effectiveness ( n =44; 54%), and Maintenance ( n =23; 58%) outcomes evidenced positive change in approximately half of studies.

The following describes the results for each research question.

Table 2 shows the frequency of studies within which an implementation strategy was used in the Control Arm, Experimental Arm(s), and tested strategies (those used exclusively in the Experimental Arm) grouped by strategy type, as specified by previous ERIC reports [ 2 , 6 ].

Control arm

In about half the studies (53%; n =69), the Control Arms were “active controls” that included at least one strategy, with an average of 1.64 (and up to 20) strategies reported in control arms. The two most common strategies used in Control Arms were: Distribute Educational Materials ( n =52) and Conduct Educational Meetings ( n =30).

Experimental arm

Experimental conditions included an average of 8.33 implementation strategies per study (Range = 1-21). Figure 2 shows a heat map of the strategies that were used in the Experimental Arms in each study. The most common strategies in the Experimental Arm were Distribute Educational Materials ( n =99), Conduct Educational Meetings ( n =96), Audit and Provide Feedback ( n =76), and External Facilitation ( n =59).

Implementation strategies used in the Experimental Arm of included studies. Explore more here: https://public.tableau.com/views/Figure2_16947070561090/Figure2?:language=en-US&:display_count=n&:origin=viz_share_link

Tested strategies

The average number of implementation strategies that were included in the Experimental Arm only (and not in the Control Arm) was 6.73 (Range = 0-20). Footnote 2 Overall, the top 10% of tested strategies included Conduct Educational Meetings ( n =68), Audit and Provide Feedback ( n =63), External Facilitation ( n =54), Distribute Educational Materials ( n =49), Tailor Strategies ( n =41), Assess for Readiness and Identify Barriers and Facilitators ( n =38) and Organize Clinician Implementation Team Meetings ( n =37). Few studies tested a single strategy ( n =9). These strategies included, Audit and Provide Feedback, Conduct Educational Meetings, Conduct Ongoing Training, Create a Learning Collaborative, External Facilitation ( n =2), Facilitate Relay of Clinical Data To Providers, Prepare Patients/Consumers to be Active Participants, and Use Other Payment Schemes. Three implementation strategies were included in the Control or Experimental Arms but were not Tested including, Use Mass Media, Stage Implementation Scale Up, and Fund and Contract for the Clinical Innovation.

Table 3 shows the five most used strategies in Experimental Arms with their top ten most frequent pairings, excluding Distribute Educational Materials and Conduct Educational Meetings, as these strategies were included in almost all Experimental and half of Control Arms. The five most used strategies in the Experimental Arm included Audit and Provide Feedback ( n =76), External Facilitation ( n =59), Tailor Strategies ( n =43), Assess for Readiness and Identify Barriers and Facilitators ( n =43), and Organize Implementation Teams ( n =42).

Strategies frequently paired with these five strategies included two educational strategies: Distribute Educational Materials and Conduct Educational Meetings. Other commonly paired strategies included Develop a Formal Implementation Blueprint, Promote Adaptability, Conduct Ongoing Training, Purposefully Reexamine the Implementation, and Develop and Implement Tools for Quality Monitoring.

We classified the strength of evidence for each strategy by evaluating both the number of studies in which each strategy appeared in the Experimental Arm and the percentage of times there was at least one significantly positive RE-AIM outcome. Using these factors, Fig. 3 shows the number of studies in which individual strategies were evaluated (on the y axis) compared to the percentage of times that studies including those strategies had at least one positive outcome (on the x axis). Due to the non-normal distribution of both factors, we used the median (rather than the mean) to create four quadrants. Strategies in the lower left quadrant were tested in fewer than the median number of studies (8.5) and were less frequently associated with a significant RE-AIM outcome (75%). The upper right quadrant included strategies that occurred in more than the median number of studies (8.5) and had more than the median percent of studies with a significant RE-AIM outcome (75%); thus those 19 strategies were viewed as having stronger evidence. Of those 19 implementation strategies, Conduct Educational Meetings, Distribute Educational Materials, External Facilitation, and Audit and Provide Feedback continued to occur frequently, appearing in 59-99 studies.

Experimental Arm Implementation Strategies with significant RE-AIM outcome. Explore more here: https://public.tableau.com/views/Figure3_16947017936500/Figure3?:language=en-US&publish=yes&:display_count=n&:origin=viz_share_link

Figure 4 graphically illustrates the proportion of significant outcomes for each RE-AIM outcome for the 19 commonly used and evidence-based implementation strategies in the upper right quadrant. These findings again show the widespread use of Conduct Educational Meetings and Distribute Educational Materials. Implementation and Effectiveness outcomes were assessed most frequently, with Implementation being the mostly commonly reported significantly positive outcome.

RE-AIM outcomes for the 19 Top-Right Quadrant Implementation Strategies . The y-axis is the number of studies and the x-axis is a stacked bar chart for each RE-AIM outcome with R=Reach, E=Effectiveness, A=Adoption, I=Implementation, M=Maintenance. Blue denotes at least one significant RE-AIM outcome; Light blue denotes studies which used the given implementation strategy and did not have a significant RE-AIM . Explore more here: https://public.tableau.com/views/Figure4_16947017112150/Figure4?:language=en-US&publish=yes&:display_count=n&:origin=viz_share_link

This systematic review identified 129 experimental studies examining the effectiveness of implementation strategies across a broad range of health and human service studies. Overall, we found that evidence is lacking for most ERIC implementation strategies, that most studies employed combinations of strategies, and that implementation outcomes, categorized by RE-AIM dimensions, have not been universally defined or applied. Accordingly, other researchers have described the need for universal outcomes definitions and descriptions across implementation research studies [ 28 , 42 ]. Our findings have important implications not only for the current state of the field but also for creating guidance to help investigators determine which strategies and in what context to examine.

The four most evaluated strategies were Distribute Educational Materials, Conduct Educational Meetings, External Facilitation, and Audit and Provide Feedback. Conducting Educational Meetings and Distributing Educational Materials were surprisingly the most common. This may reflect the fact that education strategies are generally considered to be “necessary but not sufficient” for successful implementation [ 43 , 44 ]. Because education is often embedded in interventions, it is critical to define the boundary between the innovation and the implementation strategies used to support the innovation. Further specification as to when these strategies are EBP core components or implementation strategies (e.g., booster trainings or remediation) is needed [ 45 , 46 ].

We identified 19 implementation strategies that were tested in at least 8 studies (more than the median) and were associated with positive results at least 75% of the time. These strategies can be further categorized as being used in early or pre-implementation versus later in implementation. Preparatory activities or pre-implementation, strategies that had strong evidence included educational activities (Meetings, Materials, Outreach visits, Train for Leadership, Use Train the Trainer Strategies) and site diagnostic activities (Assess for Readiness, Identify Barriers and Facilitators, Conduct Local Needs Assessment, Identify and Prepare Champions, and Assess and Redesign Workflows). Strategies that target the implementation phase include those that provide coaching and support (External and Internal Facilitation), involve additional key partners (Intervene with Patients to Enhance Uptake and Adherence), and engage in quality improvement activities (Audit and Provide Feedback, Facilitate the Relay of Clinical Data to Providers, Purposefully Reexamine the Implementation, Conduct Cyclical Small Tests of Change, Develop and Implement Tools for Quality Monitoring).

There were many ERIC strategies that were not represented in the reviewed studies, specifically the financial and policy strategies. Ten strategies were not used in any studies, including: Alter Patient/Consumer Fees, Change Liability Laws, Change Service Sites, Develop Disincentives, Develop Resource Sharing Agreements, Identify Early Adopters, Make Billing Easier, Start a Dissemination Organization, Use Capitated Payments, and Use Data Experts. One of the limitations of this investigation was that not all individual strategies or combinations were investigated. Reasons for the absence of these strategies in our review may include challenges with testing certain strategies experimentally (e.g., changing liability laws), limitations in our search terms, and the relative paucity of implementation strategy trials compared to clinical trials. Many “untested” strategies require large-scale structural changes with leadership support (see [ 47 ] for policy experiment example). Recent preliminary work has assessed the feasibility of applying policy strategies and described the challenges with doing so [ 48 , 49 , 50 ]. While not impossible in large systems like VA (for example: the randomized evaluation of the VA Stratification Tool for Opioid Risk Management) the large size, structure, and organizational imperative makes these initiatives challenging to experimentally evaluate. Likewise, the absence of these ten strategies may have been the result of our inclusion criteria, which required an experimental design. Thus, creative study designs may be needed to test high-level policy or financial strategies experimentally.

Some strategies that were likely under-represented in our search strategy included electronic medical record reminders and clinical decision support tools and systems. These are often considered “interventions” when used by clinical trialists and may not be indexed as studies involving ‘implementation strategies’ (these tools have been reviewed elsewhere [ 51 , 52 , 53 ]). Thus, strategies that are also considered interventions in the literature (e.g., education interventions) were not sought or captured. Our findings do not imply that these strategies are ineffective, rather that more study is needed. Consistent with prior investigations [ 54 ], few studies meeting inclusion criteria tested financial strategies. Accordingly, there are increasing calls to track and monitor the effects of financial strategies within implementation science to understand their effectiveness in practice [ 55 , 56 ]. However, experts have noted that the study of financial strategies can be a challenge given that they are typically implemented at the system-level and necessitate research designs for studying policy-effects (e.g., quasi-experimental methods, systems-science modeling methods) [ 57 ]. Yet, there have been some recent efforts to use financial strategies to support EBPs that appear promising [ 58 ] and could be a model for the field moving forward.

The relationship between the number of strategies used and improved outcomes has been described inconsistently in the literature. While some studies have found improved outcomes with a bundle of strategies that were uniquely combined or a standardized package of strategies (e.g., Replicating Effective Programs [ 59 , 60 ] and Getting To Outcomes [ 61 , 62 ]), others have found that “more is not always better” [ 63 , 64 , 65 ]. For example, Rogal and colleagues documented that VA hospitals implementing a new evidence-based hepatitis C treatment chose >20 strategies, when multiple years of data linking strategies to outcomes showed that 1-3 specific strategies would have yielded the same outcome [ 39 ]. Considering that most studies employed multiple or multifaceted strategies, it seems that there is a benefit of using a targeted bundle of strategies that are purposefully aligns with site/clinic/population norms, rather than simply adding more strategies [ 66 ].

It is difficult to assess the effectiveness of any one implementation strategy in bundles where multiple strategies are used simultaneously. Even a ‘single’ strategy like External Facilitation is, in actuality, a bundle of narrowly constructed strategies (e.g., Conduct Educational Meetings, Identify and Prepare Champions, and Develop a Formal Implementation Blueprint). Thus, studying External Facilitation does not allow for a test of the individual strategies that comprise it, potentially masking the effectiveness of any individual strategy. While we cannot easily disaggregate the effects of multifaceted strategies, doing so may not yield meaningful results. Because strategies often synergize, disaggregated results could either underestimate the true impact of individual strategies or conversely, actually undermine their effectiveness (i.e., when their effectiveness comes from their combination with other strategies). The complexity of health and human service settings, imperative to improve public health outcomes, and engagement with community partners often requires the use of multiple strategies simultaneously. Therefore, the need to improve real-world implementation may outweigh the theoretical need to identify individual strategy effectiveness. In situations where it would be useful to isolate the impact of single strategies, we suggest that the same methods for documenting and analyzing the critical components (or core functions) of complex interventions [ 67 , 68 , 69 , 70 ] may help to identify core components of multifaceted implementation strategies [ 71 , 72 , 73 , 74 ].

In addition, to truly assess the impacts of strategies on outcomes, it may be necessary to track fidelity to implementation strategies (not just the EBPs they support). While this can be challenging, without some degree of tracking and fidelity checks, one cannot determine whether a strategy’s apparent failure to work was because it 1) was ineffective or 2) was not applied well. To facilitate this tracking there are pragmatic tools to support researchers. For example, the Longitudinal Implementation Strategy Tracking System (LISTS) offers a pragmatic and feasible means to assess fidelity to and adaptations of strategies [ 75 ].

Implications for implementation science: four recommendations

Based on our findings, we offer four recommended “best practices” for implementation studies.

Prespecify strategies using standard nomenclature. This study reaffirmed the need to apply not only a standard naming convention (e.g., ERIC) but also a standard reporting of for implementation strategies. While reporting systems like those by Proctor [ 1 ] or Pinnock [ 75 ] would optimize learning across studies, few manuscripts specify strategies as recommended [ 76 , 77 ]. Pre-specification allows planners and evaluators to assess the feasibility and acceptability of strategies with partners and community members [ 24 , 78 , 79 ] and allows evaluators and implementers to monitor and measure the fidelity, dose, and adaptations to strategies delivered over the course of implementation [ 27 ]. In turn, these data can be used to assess the costs, analyze their effectiveness [ 38 , 80 , 81 ], and ensure more accurate reporting [ 82 , 83 , 84 , 85 ]. This specification should include, among other data, the intensity, stage of implementation, and justification for the selection. Information regarding why strategies were selected for specific settings would further the field and be of great use to practitioners. [ 63 , 65 , 69 , 79 , 86 ].

Ensure that standards for measuring and reporting implementation outcomes are consistently applied and account for the complexity of implementation studies. Part of improving standardized reporting must include clearly defining outcomes and linking each outcome to particular implementation strategies. It was challenging in the present review to disentangle the impact of the intervention(s) (i.e., the EBP) versus the impact of the implementation strategy(ies) for each RE-AIM dimension. For example, often fidelity to the EBP was reported but not for the implementation strategies. Similarly, Reach and Adoption of the intervention would be reported for the Experimental Arm but not for the Control Arm, prohibiting statistical comparisons of strategies on the relative impact of the EBP between study arms. Moreover, there were many studies evaluating numerous outcomes, risking data dredging. Further, the significant heterogeneity in the ways in which implementation outcomes are operationalized and reported is a substantial barrier to conducting large-scale meta-analytic approaches to synthesizing evidence for implementation strategies [ 67 ]. The field could look to others in the social and health sciences for examples in how to test, validate, and promote a common set of outcome measures to aid in bringing consistency across studies and real-world practice (e.g., the NIH-funded Patient-Reported Outcomes Measurement Information System [PROMIS], https://www.healthmeasures.net/explore-measurement-systems/promis ).

Develop infrastructure to learn cross-study lessons in implementation science. Data repositories, like those developed by NCI for rare diseases, U.S. HIV Implementation Science Coordination Initiative [ 87 ], and the Behavior Change Technique Ontology [ 88 ], could allow implementation scientists to report their findings in a more standardized manner, which would promote ease of communication and contextualization of findings across studies. For example, the HIV Implementation Science Coordination Initiative requested all implementation projects use common frameworks, developed user friendly databases to enable practitioners to match strategies to determinants, and developed a dashboard of studies that assessed implementation determinants [ 89 , 90 , 91 , 92 , 93 , 94 ].

Develop and apply methods to rigorously study common strategies and bundles. These findings support prior recommendations for improved empirical rigor in implementation studies [ 46 , 95 ]. Many studies were excluded from our review based on not meeting methodological rigor standards. Understanding the effectiveness of discrete strategies deployed alone or in combination requires reliable and low burden tracking methods to collect information about strategy use and outcomes. For example, frameworks like the Implementation Replication Framework [ 96 ] could help interpret findings across studies using the same strategy bundle. Other tracking approaches may leverage technology (e.g., cell phones, tablets, EMR templates) [ 78 , 97 ] or find novel, pragmatic approaches to collect recommended strategy specifications over time (e.g.., dose, deliverer, and mechanism) [ 1 , 9 , 27 , 98 , 99 ]. Rigorous reporting standards could inform more robust analyses and conclusions (e.g., moving toward the goal of understanding causality, microcosting efforts) [ 24 , 38 , 100 , 101 ]. Such detailed tracking is also required to understand how site-level factors moderate implementation strategy effects [ 102 ]. In some cases, adaptive trial designs like sequential multiple assignment randomized trials (SMARTs) and just-in-time adaptive interventions (JITAIs) can be helpful for planning strategy escalation.

Limitations

Despite the strengths of this review, there were certain notable limitations. For one, we only included experimental studies, omitting many informative observational investigations that cover the range of implementation strategies. Second, our study period was centered on the creation of the journal Implementation Science and not on the standardization and operationalization of implementation strategies in the publication of the ERIC taxonomy (which came later). This, in conjunction with latency in reporting study results and funding cycles, means that the employed taxonomy was not applied in earlier studies. To address this limitation, we retroactively mapped strategies to ERIC, but it is possible that some studies were missed. Additionally, indexing approaches used by academic databases may have missed relevant studies. We addressed this particular concern by reviewing other systematic reviews of implementation strategies and soliciting recommendations from global implementation science experts.

Another potential limitation comes from the ERIC taxonomy itself—i.e., strategy listings like ERIC are only useful when they are widely adopted and used in conjunction with guidelines for specifying and reporting strategies [ 1 ] in protocol and outcome papers. Although the ERIC paper has been widely cited (over three thousand times, accessed about 186 thousand times), it is still not universally applied, making tracking the impact of specific strategies more difficult. However, our experience with this review seemed to suggest that ERIC’s use was increasing over time. Also, some have commented that ERIC strategies can be unclear and are missing key domains. Thus, researchers are making definitions clearer for lay users [ 37 , 103 ], increasing the number of discrete strategies for specific domains like HIV treatment, acknowledging strategies for new functions (e.g., de-implementation [ 104 ], local capacity building), accounting for phases of implementation (dissemination, sustainment [ 13 ], scale-up), addressing settings [ 12 , 20 ], actors roles in the process, and making mechanisms of change to select strategies more user-friendly through searchable databases [ 9 , 10 , 54 , 73 , 104 , 105 , 106 ]. In sum, we found the utility of the ERIC taxonomy to outweigh any of the taxonomy’s current limitations.

As with all reviews, the search terms influenced our findings. As such, the broad terms for implementation strategies (e.g., “evidence-based interventions”[ 7 ] or “behavior change techniques” [ 107 ]) may have led to inadvertent omissions of studies of specific strategies. For example, the search terms may not have captured tests of policies, financial strategies, community health promotion initiatives, or electronic medical record reminders, due to differences in terminology used in corresponding subfields of research (e.g., health economics, business, health information technology, and health policy). To manage this, we asked experts to inform us about any studies that they would include and cross-checked their lists with what was identified through our search terms, which yielded very few additional studies. We included standard coding using the ERIC taxonomy, which was a strength, but future work should consider including the additional strategies that have been recommended to augment ERIC, around sustainment [ 13 , 79 , 106 , 108 ], community and public health research [ 12 , 109 , 110 , 111 ], consumer or service user engagement [ 112 ], de-implementation [ 104 , 113 , 114 , 115 , 116 , 117 ] and related terms [ 118 ].

We were unable to assess the bias of studies due to non-standard reporting across the papers and the heterogeneity of study designs, measurement of implementation strategies and outcomes, and analytic approaches. This could have resulted in over- or underestimating the results of our synthesis. We addressed this limitation by being cautious in our reporting of findings, specifically in identifying “effective” implementation strategies. Further, we were not able to gather primary data to evaluate effect sizes across studies in order to systematically evaluate bias, which would be fruitful for future study.

Conclusions

This novel review of 129 studies summarized the body of evidence supporting the use of ERIC-defined implementation strategies to improve health or healthcare. We identified commonly occurring implementation strategies, frequently used bundles, and the strategies with the highest degree of supportive evidence, while simultaneously identifying gaps in the literature. Additionally, we identified several key areas for future growth and operationalization across the field of implementation science with the goal of improved reporting and assessment of implementation strategies and related outcomes.

Availability and materials

All data for this study are included in this published article and its supplementary information files.

We modestly revised the following research questions from our PROSPERO registration after reading the articles and better understanding the nature of the literature: 1) What is the available evidence regarding the effectiveness of implementation strategies in supporting the uptake and sustainment of evidence intended to improve health and healthcare outcomes? 2) What are the current gaps in the literature (i.e., implementation strategies that do not have sufficient evidence of effectiveness) that require further exploration?

Tested strategies are those which exist in the Experimental Arm but not in the Control Arm. Comparative effectiveness or time staggered trials may not have any unique strategies in the Experimental Arm and therefore in our analysis would have no Tested Strategies.

Abbreviations

Centers for Disease Control

Cumulated Index to Nursing and Allied Health Literature

Dissemination and Implementation

Evidence-based practices or programs

Expert Recommendations for Implementing Change

Multiphase Optimization Strategy

National Cancer Institute

National Institutes of Health

The Pittsburgh Dissemination and Implementation Science Collaborative

Sequential Multiple Assignment Randomized Trial

United States

Department of Veterans Affairs

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

Article PubMed PubMed Central Google Scholar

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

Waltz TJ, Powell BJ, Chinman MJ, Smith JL, Matthieu MM, Proctor EK, et al. Expert recommendations for implementing change (ERIC): protocol for a mixed methods study. Implement Sci IS. 2014;9:39.

Article PubMed Google Scholar

Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health. Med Care Res Rev. 2012;69:123–57.

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:109.

Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implement Sci. 2019;14(1):32.

Community Preventive Services Task Force. Community Preventive Services Task Force: All Active Findings June 2023 [Internet]. 2023 [cited 2023 Aug 7]. Available from: https://www.thecommunityguide.org/media/pdf/CPSTF-All-Findings-508.pdf

Solberg LI, Kuzel A, Parchman ML, Shelley DR, Dickinson WP, Walunas TL, et al. A Taxonomy for External Support for Practice Transformation. J Am Board Fam Med JABFM. 2021;34:32–9.

Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice. Implement Sci. 2017;12:1–9.

Article Google Scholar

Leeman J, Calancie L, Hartman MA, Escoffery CT, Herrmann AK, Tague LE, et al. What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective?: a systematic review. Implement Sci. 2015;10:1–15.

Nathan N, Shelton RC, Laur CV, Hailemariam M, Hall A. Editorial: Sustaining the implementation of evidence-based interventions in clinical and community settings. Front Health Serv. 2023;3:1176023.

Balis LE, Houghtaling B, Harden SM. Using implementation strategies in community settings: an introduction to the Expert Recommendations for Implementing Change (ERIC) compilation and future directions. Transl Behav Med. 2022;12:965–78.

Nathan N, Powell BJ, Shelton RC, Laur CV, Wolfenden L, Hailemariam M, et al. Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment? Front Health Serv. 2022;2:905909.

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259.

Google Scholar

Moore L, Guertin JR, Tardif P-A, Ivers NM, Hoch J, Conombo B, et al. Economic evaluations of audit and feedback interventions: a systematic review. BMJ Qual Saf. 2022;31:754–67.

Sykes MJ, McAnuff J, Kolehmainen N. When is audit and feedback effective in dementia care? A systematic review. Int J Nurs Stud. 2018;79:27–35.

Barnes C, McCrabb S, Stacey F, Nathan N, Yoong SL, Grady A, et al. Improving implementation of school-based healthy eating and physical activity policies, practices, and programs: a systematic review. Transl Behav Med. 2021;11:1365–410.

Tomasone JR, Kauffeldt KD, Chaudhary R, Brouwers MC. Effectiveness of guideline dissemination and implementation strategies on health care professionals’ behaviour and patient outcomes in the cancer care context: a systematic review. Implement Sci. 2020;15:1–18.

Seda V, Moles RJ, Carter SR, Schneider CR. Assessing the comparative effectiveness of implementation strategies for professional services to community pharmacy: A systematic review. Res Soc Adm Pharm. 2022;18:3469–83.

Lovero KL, Kemp CG, Wagenaar BH, Giusto A, Greene MC, Powell BJ, et al. Application of the Expert Recommendations for Implementing Change (ERIC) compilation of strategies to health intervention implementation in low- and middle-income countries: a systematic review. Implement Sci. 2023;18:56.

Chapman A, Rankin NM, Jongebloed H, Yoong SL, White V, Livingston PM, et al. Overcoming challenges in conducting systematic reviews in implementation science: a methods commentary. Syst Rev. 2023;12:1–6.

Article CAS Google Scholar

Proctor EK, Bunger AC, Lengnick-Hall R, Gerke DR, Martin JK, Phillips RJ, et al. Ten years of implementation outcomes research: a scoping review. Implement Sci. 2023;18:1–19.

Michaud TL, Pereira E, Porter G, Golden C, Hill J, Kim J, et al. Scoping review of costs of implementation strategies in community, public health and healthcare settings. BMJ Open. 2022;12:e060785.

Sohn H, Tucker A, Ferguson O, Gomes I, Dowdy D. Costing the implementation of public health interventions in resource-limited settings: a conceptual framework. Implement Sci. 2020;15:1–8.

Peek C, Glasgow RE, Stange KC, Klesges LM, Purcell EP, Kessler RS. The 5 R’s: an emerging bold standard for conducting relevant research in a changing world. Ann Fam Med. 2014;12:447–55.

Article CAS PubMed PubMed Central Google Scholar

Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7.

Shelton RC, Chambers DA, Glasgow RE. An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Front Public Health. 2020;8:134.

Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: Clarifications and resources. J Clin Transl Sci. 2021;5:e126.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.

Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ [Internet]. 2021;372. Available from: https://www.bmj.com/content/372/bmj.n71

Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A Glossary for Dissemination and Implementation Research in Health. J Public Health Manag Pract. 2008;14:117–23.

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1.

Article PubMed Central Google Scholar

Miller WR, Wilbourne PL. Mesa Grande: a methodological analysis of clinical trials of treatments for alcohol use disorders. Addict Abingdon Engl. 2002;97:265–77.

Miller WR, Brown JM, Simpson TL, Handmaker NS, Bien TH, Luckie LF, et al. What works? A methodological analysis of the alcohol treatment outcome literature. Handb Alcohol Treat Approaches Eff Altern 2nd Ed. Needham Heights, MA, US: Allyn & Bacon; 1995:12–44.

Wells S, Tamir O, Gray J, Naidoo D, Bekhit M, Goldmann D. Are quality improvement collaboratives effective? A systematic review BMJ Qual Saf. 2018;27:226–40.

Yakovchenko V, Chinman MJ, Lamorte C, Powell BJ, Waltz TJ, Merante M, et al. Refining Expert Recommendations for Implementing Change (ERIC) strategy surveys using cognitive interviews with frontline providers. Implement Sci Commun. 2023;4:1–14.

Wagner TH, Yoon J, Jacobs JC, So A, Kilbourne AM, Yu W, et al. Estimating costs of an implementation intervention. Med Decis Making. 2020;40:959–67.

Gold HT, McDermott C, Hoomans T, Wagner TH. Cost data in implementation science: categories and approaches to costing. Implement Sci. 2022;17:11.

Boutron I, Page MJ, Higgins JP, Altman DG, Lundh A, Hróbjartsson A. Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions. 2019. https://doi.org/10.1002/9781119536604.ch7 .

Higgins JP, Savović J, Page MJ, Elbers RG, Sterne J. Assessing risk of bias in a randomized trial. Cochrane Handb Syst Rev Interv. 2019;6:205–28.

Reilly KL, Kennedy S, Porter G, Estabrooks P. Comparing, Contrasting, and Integrating Dissemination and Implementation Outcomes Included in the RE-AIM and Implementation Outcomes Frameworks. Front Public Health [Internet]. 2020 [cited 2024 Apr 24];8. Available from: https://www.frontiersin.org/journals/public-health/articles/ https://doi.org/10.3389/fpubh.2020.00430/full

Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess Winch Engl. 2004;8:iii–iv 1-72.

CAS Google Scholar

Beidas RS, Kendall PC. Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. Clin Psychol Publ Div Clin Psychol Am Psychol Assoc. 2010;17:1–30.

Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, et al. Methods to Improve the Selection and Tailoring of Implementation Strategies. J Behav Health Serv Res. 2017;44:177–94.

Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front Public Health [Internet]. 2019 [cited 2021 Mar 31];7. Available from: https://www.frontiersin.org/articles/ https://doi.org/10.3389/fpubh.2019.00003/full

Frakt AB, Prentice JC, Pizer SD, Elwy AR, Garrido MM, Kilbourne AM, et al. Overcoming Challenges to Evidence-Based Policy Development in a Large. Integrated Delivery System Health Serv Res. 2018;53:4789–807.

PubMed Google Scholar

Crable EL, Lengnick-Hall R, Stadnick NA, Moullin JC, Aarons GA. Where is “policy” in dissemination and implementation science? Recommendations to advance theories, models, and frameworks: EPIS as a case example. Implement Sci. 2022;17:80.

Crable EL, Grogan CM, Purtle J, Roesch SC, Aarons GA. Tailoring dissemination strategies to increase evidence-informed policymaking for opioid use disorder treatment: study protocol. Implement Sci Commun. 2023;4:16.

Bond GR. Evidence-based policy strategies: A typology. Clin Psychol Sci Pract. 2018;25:e12267.

Loo TS, Davis RB, Lipsitz LA, Irish J, Bates CK, Agarwal K, et al. Electronic Medical Record Reminders and Panel Management to Improve Primary Care of Elderly Patients. Arch Intern Med. 2011;171:1552–8.

Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J. Effect of point-of-care computer reminders on physician behaviour: a systematic review. CMAJ Can Med Assoc J. 2010;182:E216-25.

Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, et al. A Randomized Trial of Electronic Clinical Reminders to Improve Quality of Care for Diabetes and Coronary Artery Disease. J Am Med Inform Assoc JAMIA. 2005;12:431–7.

Dopp AR, Kerns SEU, Panattoni L, Ringel JS, Eisenberg D, Powell BJ, et al. Translating economic evaluations into financing strategies for implementing evidence-based practices. Implement Sci. 2021;16:1–12.

Dopp AR, Hunter SB, Godley MD, Pham C, Han B, Smart R, et al. Comparing two federal financing strategies on penetration and sustainment of the adolescent community reinforcement approach for substance use disorders: protocol for a mixed-method study. Implement Sci Commun. 2022;3:51.

Proctor EK, Toker E, Tabak R, McKay VR, Hooley C, Evanoff B. Market viability: a neglected concept in implementation science. Implement Sci. 2021;16:98.

Dopp AR, Narcisse M-R, Mundey P, Silovsky JF, Smith AB, Mandell D, et al. A scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: State of the literature and future directions. Implement Res Pract. 2020;1:2633489520939980.

PubMed PubMed Central Google Scholar

Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care:application of the replicating effective programs framework. Implement Sci. 2007;2:42–51.

Kegeles SM, Rebchook GM, Hays RB, Terry MA, O’Donnell L, Leonard NR, et al. From science to application: the development of an intervention package. AIDS Educ Prev Off Publ Int Soc AIDS Educ. 2000;12:62–74.

Wandersman A, Imm P, Chinman M, Kaftarian S. Getting to outcomes: a results-based approach to accountability. Eval Program Plann. 2000;23:389–95.

Wandersman A, Chien VH, Katz J. Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. Am J Community Psychol. 2012;50:445–59.

Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Kirchner JE, Proctor EK, et al. The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. Implement Sci. 2017;12:1–13.

Smith SN, Almirall D, Prenovost K, Liebrecht C, Kyle J, Eisenberg D, et al. Change in patient outcomes after augmenting a low-level implementation strategy in community practices that are slow to adopt a collaborative chronic care model: a cluster randomized implementation trial. Med Care. 2019;57:503.

Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Gonzalez R, Park A, et al. Longitudinal assessment of the association between implementation strategy use and the uptake of hepatitis C treatment: Year 2. Implement Sci. 2019;14:1–12.

Harvey G, Kitson A. Translating evidence into healthcare policy and practice: Single versus multi-faceted implementation strategies – is there a simple answer to a complex question? Int J Health Policy Manag. 2015;4:123–6.

Engell T, Stadnick NA, Aarons GA, Barnett ML. Common Elements Approaches to Implementation Research and Practice: Methods and Integration with Intervention Science. Glob Implement Res Appl. 2023;3:1–15.

Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci IS. 2009;4:40.

Smith JD, Li DH, Rafferty MR. The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci IS. 2020;15:84.

Perez Jolles M, Lengnick-Hall R, Mittman BS. Core Functions and Forms of Complex Health Interventions: a Patient-Centered Medical Home Illustration. JGIM J Gen Intern Med. 2019;34:1032–8.

Schroeck FR, Ould Ismail AA, Haggstrom DA, Sanchez SL, Walker DR, Zubkoff L. Data-driven approach to implementation mapping for the selection of implementation strategies: a case example for risk-aligned bladder cancer surveillance. Implement Sci IS. 2022;17:58.

Frank HE, Kemp J, Benito KG, Freeman JB. Precision Implementation: An Approach to Mechanism Testing in Implementation Research. Adm Policy Ment Health. 2022;49:1084–94.

Lewis CC, Klasnja P, Lyon AR, Powell BJ, Lengnick-Hall R, Buchanan G, et al. The mechanics of implementation strategies and measures: advancing the study of implementation mechanisms. Implement Sci Commun. 2022;3:114.

Geng EH, Baumann AA, Powell BJ. Mechanism mapping to advance research on implementation strategies. PLoS Med. 2022;19:e1003918.

Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;356:i6795.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38:65–76.

Hooley C, Amano T, Markovitz L, Yaeger L, Proctor E. Assessing implementation strategy reporting in the mental health literature: a narrative review. Adm Policy Ment Health Ment Health Serv Res. 2020;47:19–35.

Proctor E, Ramsey AT, Saldana L, Maddox TM, Chambers DA, Brownson RC. FAST: a framework to assess speed of translation of health innovations to practice and policy. Glob Implement Res Appl. 2022;2:107–19.

Cullen L, Hanrahan K, Edmonds SW, Reisinger HS, Wagner M. Iowa Implementation for Sustainability Framework. Implement Sci IS. 2022;17:1.

Saldana L, Ritzwoller DP, Campbell M, Block EP. Using economic evaluations in implementation science to increase transparency in costs and outcomes for organizational decision-makers. Implement Sci Commun. 2022;3:40.

Eisman AB, Kilbourne AM, Dopp AR, Saldana L, Eisenberg D. Economic evaluation in implementation science: making the business case for implementation strategies. Psychiatry Res. 2020;283:112433.

Akiba CF, Powell BJ, Pence BW, Nguyen MX, Golin C, Go V. The case for prioritizing implementation strategy fidelity measurement: benefits and challenges. Transl Behav Med. 2022;12:335–42.

Akiba CF, Powell BJ, Pence BW, Muessig K, Golin CE, Go V. “We start where we are”: a qualitative study of barriers and pragmatic solutions to the assessment and reporting of implementation strategy fidelity. Implement Sci Commun. 2022;3:117.

Rudd BN, Davis M, Doupnik S, Ordorica C, Marcus SC, Beidas RS. Implementation strategies used and reported in brief suicide prevention intervention studies. JAMA Psychiatry. 2022;79:829–31.

Painter JT, Raciborski RA, Matthieu MM, Oliver CM, Adkins DA, Garner KK. Engaging stakeholders to retrospectively discern implementation strategies to support program evaluation: Proposed method and case study. Eval Program Plann. 2024;103:102398.

Bunger AC, Powell BJ, Robertson HA, MacDowell H, Birken SA, Shea C. Tracking implementation strategies: a description of a practical approach and early findings. Health Res Policy Syst. 2017;15:1–12.

Mustanski B, Smith JD, Keiser B, Li DH, Benbow N. Supporting the growth of domestic HIV implementation research in the united states through coordination, consultation, and collaboration: how we got here and where we are headed. JAIDS J Acquir Immune Defic Syndr. 2022;90:S1-8.

Marques MM, Wright AJ, Corker E, Johnston M, West R, Hastings J, et al. The Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1. Wellcome Open Res. 2023;8:308.

Merle JL, Li D, Keiser B, Zamantakis A, Queiroz A, Gallo CG, et al. Categorising implementation determinants and strategies within the US HIV implementation literature: a systematic review protocol. BMJ Open. 2023;13:e070216.

Glenshaw MT, Gaist P, Wilson A, Cregg RC, Holtz TH, Goodenow MM. Role of NIH in the Ending the HIV Epidemic in the US Initiative: Research Improving Practice. J Acquir Immune Defic Syndr. 1999;2022(90):S9-16.

Purcell DW, Namkung Lee A, Dempsey A, Gordon C. Enhanced Federal Collaborations in Implementation Science and Research of HIV Prevention and Treatment. J Acquir Immune Defic Syndr. 1999;2022(90):S17-22.

Queiroz A, Mongrella M, Keiser B, Li DH, Benbow N, Mustanski B. Profile of the Portfolio of NIH-Funded HIV Implementation Research Projects to Inform Ending the HIV Epidemic Strategies. J Acquir Immune Defic Syndr. 1999;2022(90):S23-31.

Zamantakis A, Li DH, Benbow N, Smith JD, Mustanski B. Determinants of Pre-exposure Prophylaxis (PrEP) Implementation in Transgender Populations: A Qualitative Scoping Review. AIDS Behav. 2023;27:1600–18.

Li DH, Benbow N, Keiser B, Mongrella M, Ortiz K, Villamar J, et al. Determinants of Implementation for HIV Pre-exposure Prophylaxis Based on an Updated Consolidated Framework for Implementation Research: A Systematic Review. J Acquir Immune Defic Syndr. 1999;2022(90):S235-46.

Chambers DA, Emmons KM. Navigating the field of implementation science towards maturity: challenges and opportunities. Implement Sci. 2024;19:26, s13012-024-01352–0.

Chinman M, Acosta J, Ebener P, Shearer A. “What we have here, is a failure to [replicate]”: Ways to solve a replication crisis in implementation science. Prev Sci. 2022;23:739–50.

Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117.

Lengnick-Hall R, Gerke DR, Proctor EK, Bunger AC, Phillips RJ, Martin JK, et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci. 2022;17:16.

Miller CJ, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implement Sci IS. 2021;16:36.

Xu X, Lazar CM, Ruger JP. Micro-costing in health and medicine: a critical appraisal. Health Econ Rev. 2021;11:1.

Barnett ML, Dopp AR, Klein C, Ettner SL, Powell BJ, Saldana L. Collaborating with health economists to advance implementation science: a qualitative study. Implement Sci Commun. 2020;1:82.

Lengnick-Hall R, Williams NJ, Ehrhart MG, Willging CE, Bunger AC, Beidas RS, et al. Eight characteristics of rigorous multilevel implementation research: a step-by-step guide. Implement Sci. 2023;18:52.

Riley-Gibson E, Hall A, Shoesmith A, Wolfenden L, Shelton RC, Doherty E, et al. A systematic review to determine the effect of strategies to sustain chronic disease prevention interventions in clinical and community settings: study protocol. Res Sq [Internet]. 2023 [cited 2024 Apr 19]; Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10312971/

Ingvarsson S, Hasson H, von Thiele Schwarz U, Nilsen P, Powell BJ, Lindberg C, et al. Strategies for de-implementation of low-value care—a scoping review. Implement Sci IS. 2022;17:73.

Lewis CC, Powell BJ, Brewer SK, Nguyen AM, Schriger SH, Vejnoska SF, et al. Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda. BMJ Open. 2021;11:e053474.

Hailemariam M, Bustos T, Montgomery B, Barajas R, Evans LB, Drahota A. Evidence-based intervention sustainability strategies: a systematic review. Implement Sci. 2019;14:N.PAG-N.PAG.

Michie S, Atkins L, West R. The behaviour change wheel. Guide Des Interv 1st Ed G B Silverback Publ. 2014;1003:1010.

Birken SA, Haines ER, Hwang S, Chambers DA, Bunger AC, Nilsen P. Advancing understanding and identifying strategies for sustaining evidence-based practices: a review of reviews. Implement Sci IS. 2020;15:88.

Metz A, Jensen T, Farley A, Boaz A, Bartley L, Villodas M. Building trusting relationships to support implementation: A proposed theoretical model. Front Health Serv. 2022;2:894599.

Rabin BA, Cain KL, Watson P, Oswald W, Laurent LC, Meadows AR, et al. Scaling and sustaining COVID-19 vaccination through meaningful community engagement and care coordination for underserved communities: hybrid type 3 effectiveness-implementation sequential multiple assignment randomized trial. Implement Sci IS. 2023;18:28.

Gyamfi J, Iwelunmor J, Patel S, Irazola V, Aifah A, Rakhra A, et al. Implementation outcomes and strategies for delivering evidence-based hypertension interventions in lower-middle-income countries: Evidence from a multi-country consortium for hypertension control. PLOS ONE. 2023;18:e0286204.

Woodward EN, Ball IA, Willging C, Singh RS, Scanlon C, Cluck D, et al. Increasing consumer engagement: tools to engage service users in quality improvement or implementation efforts. Front Health Serv. 2023;3:1124290.

Norton WE, Chambers DA. Unpacking the complexities of de-implementing inappropriate health interventions. Implement Sci IS. 2020;15:2.

Norton WE, McCaskill-Stevens W, Chambers DA, Stella PJ, Brawley OW, Kramer BS. DeImplementing Ineffective and Low-Value Clinical Practices: Research and Practice Opportunities in Community Oncology Settings. JNCI Cancer Spectr. 2021;5:pkab020.

McKay VR, Proctor EK, Morshed AB, Brownson RC, Prusaczyk B. Letting Go: Conceptualizing Intervention De-implementation in Public Health and Social Service Settings. Am J Community Psychol. 2018;62:189–202.

Patey AM, Grimshaw JM, Francis JJ. Changing behaviour, ‘more or less’: do implementation and de-implementation interventions include different behaviour change techniques? Implement Sci IS. 2021;16:20.

Rodriguez Weno E, Allen P, Mazzucca S, Farah Saliba L, Padek M, Moreland-Russell S, et al. Approaches for Ending Ineffective Programs: Strategies From State Public Health Practitioners. Front Public Health. 2021;9:727005.

Gnjidic D, Elshaug AG. De-adoption and its 43 related terms: harmonizing low-value care terminology. BMC Med. 2015;13:273.

Download references

Acknowledgements

The authors would like to acknowledge the early contributions of the Pittsburgh Dissemination and Implementation Science Collaborative (Pitt DISC). LEA would like to thank Dr. Billie Davis for analytical support. The authors would like to acknowledge the implementation science experts who recommended articles for our review, including Greg Aarons, Mark Bauer, Rinad Beidas, Geoffrey Curran, Laura Damschroder, Rani Elwy, Amy Kilbourne, JoAnn Kirchner, Jennifer Leeman, Cara Lewis, Dennis Li, Aaron Lyon, Gila Neta, and Borsika Rabin.

Dr. Rogal’s time was funded in part by a University of Pittsburgh K award (K23-DA048182) and by a VA Health Services Research and Development grant (PEC 19-207). Drs. Bachrach and Quinn were supported by VA HSR Career Development Awards (CDA 20-057, PI: Bachrach; CDA 20-224, PI: Quinn). Dr. Scheunemann’s time was funded by the US Agency for Healthcare Research and Quality (K08HS027210). Drs. Hero, Chinman, Goodrich, Ernecoff, and Mr. Qureshi were funded by the Patient-Centered Outcomes Research Institute (PCORI) AOSEPP2 Task Order 12 to conduct a landscape review of US studies on the effectiveness of implementation strategies with results reported here ( https://www.pcori.org/sites/default/files/PCORI-Implementation-Strategies-for-Evidence-Based-Practice-in-Health-and-Health-Care-A-Review-of-the-Evidence-Full-Report.pdf and https://www.pcori.org/sites/default/files/PCORI-Implementation-Strategies-for-Evidence-Based-Practice-in-Health-and-Health-Care-Brief-Report-Summary.pdf ). Dr. Ashcraft and Ms. Phares were funded by the Center for Health Equity Research and Promotion, (CIN 13-405). The funders had no involvement in this study.

Author information

Shari S. Rogal and Matthew J. Chinman are co-senior authors.

Authors and Affiliations

Center for Health Equity Research and Promotion, Corporal Michael Crescenz VA Medical Center, Philadelphia, PA, USA

Laura Ellen Ashcraft

Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA

Center for Health Equity Research and Promotion, VA Pittsburgh Healthcare System, Pittsburgh, PA, USA

David E. Goodrich, Angela Phares, Deirdre A. Quinn, Shari S. Rogal & Matthew J. Chinman

Division of General Internal Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, PA, USA

David E. Goodrich, Deirdre A. Quinn & Matthew J. Chinman

Clinical & Translational Science Institute, University of Pittsburgh, Pittsburgh, PA, USA

David E. Goodrich & Lisa G. Lederer

RAND Corporation, Pittsburgh, PA, USA

Joachim Hero, Nabeel Qureshi, Natalie C. Ernecoff & Matthew J. Chinman

Center for Clinical Management Research, VA Ann Arbor Healthcare System, Ann Arbor, Michigan, USA

Rachel L. Bachrach

Department of Psychiatry, University of Michigan Medical School, Ann Arbor, MI, USA

Division of Geriatric Medicine, University of Pittsburgh, Department of Medicine, Pittsburgh, PA, USA

Leslie Page Scheunemann

Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, University of Pittsburgh, Department of Medicine, Pittsburgh, PA, USA

Departments of Medicine and Surgery, University of Pittsburgh, Pittsburgh, Pennsylvania, USA

Shari S. Rogal

You can also search for this author in PubMed Google Scholar

Contributions

LEA, SSR, and MJC conceptualized the study. LEA, SSR, MJC, and JOH developed the study design. LEA and JOH acquired the data. LEA, DEG, AP, RLB, DAQ, LGL, LPS, SSR, NQ, and MJC conducted the abstract, full text review, and rigor assessment. LEA, DEG, JOH, AP, RLB, DAQ, NQ, NCE, SSR, and MJC conducted the data abstraction. DEG, SSR, and MJC adjudicated conflicts. LEA and SSR analyzed the data. LEA, SSR, JOH, and MJC interpreted the data. LEA, SSR, and MJC drafted the work. All authors substantially revised the work. All authors approved the submitted version and agreed to be personally accountable for their contributions and the integrity of the work.

Corresponding author

Correspondence to Laura Ellen Ashcraft .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

The manuscript does not contain any individual person’s data.

Competing interests

Additional information, publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., supplementary material 5., supplementary material 6., supplementary material 7., supplementary material 8., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ashcraft, L.E., Goodrich, D.E., Hero, J. et al. A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022. Implementation Sci 19 , 43 (2024). https://doi.org/10.1186/s13012-024-01369-5

Download citation

Received : 09 November 2023

Accepted : 27 May 2024

Published : 24 June 2024

DOI : https://doi.org/10.1186/s13012-024-01369-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Implementation strategy
Health-related outcomes

Implementation Science

ISSN: 1748-5908

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

Open access
Published: 03 July 2024

The impact of evidence-based nursing leadership in healthcare settings: a mixed methods systematic review

Maritta Välimäki 1 , 2 ,
Shuang Hu 3 ,
Tella Lantta 1 ,
Kirsi Hipp 1 , 4 ,
Jaakko Varpula 1 ,
Jiarui Chen 3 ,
Gaoming Liu 5 ,
Yao Tang 3 ,
Wenjun Chen 3 &
Xianhong Li 3

BMC Nursing volume 23 , Article number: 452 ( 2024 ) Cite this article

Metrics details

The central component in impactful healthcare decisions is evidence. Understanding how nurse leaders use evidence in their own managerial decision making is still limited. This mixed methods systematic review aimed to examine how evidence is used to solve leadership problems and to describe the measured and perceived effects of evidence-based leadership on nurse leaders and their performance, organizational, and clinical outcomes.

We included articles using any type of research design. We referred nurses, nurse managers or other nursing staff working in a healthcare context when they attempt to influence the behavior of individuals or a group in an organization using an evidence-based approach. Seven databases were searched until 11 November 2021. JBI Critical Appraisal Checklist for Quasi-experimental studies, JBI Critical Appraisal Checklist for Case Series, Mixed Methods Appraisal Tool were used to evaluate the Risk of bias in quasi-experimental studies, case series, mixed methods studies, respectively. The JBI approach to mixed methods systematic reviews was followed, and a parallel-results convergent approach to synthesis and integration was adopted.

Thirty-one publications were eligible for the analysis: case series ( n = 27), mixed methods studies ( n = 3) and quasi-experimental studies ( n = 1). All studies were included regardless of methodological quality. Leadership problems were related to the implementation of knowledge into practice, the quality of nursing care and the resource availability. Organizational data was used in 27 studies to understand leadership problems, scientific evidence from literature was sought in 26 studies, and stakeholders’ views were explored in 24 studies. Perceived and measured effects of evidence-based leadership focused on nurses’ performance, organizational outcomes, and clinical outcomes. Economic data were not available.

Conclusions

This is the first systematic review to examine how evidence is used to solve leadership problems and to describe its measured and perceived effects from different sites. Although a variety of perceptions and effects were identified on nurses’ performance as well as on organizational and clinical outcomes, available knowledge concerning evidence-based leadership is currently insufficient. Therefore, more high-quality research and clinical trial designs are still needed.

Trail registration

The study was registered (PROSPERO CRD42021259624).

Peer Review reports

Global health demands have set new roles for nurse leaders [ 1 ].Nurse leaders are referred to as nurses, nurse managers, or other nursing staff working in a healthcare context who attempt to influence the behavior of individuals or a group based on goals that are congruent with organizational goals [ 2 ]. They are seen as professionals “armed with data and evidence, and a commitment to mentorship and education”, and as a group in which “leaders innovate, transform, and achieve quality outcomes for patients, health care professionals, organizations, and communities” [ 3 ]. Effective leadership occurs when team members critically follow leaders and are motivated by a leader’s decisions based on the organization’s requests and targets [ 4 ]. On the other hand, problems caused by poor leadership may also occur, regarding staff relations, stress, sickness, or retention [ 5 ]. Therefore, leadership requires an understanding of different problems to be solved using synthesizing evidence from research, clinical expertise, and stakeholders’ preferences [ 6 , 7 ]. If based on evidence, leadership decisions, also referred as leadership decision making [ 8 ], could ensure adequate staffing [ 7 , 9 ] and to produce sufficient and cost-effective care [ 10 ]. However, nurse leaders still rely on their decision making on their personal [ 11 ] and professional experience [ 10 ] over research evidence, which can lead to deficiencies in the quality and safety of care delivery [ 12 , 13 , 14 ]. As all nurses should demonstrate leadership in their profession, their leadership competencies should be strengthened [ 15 ].

Evidence-informed decision-making, referred to as evidence appraisal and application, and evaluation of decisions [ 16 ], has been recognized as one of the core competencies for leaders [ 17 , 18 ]. The role of evidence in nurse leaders’ managerial decision making has been promoted by public authorities [ 19 , 20 , 21 ]. Evidence-based management, another concept related to evidence-based leadership, has been used as the potential to improve healthcare services [ 22 ]. It can guide nursing leaders, in developing working conditions, staff retention, implementation practices, strategic planning, patient care, and success of leadership [ 13 ]. Collins and Holton [ 23 ] in their systematic review and meta-analysis examined 83 studies regarding leadership development interventions. They found that leadership training can result in significant improvement in participants’ skills, especially in knowledge level, although the training effects varied across studies. Cummings et al. [ 24 ] reviewed 100 papers (93 studies) and concluded that participation in leadership interventions had a positive impact on the development of a variety of leadership styles. Clavijo-Chamorro et al. [ 25 ] in their review of 11 studies focused on leadership-related factors that facilitate evidence implementation: teamwork, organizational structures, and transformational leadership. The role of nurse managers was to facilitate evidence-based practices by transforming contexts to motivate the staff and move toward a shared vision of change.

As far as we are aware, however, only a few systematic reviews have focused on evidence-based leadership or related concepts in the healthcare context aiming to analyse how nurse leaders themselves uses evidence in the decision-making process. Young [ 26 ] targeted definitions and acceptance of evidence-based management (EBMgt) in healthcare while Hasanpoor et al. [ 22 ] identified facilitators and barriers, sources of evidence used, and the role of evidence in the process of decision making. Both these reviews concluded that EBMgt was of great importance but used limitedly in healthcare settings due to a lack of time, a lack of research management activities, and policy constraints. A review by Williams [ 27 ] showed that the usage of evidence to support management in decision making is marginal due to a shortage of relevant evidence. Fraser [ 28 ] in their review further indicated that the potential evidence-based knowledge is not used in decision making by leaders as effectively as it could be. Non-use of evidence occurs and leaders base their decisions mainly on single studies, real-world evidence, and experts’ opinions [ 29 ]. Systematic reviews and meta-analyses rarely provide evidence of management-related interventions [ 30 ]. Tate et al. [ 31 ] concluded based on their systematic review and meta-analysis that the ability of nurse leaders to use and critically appraise research evidence may influence the way policy is enacted and how resources and staff are used to meet certain objectives set by policy. This can further influence staff and workforce outcomes. It is therefore important that nurse leaders have the capacity and motivation to use the strongest evidence available to effect change and guide their decision making [ 27 ].

Despite of a growing body of evidence, we found only one review focusing on the impact of evidence-based knowledge. Geert et al. [ 32 ] reviewed literature from 2007 to 2016 to understand the elements of design, delivery, and evaluation of leadership development interventions that are the most reliably linked to outcomes at the level of the individual and the organization, and that are of most benefit to patients. The authors concluded that it is possible to improve individual-level outcomes among leaders, such as knowledge, motivation, skills, and behavior change using evidence-based approaches. Some of the most effective interventions included, for example, interactive workshops, coaching, action learning, and mentoring. However, these authors found limited research evidence describing how nurse leaders themselves use evidence to support their managerial decisions in nursing and what the outcomes are.

To fill the knowledge gap and compliment to existing knowledgebase, in this mixed methods review we aimed to (1) examine what leadership problems nurse leaders solve using an evidence-based approach and (2) how they use evidence to solve these problems. We also explored (3) the measured and (4) perceived effects of the evidence-based leadership approach in healthcare settings. Both qualitative and quantitative components of the effects of evidence-based leadership were examined to provide greater insights into the available literature [ 33 ]. Together with the evidence-based leadership approach, and its impact on nursing [ 34 , 35 ], this knowledge gained in this review can be used to inform clinical policy or organizational decisions [ 33 ]. The study is registered (PROSPERO CRD42021259624). The methods used in this review were specified in advance and documented in a priori in a published protocol [ 36 ]. Key terms of the review and the search terms are defined in Table 1 (population, intervention, comparison, outcomes, context, other).

In this review, we used a mixed methods approach [ 37 ]. A mixed methods systematic review was selected as this approach has the potential to produce direct relevance to policy makers and practitioners [ 38 ]. Johnson and Onwuegbuzie [ 39 ] have defined mixed methods research as “the class of research in which the researcher mixes or combines quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study.” Therefore, we combined quantitative and narrative analysis to appraise and synthesize empirical evidence, and we held them as equally important in informing clinical policy or organizational decisions [ 34 ]. In this review, a comprehensive synthesis of quantitative and qualitative data was performed first and then discussed in discussion part (parallel-results convergent design) [ 40 ]. We hoped that different type of analysis approaches could complement each other and deeper picture of the topic in line with our research questions could be gained [ 34 ].

Inclusion and exclusion criteria

Inclusion and exclusion criteria of the study are described in Table 1 .

Search strategy

A three-step search strategy was utilized. First, an initial limited search with #MEDLINE was undertaken, followed by analysis of the words used in the title, abstract, and the article’s key index terms. Second, the search strategy, including identified keywords and index terms, was adapted for each included data base and a second search was undertaken on 11 November 2021. The full search strategy for each database is described in Additional file 1 . Third, the reference list of all studies included in the review were screened for additional studies. No year limits or language restrictions were used.

Information sources

The database search included the following: CINAHL (EBSCO), Cochrane Library (academic database for medicine and health science and nursing), Embase (Elsevier), PsycINFO (EBSCO), PubMed (MEDLINE), Scopus (Elsevier) and Web of Science (academic database across all scientific and technical disciplines, ranging from medicine and social sciences to arts and humanities). These databases were selected as they represent typical databases in health care context. Subject headings from each of the databases were included in the search strategies. Boolean operators ‘AND’ and ‘OR’ were used to combine the search terms. An information specialist from the University of Turku Library was consulted in the formation of the search strategies.

Study selection

All identified citations were collated and uploaded into Covidence software (Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia www.covidence.org ), and duplicates were removed by the software. Titles and abstracts were screened and assessed against the inclusion criteria independently by two reviewers out of four, and any discrepancies were resolved by the third reviewer (MV, KH, TL, WC). Studies meeting the inclusion criteria were retrieved in full and archived in Covidence. Access to one full-text article was lacking: the authors for one study were contacted about the missing full text, but no full text was received. All remaining hits of the included studies were retrieved and assessed independently against the inclusion criteria by two independent reviewers of four (MV, KH, TL, WC). Studies that did not meet the inclusion criteria were excluded, and the reasons for exclusion were recorded in Covidence. Any disagreements that arose between the reviewers were resolved through discussions with XL.

Assessment of methodological quality

Eligible studies were critically appraised by two independent reviewers (YT, SH). Standardized critical appraisal instruments based on the study design were used. First, quasi-experimental studies were assessed using the JBI Critical Appraisal Checklist for Quasi-experimental studies [ 44 ]. Second, case series were assessed using the JBI Critical Appraisal Checklist for Case Series [ 45 ]. Third, mixed methods studies were appraised using the Mixed Methods Appraisal Tool [ 46 ].

To increase inter-reviewer reliability, the review agreement was calculated (SH) [ 47 ]. A kappa greater than 0.8 was considered to represent a high level of agreement (0–0.1). In our data, the agreement was 0.75. Discrepancies raised between two reviewers were resolved through discussion and modifications and confirmed by XL. As an outcome, studies that met the inclusion criteria were proceeded to critical appraisal and assessed as suitable for inclusion in the review. The scores for each item and overall critical appraisal scores were presented.

Data extraction

For data extraction, specific tables were created. First, study characteristics (author(s), year, country, design, number of participants, setting) were extracted by two authors independently (JC, MV) and reviewed by TL. Second, descriptions of the interventions were extracted by two reviewers (JV, JC) using the structure of the TIDIeR (Template for Intervention Description and Replication) checklist (brief name, the goal of the intervention, material and procedure, models of delivery and location, dose, modification, adherence and fidelity) [ 48 ]. The extractions were confirmed (MV).

Third, due to a lack of effectiveness data and a wide heterogeneity between study designs and presentation of outcomes, no attempt was made to pool the quantitative data statistically; the findings of the quantitative data were presented in narrative form only [ 44 ]. The separate data extraction tables for each research question were designed specifically for this study. For both qualitative (and a qualitative component of mixed-method studies) and quantitative studies, the data were extracted and tabulated into text format according to preplanned research questions [ 36 ]. To test the quality of the tables and the data extraction process, three authors independently extracted the data from the first five studies (in alphabetical order). After that, the authors came together to share and determine whether their approaches of the data extraction were consistent with each other’s output and whether the content of each table was in line with research question. No reason was found to modify the data extraction tables or planned process. After a consensus of the data extraction process was reached, the data were extracted in pairs by independent reviewers (WC, TY, SH, GL). Any disagreements that arose between the reviewers were resolved through discussion and with a third reviewer (MV).

Data analysis

We were not able to conduct a meta-analysis due to a lack of effectiveness data based on clinical trials. Instead, we used inductive thematic analysis with constant comparison to answer the research question [ 46 , 49 ] using tabulated primary data from qualitative and quantitative studies as reported by the original authors in narrative form only [ 47 ]. In addition, the qualitizing process was used to transform quantitative data to qualitative data; this helped us to convert the whole data into themes and categories. After that we used the thematic analysis for the narrative data as follows. First, the text was carefully read, line by line, to reveal topics answering each specific review question (MV). Second, the data coding was conducted, and the themes in the data were formed by data categorization. The process of deriving the themes was inductive based on constant comparison [ 49 ]. The results of thematic analysis and data categorization was first described in narrative format and then the total number of studies was calculated where the specific category was identified (%).

Stakeholder involvement

The method of reporting stakeholders’ involvement follows the key components by [ 50 ]: (1) people involved, (2) geographical location, (3) how people were recruited, (4) format of involvement, (5) amount of involvement, (6) ethical approval, (7) financial compensation, and (8) methods for reporting involvement.

In our review, stakeholder involvement targeted nurses and nurse leader in China. Nurse Directors of two hospitals recommended potential participants who received a personal invitation letter from researchers to participate in a discussion meeting. Stakeholders’ participation was based on their own free will. Due to COVID-19, one online meeting (1 h) was organized (25 May 2022). Eleven participants joined the meeting. Ethical approval was not applied and no financial compensation was offered. At the end of the meeting, experiences of stakeholders’ involvement were explored.

The meeting started with an introductory presentation with power points. The rationale, methods, and preliminary review results were shared with the participants [ 51 ].The meeting continued with general questions for the participants: (1) Are you aware of the concepts of evidence-based practice or evidence-based leadership?; (2) How important is it to use evidence to support decisions among nurse leaders?; (3) How is the evidence-based approach used in hospital settings?; and (4) What type of evidence is currently used to support nurse leaders’ decision making (e.g. scientific literature, organizational data, stakeholder views)?

Two people took notes on the course and content of the conversation. The notes were later transcripted in verbatim, and the key points of the discussions were summarised. Although answers offered by the stakeholders were very short, the information was useful to validate the preliminary content of the results, add the rigorousness of the review, and obtain additional perspectives. A recommendation of the stakeholders was combined in the Discussion part of this review increasing the applicability of the review in the real world [ 50 ]. At the end of the discussion, the value of stakeholders’ involvement was asked. Participants shared that the experience of participating was unique and the topic of discussion was challenging. Two authors of the review group further represented stakeholders by working together with the research team throughout the review study.

Search results

From seven different electronic databases, 6053 citations were identified as being potentially relevant to the review. Then, 3133 duplicates were removed by an automation tool (Covidence: www.covidence.org ), and one was removed manually. The titles and abstracts of 3040 of citations were reviewed, and a total of 110 full texts were included (one extra citation was found on the reference list but later excluded). Based on the eligibility criteria, 31 studies (32 hits) were critically appraised and deemed suitable for inclusion in the review. The search results and selection process are presented in the PRISMA [ 52 ] flow diagram Fig. 1 . The full list of references for included studies can be find in Additional file 2 . To avoid confusion between articles of the reference list and studies included in the analysis, the studies included in the review are referred inside the article using the reference number of each study (e.g. ref 1, ref 2).

Search results and study selection and inclusion process [ 52 ]

Characteristics of included studies

The studies had multiple purposes, aiming to develop practice, implement a new approach, improve quality, or to develop a model. The 31 studies (across 32 hits) were case series studies ( n = 27), mixed methods studies ( n = 3) and a quasi-experimental study ( n = 1). All studies were published between the years 2004 and 2021. The highest number of papers was published in year 2020.

Table 2 describes the characteristics of included studies and Additional file 3 offers a narrative description of the studies.

Methodological quality assessment

Quasi-experimental studies.

We had one quasi-experimental study (ref 31). All questions in the critical appraisal tool were applicable. The total score of the study was 8 (out of a possible 9). Only one response of the tool was ‘no’ because no control group was used in the study (see Additional file 4 for the critical appraisal of included studies).

Case series studies . A case series study is typically defined as a collection of subjects with common characteristics. The studies do not include a comparison group and are often based on prevalent cases and on a sample of convenience [ 53 ]. Munn et al. [ 45 ] further claim that case series are best described as observational studies, lacking experimental and randomized characteristics, being descriptive studies, without a control or comparator group. Out of 27 case series studies included in our review, the critical appraisal scores varied from 1 to 9. Five references were conference abstracts with empirical study results, which were scored from 1 to 3. Full reports of these studies were searched in electronic databases but not found. Critical appraisal scores for the remaining 22 studies ranged from 1 to 9 out of a possible score of 10. One question (Q3) was not applicable to 13 studies: “Were valid methods used for identification of the condition for all participants included in the case series?” Only two studies had clearly reported the demographic of the participants in the study (Q6). Twenty studies met Criteria 8 (“Were the outcomes or follow-up results of cases clearly reported?”) and 18 studies met Criteria 7 (“Q7: Was there clear reporting of clinical information of the participants?”) (see Additional file 4 for the critical appraisal of included studies).

Mixed-methods studies

Mixed-methods studies involve a combination of qualitative and quantitative methods. This is a common design and includes convergent design, sequential explanatory design, and sequential exploratory design [ 46 ]. There were three mixed-methods studies. The critical appraisal scores for the three studies ranged from 60 to 100% out of a possible 100%. Two studies met all the criteria, while one study fulfilled 60% of the scored criteria due to a lack of information to understand the relevance of the sampling strategy well enough to address the research question (Q4.1) or to determine whether the risk of nonresponse bias was low (Q4.4) (see Additional file 4 for the critical appraisal of included studies).

Intervention or program components

The intervention of program components were categorized and described using the TiDier checklist: name and goal, theory or background, material, procedure, provider, models of delivery, location, dose, modification, and adherence and fidelity [ 48 ]. A description of intervention in each study is described in Additional file 5 and a narrative description in Additional file 6 .

Leadership problems

In line with the inclusion criteria, data for the leadership problems were categorized in all 31 included studies (see Additional file 7 for leadership problems). Three types of leadership problems were identified: implementation of knowledge into practice, the quality of clinical care, and resources in nursing care. A narrative summary of the results is reported below.

Implementing knowledge into practice

Eleven studies (35%) aimed to solve leadership problems related to implementation of knowledge into practice. Studies showed how to support nurses in evidence-based implementation (EBP) (ref 3, ref 5), how to engage nurses in using evidence in practice (ref 4), how to convey the importance of EBP (ref 22) or how to change practice (ref 4). Other problems were how to facilitate nurses to use guideline recommendations (ref 7) and how nurses can make evidence-informed decisions (ref 8). General concerns also included the linkage between theory and practice (ref 1) as well as how to implement the EBP model in practice (ref 6). In addition, studies were motivated by the need for revisions or updates of protocols to improve clinical practice (ref 10) as well as the need to standardize nursing activities (ref 11, ref 14).

The quality of the care

Thirteen (42%) focused on solving problems related to the quality of clinical care. In these studies, a high number of catheter infections led a lack of achievement of organizational goals (ref 2, ref 9). A need to reduce patient symptoms in stem cell transplant patients undergoing high-dose chemotherapy (ref 24) was also one of the problems to be solved. In addition, the projects focused on how to prevent pressure ulcers (ref 26, ref 29), how to enhance the quality of cancer treatment (ref 25) and how to reduce the need for invasive constipation treatment (ref 30). Concerns about patient safety (ref 15), high fall rates (ref 16, ref 19), dissatisfaction of patients (ref 16, ref 18) and nurses (ref 16, ref 30) were also problems that had initiated the projects. Studies addressed concerns about how to promote good contingency care in residential aged care homes (ref 20) and about how to increase recognition of human trafficking problems in healthcare (ref 21).

Resources in nursing care

Nurse leaders identified problems in their resources, especially in staffing problems. These problems were identified in seven studies (23%), which involved concerns about how to prevent nurses from leaving the job (ref 31), how to ensure appropriate recruitment, staffing and retaining of nurses (ref 13) and how to decrease nurses’ burden and time spent on nursing activities (ref 12). Leadership turnover was also reported as a source of dissatisfaction (ref 17); studies addressed a lack of structured transition and training programs, which led to turnover (ref 23), as well as how to improve intershift handoff among nurses (ref 28). Optimal design for new hospitals was also examined (ref 27).

Main features of evidence-based leadership

Out of 31 studies, 17 (55%) included all four domains of an evidence-based leadership approach, and four studies (13%) included evidence of critical appraisal of the results (see Additional file 8 for the main features of evidence-based Leadership) (ref 11, ref 14, ref 23, ref 27).

Organizational evidence

Twenty-seven studies (87%) reported how organizational evidence was collected and used to solve leadership problems (ref 2). Retrospective chart reviews (ref 5), a review of the extent of specific incidents (ref 19), and chart auditing (ref 7, ref 25) were conducted. A gap between guideline recommendations and actual care was identified using organizational data (ref 7) while the percentage of nurses’ working time spent on patient care was analyzed using an electronic charting system (ref 12). Internal data (ref 22), institutional data, and programming metrics were also analyzed to understand the development of the nurse workforce (ref 13).

Surveys (ref 3, ref 25), interviews (ref 3, ref 25) and group reviews (ref 18) were used to better understand the leadership problem to be solved. Employee opinion surveys on leadership (ref 17), a nurse satisfaction survey (ref 30) and a variety of reporting templates were used for the data collection (ref 28) reported. Sometimes, leadership problems were identified by evidence facilitators or a PI’s team who worked with staff members (ref 15, ref 17). Problems in clinical practice were also identified by the Nursing Professional Council (ref 14), managers (ref 26) or nurses themselves (ref 24). Current practices were reviewed (ref 29) and a gap analysis was conducted (ref 4, ref 16, ref 23) together with SWOT analysis (ref 16). In addition, hospital mission and vision statements, research culture established and the proportion of nursing alumni with formal EBP training were analyzed (ref 5). On the other hand, it was stated that no systematic hospital-specific sources of data regarding job satisfaction or organizational commitment were used (ref 31). In addition, statements of organizational analysis were used on a general level only (ref 1).

Scientific evidence identified

Twenty-six studies (84%) reported the use of scientific evidence in their evidence-based leadership processes. A literature search was conducted (ref 21) and questions, PICO, and keywords were identified (ref 4) in collaboration with a librarian. Electronic databases, including PubMed (ref 14, ref 31), Cochrane, and EMBASE (ref 31) were searched. Galiano (ref 6) used Wiley Online Library, Elsevier, CINAHL, Health Source: Nursing/Academic Edition, PubMed, and the Cochrane Library while Hoke (ref 11) conducted an electronic search using CINAHL and PubMed to retrieve articles.

Identified journals were reviewed manually (ref 31). The findings were summarized using ‘elevator speech’ (ref 4). In a study by Gifford et al. (ref 9) evidence facilitators worked with participants to access, appraise, and adapt the research evidence to the organizational context. Ostaszkiewicz (ref 20) conducted a scoping review of literature and identified and reviewed frameworks and policy documents about the topic and the quality standards. Further, a team of nursing administrators, directors, staff nurses, and a patient representative reviewed the literature and made recommendations for practice changes.

Clinical practice guidelines were also used to offer scientific evidence (ref 7, ref 19). Evidence was further retrieved from a combination of nursing policies, guidelines, journal articles, and textbooks (ref 12) as well as from published guidelines and literature (ref 13). Internal evidence, professional practice knowledge, relevant theories and models were synthesized (ref 24) while other study (ref 25) reviewed individual studies, synthesized with systematic reviews or clinical practice guidelines. The team reviewed the research evidence (ref 3, ref 15) or conducted a literature review (ref 22, ref 28, ref 29), a literature search (ref 27), a systematic review (ref 23), a review of the literature (ref 30) or ‘the scholarly literature was reviewed’ (ref 18). In addition, ‘an extensive literature review of evidence-based best practices was carried out’ (ref 10). However, detailed description how the review was conducted was lacking.

Views of stakeholders

A total of 24 studies (77%) reported methods for how the views of stakeholders, i.e., professionals or experts, were considered. Support to run this study was received from nursing leadership and multidisciplinary teams (ref 29). Experts and stakeholders joined the study team in some cases (ref 25, ref 30), and in other studies, their opinions were sought to facilitate project success (ref 3). Sometimes a steering committee was formed by a Chief Nursing Officer and Clinical Practice Specialists (ref 2). More specifically, stakeholders’ views were considered using interviews, workshops and follow-up teleconferences (ref 7). The literature review was discussed with colleagues (ref 11), and feedback and support from physicians as well as the consensus of staff were sought (ref 16).

A summary of the project findings and suggestions for the studies were discussed at 90-minute weekly meetings by 11 charge nurses. Nurse executive directors were consulted over a 10-week period (ref 31). An implementation team (nurse, dietician, physiotherapist, occupational therapist) was formed to support the implementation of evidence-based prevention measures (ref 26). Stakeholders volunteered to join in the pilot implementation (ref 28) or a stakeholder team met to determine the best strategy for change management, shortcomings in evidence-based criteria were discussed, and strategies to address those areas were planned (ref 5). Nursing leaders, staff members (ref 22), ‘process owners (ref 18) and program team members (ref 18, ref 19, ref 24) met regularly to discuss the problems. Critical input was sought from clinical educators, physicians, nutritionists, pharmacists, and nurse managers (ref 24). The unit director and senior nursing staff reviewed the contents of the product, and the final version of clinical pathways were reviewed and approved by the Quality Control Commission of the Nursing Department (ref 12). In addition, two co-design workshops with 18 residential aged care stakeholders were organized to explore their perspectives about factors to include in a model prototype (ref 20). Further, an agreement of stakeholders in implementing continuous quality services within an open relationship was conducted (ref 1).

Critical appraisal

In five studies (16%), a critical appraisal targeting the literature search was carried out. The appraisals were conducted by interns and teams who critiqued the evidence (ref 4). In Hoke’s study, four areas that had emerged in the literature were critically reviewed (ref 11). Other methods were to ‘critically appraise the search results’ (ref 14). Journal club team meetings (ref 23) were organized to grade the level and quality of evidence and the team ‘critically appraised relevant evidence’ (ref 27). On the other hand, the studies lacked details of how the appraisals were done in each study.

The perceived effects of evidence-based leadership

Perceived effects of evidence-based leadership on nurses’ performance.

Eleven studies (35%) described perceived effects of evidence-based leadership on nurses’ performance (see Additional file 9 for perceived effects of evidence-based leadership), which were categorized in four groups: awareness and knowledge, competence, ability to understand patients’ needs, and engagement. First, regarding ‘awareness and knowledge’, different projects provided nurses with new learning opportunities (ref 3). Staff’s knowledge (ref 20, ref 28), skills, and education levels improved (ref 20), as did nurses’ knowledge comprehension (ref 21). Second, interventions and approaches focusing on management and leadership positively influenced participants’ competence level to improve the quality of services. Their confidence level (ref 1) and motivation to change practice increased, self-esteem improved, and they were more positive and enthusiastic in their work (ref 22). Third, some nurses were relieved that they had learned to better handle patients’ needs (ref 25). For example, a systematic work approach increased nurses’ awareness of the patients who were at risk of developing health problems (ref 26). And last, nurse leaders were more engaged with staff, encouraging them to adopt the new practices and recognizing their efforts to change (ref 8).

Perceived effects on organizational outcomes

Nine studies (29%) described the perceived effects of evidence-based leadership on organizational outcomes (see Additional file 9 for perceived effects of evidence-based leadership). These were categorized into three groups: use of resources, staff commitment, and team effort. First, more appropriate use of resources was reported (ref 15, ref 20), and working time was more efficiently used (ref 16). In generally, a structured approach made implementing change more manageable (ref 1). On the other hand, in the beginning of the change process, the feedback from nurses was unfavorable, and they experienced discomfort in the new work style (ref 29). New approaches were also perceived as time consuming (ref 3). Second, nurse leaders believed that fewer nursing staff than expected left the organization over the course of the study (ref 31). Third, the project helped staff in their efforts to make changes, and it validated the importance of working as a team (ref 7). Collaboration and support between the nurses increased (ref 26). On the other hand, new work style caused challenges in teamwork (ref 3).

Perceived effects on clinical outcomes

Five studies (16%) reported the perceived effects of evidence-based leadership on clinical outcomes (see Additional file 9 for perceived effects of evidence-based leadership), which were categorized in two groups: general patient outcomes and specific clinical outcomes. First, in general, the project assisted in connecting the guideline recommendations and patient outcomes (ref 7). The project was good for the patients in general, and especially to improve patient safety (ref 16). On the other hand, some nurses thought that the new working style did not work at all for patients (ref 28). Second, the new approach used assisted in optimizing patients’ clinical problems and person-centered care (ref 20). Bowel management, for example, received very good feedback (ref 30).

The measured effects of evidence-based leadership

The measured effects on nurses’ performance.

Data were obtained from 20 studies (65%) (see Additional file 10 for measured effects of evidence-based leadership) and categorized nurse performance outcomes for three groups: awareness and knowledge, engagement, and satisfaction. First, six studies (19%) measured the awareness and knowledge levels of participants. Internship for staff nurses was beneficial to help participants to understand the process for using evidence-based practice and to grow professionally, to stimulate for innovative thinking, to give knowledge needed to use evidence-based practice to answer clinical questions, and to make possible to complete an evidence-based practice project (ref 3). Regarding implementation program of evidence-based practice, those with formal EBP training showed an improvement in knowledge, attitude, confidence, awareness and application after intervention (ref 3, ref 11, ref 20, ref 23, ref 25). On the contrary, in other study, attitude towards EBP remained stable ( p = 0.543). and those who applied EBP decreased although no significant differences over the years ( p = 0.879) (ref 6).

Second, 10 studies (35%) described nurses’ engagement to new practices (ref 5, ref 6, ref 7, ref 10, ref 16, ref 17, ref 18, ref 21, ref 25, ref 27). 9 studies (29%) studies reported that there was an improvement of compliance level of participants (ref 6, ref 7, ref 10, ref 16, ref 17, ref 18, ref 21, ref 25, ref 27). On the contrary, in DeLeskey’s (ref 5) study, although improvement was found in post-operative nausea and vomiting’s (PONV) risk factors documented’ (2.5–63%), and ’risk factors communicated among anaesthesia and surgical staff’ (0–62%), the improvement did not achieve the goal. The reason was a limited improvement was analysed. It was noted that only those patients who had been seen by the pre-admission testing nurse had risk assessments completed. Appropriate treatment/prophylaxis increased from 69 to 77%, and from 30 to 49%; routine assessment for PONV/rescue treatment 97% and 100% was both at 100% following the project. The results were discussed with staff but further reasons for a lack of engagement in nursing care was not reported.

And third, six studies (19%) reported nurses’ satisfaction with project outcomes. The study results showed that using evidence in managerial decisions improved nurses’ satisfaction and attitudes toward their organization ( P < 0.05) (ref 31). Nurses’ overall job satisfaction improved as well (ref 17). Nurses’ satisfaction with usability of the electronic charting system significantly improved after introduction of the intervention (ref 12). In handoff project in seven hospitals, improvement was reported in all satisfaction indicators used in the study although improvement level varied in different units (ref 28). In addition, positive changes were reported in nurses’ ability to autonomously perform their job (“How satisfied are you with the tools and resources available for you treat and prevent patient constipation?” (54%, n = 17 vs. 92%, n = 35, p < 0.001) (ref 30).

The measured effects on organizational outcomes

Thirteen studies (42%) described the effects of a project on organizational outcomes (see Additional file 10 for measured effects of evidence-based leadership), which were categorized in two groups: staff compliance, and changes in practices. First, studies reported improved organizational outcomes due to staff better compliance in care (ref 4, ref 13, ref 17, ref 23, ref 27, ref 31). Second, changes in organization practices were also described (ref 11) like changes in patient documentation (ref 12, ref 21). Van Orne (ref 30) found a statistically significant reduction in the average rate of invasive medication administration between pre-intervention and post-intervention ( p = 0.01). Salvador (ref 24) also reported an improvement in a proactive approach to mucositis prevention with an evidence-based oral care guide. On the contrary, concerns were also raised such as not enough time for new bedside report (ref 16) or a lack of improvement of assessment of diabetic ulcer (ref 8).

The measured effects on clinical outcomes

A variety of improvements in clinical outcomes were reported (see Additional file 10 for measured effects of evidence-based leadership): improvement in patient clinical status and satisfaction level. First, a variety of improvement in patient clinical status was reported. improvement in Incidence of CAUTI decreased 27.8% between 2015 and 2019 (ref 2) while a patient-centered quality improvement project reduced CAUTI rates to 0 (ref 10). A significant decrease in transmission rate of MRSA transmission was also reported (ref 27) and in other study incidences of CLABSIs dropped following of CHG bathing (ref 14). Further, it was possible to decrease patient nausea from 18 to 5% and vomiting to 0% (ref 5) while the percentage of patients who left the hospital without being seen was below 2% after the project (ref 17). In addition, a significant reduction in the prevalence of pressure ulcers was found (ref 26, ref 29) and a significant reduction of mucositis severity/distress was achieved (ref 24). Patient falls rate decreased (ref 15, ref 16, ref 19, ref 27).

Second, patient satisfaction level after project implementation improved (ref 28). The scale assessing healthcare providers by consumers showed improvement, but the changes were not statistically significant. Improvement in an emergency department leadership model and in methods of communication with patients improved patient satisfaction scores by 600% (ref 17). In addition, new evidence-based unit improved patient experiences about the unit although not all items improved significantly (ref 18).

Stakeholder involvement in the mixed-method review

To ensure stakeholders’ involvement in the review, the real-world relevance of our research [ 53 ], achieve a higher level of meaning in our review results, and gain new perspectives on our preliminary findings [ 50 ], a meeting with 11 stakeholders was organized. First, we asked if participants were aware of the concepts of evidence-based practice or evidence-based leadership. Responses revealed that participants were familiar with the concept of evidence-based practice, but the topic of evidence-based leadership was totally new. Examples of nurses and nurse leaders’ responses are as follows: “I have heard a concept of evidence-based practice but never a concept of evidence-based leadership.” Another participant described: “I have heard it [evidence-based leadership] but I do not understand what it means.”

Second, as stakeholder involvement is beneficial to the relevance and impact of health research [ 54 ], we asked how important evidence is to them in supporting decisions in health care services. One participant described as follows: “Using evidence in decisions is crucial to the wards and also to the entire hospital.” Third, we asked how the evidence-based approach is used in hospital settings. Participants expressed that literature is commonly used to solve clinical problems in patient care but not to solve leadership problems. “In [patient] medication and care, clinical guidelines are regularly used. However, I am aware only a few cases where evidence has been sought to solve leadership problems.”

And last, we asked what type of evidence is currently used to support nurse leaders’ decision making (e.g. scientific literature, organizational data, stakeholder views)? The participants were aware that different types of information were collected in their organization on a daily basis (e.g. patient satisfaction surveys). However, the information was seldom used to support decision making because nurse leaders did not know how to access this information. Even so, the participants agreed that the use of evidence from different sources was important in approaching any leadership or managerial problems in the organization. Participants also suggested that all nurse leaders should receive systematic training related to the topic; this could support the daily use of the evidence-based approach.

To our knowledge, this article represents the first mixed-methods systematic review to examine leadership problems, how evidence is used to solve these problems and what the perceived and measured effects of evidence-based leadership are on nurse leaders and their performance, organizational, and clinical outcomes. This review has two key findings. First, the available research data suggests that evidence-based leadership has potential in the healthcare context, not only to improve knowledge and skills among nurses, but also to improve organizational outcomes and the quality of patient care. Second, remarkably little published research was found to explore the effects of evidence-based leadership with an efficient trial design. We validated the preliminary results with nurse stakeholders, and confirmed that nursing staff, especially nurse leaders, were not familiar with the concept of evidence-based leadership, nor were they used to implementing evidence into their leadership decisions. Our data was based on many databases, and we screened a large number of studies. We also checked existing registers and databases and found no registered or ongoing similar reviews being conducted. Therefore, our results may not change in the near future.

We found that after identifying the leadership problems, 26 (84%) studies out of 31 used organizational data, 25 (81%) studies used scientific evidence from the literature, and 21 (68%) studies considered the views of stakeholders in attempting to understand specific leadership problems more deeply. However, only four studies critically appraised any of these findings. Considering previous critical statements of nurse leaders’ use of evidence in their decision making [ 14 , 30 , 31 , 34 , 55 ], our results are still quite promising.

Our results support a previous systematic review by Geert et al. [ 32 ], which concluded that it is possible to improve leaders’ individual-level outcomes, such as knowledge, motivation, skills, and behavior change using evidence-based approaches. Collins and Holton [ 23 ] particularly found that leadership training resulted in significant knowledge and skill improvements, although the effects varied widely across studies. In our study, evidence-based leadership was seen to enable changes in clinical practice, especially in patient care. On the other hand, we understand that not all efforts to changes were successful [ 56 , 57 , 58 ]. An evidence-based approach causes negative attitudes and feelings. Negative emotions in participants have also been reported due to changes, such as discomfort with a new working style [ 59 ]. Another study reported inconvenience in using a new intervention and its potential risks for patient confidentiality. Sometimes making changes is more time consuming than continuing with current practice [ 60 ]. These findings may partially explain why new interventions or program do not always fully achieve their goals. On the other hand, Dubose et al. [ 61 ] state that, if prepared with knowledge of resistance, nurse leaders could minimize the potential negative consequences and capitalize on a powerful impact of change adaptation.

We found that only six studies used a specific model or theory to understand the mechanism of change that could guide leadership practices. Participants’ reactions to new approaches may be an important factor in predicting how a new intervention will be implemented into clinical practice. Therefore, stronger effort should be put to better understanding the use of evidence, how participants’ reactions and emotions or practice changes could be predicted or supported using appropriate models or theories, and how using these models are linked with leadership outcomes. In this task, nurse leaders have an important role. At the same time, more responsibilities in developing health services have been put on the shoulders of nurse leaders who may already be suffering under pressure and increased burden at work. Working in a leadership position may also lead to role conflict. A study by Lalleman et al. [ 62 ] found that nurses were used to helping other people, often in ad hoc situations. The helping attitude of nurses combined with structured managerial role may cause dilemmas, which may lead to stress. Many nurse leaders opt to leave their positions less than 5 years [ 63 ].To better fulfill the requirements of health services in the future, the role of nurse leaders in evidence-based leadership needs to be developed further to avoid ethical and practical dilemmas in their leadership practices.

It is worth noting that the perceived and measured effects did not offer strong support to each other but rather opened a new venue to understand the evidence-based leadership. Specifically, the perceived effects did not support to measured effects (competence, ability to understand patients’ needs, use of resources, team effort, and specific clinical outcomes) while the measured effects could not support to perceived effects (nurse’s performance satisfaction, changes in practices, and clinical outcomes satisfaction). These findings may indicate that different outcomes appear if the effects of evidence-based leadership are looked at using different methodological approach. Future study is encouraged using well-designed study method including mixed-method study to examine the consistency between perceived and measured effects of evidence-based leadership in health care.

There is a potential in nursing to support change by demonstrating conceptual and operational commitment to research-based practices [ 64 ]. Nurse leaders are well positioned to influence and lead professional governance, quality improvement, service transformation, change and shared governance [ 65 ]. In this task, evidence-based leadership could be a key in solving deficiencies in the quality, safety of care [ 14 ] and inefficiencies in healthcare delivery [ 12 , 13 ]. As WHO has revealed, there are about 28 million nurses worldwide, and the demand of nurses will put nurse resources into the specific spotlight [ 1 ]. Indeed, evidence could be used to find solutions for how to solve economic deficits or other problems using leadership skills. This is important as, when nurses are able to show leadership and control in their own work, they are less likely to leave their jobs [ 66 ]. On the other hand, based on our discussions with stakeholders, nurse leaders are not used to using evidence in their own work. Further, evidence-based leadership is not possible if nurse leaders do not have access to a relevant, robust body of evidence, adequate funding, resources, and organizational support, and evidence-informed decision making may only offer short-term solutions [ 55 ]. We still believe that implementing evidence-based strategies into the work of nurse leaders may create opportunities to protect this critical workforce from burnout or leaving the field [ 67 ]. However, the role of the evidence-based approach for nurse leaders in solving these problems is still a key question.

Limitations

This study aimed to use a broad search strategy to ensure a comprehensive review but, nevertheless, limitations exist: we may have missed studies not included in the major international databases. To keep search results manageable, we did not use specific databases to systematically search grey literature although it is a rich source of evidence used in systematic reviews and meta-analysis [ 68 ]. We still included published conference abstract/proceedings, which appeared in our scientific databases. It has been stated that conference abstracts and proceedings with empirical study results make up a great part of studies cited in systematic reviews [ 69 ]. At the same time, a limited space reserved for published conference publications can lead to methodological issues reducing the validity of the review results [ 68 ]. We also found that the great number of studies were carried out in western countries, restricting the generalizability of the results outside of English language countries. The study interventions and outcomes were too different across studies to be meaningfully pooled using statistical methods. Thus, our narrative synthesis could hypothetically be biased. To increase transparency of the data and all decisions made, the data, its categorization and conclusions are based on original studies and presented in separate tables and can be found in Additional files. Regarding a methodological approach [ 34 ], we used a mixed methods systematic review, with the core intention of combining quantitative and qualitative data from primary studies. The aim was to create a breadth and depth of understanding that could confirm to or dispute evidence and ultimately answer the review question posed [ 34 , 70 ]. Although the method is gaining traction due to its usefulness and practicality, guidance in combining quantitative and qualitative data in mixed methods systematic reviews is still limited at the theoretical stage [ 40 ]. As an outcome, it could be argued that other methodologies, for example, an integrative review, could have been used in our review to combine diverse methodologies [ 71 ]. We still believe that the results of this mixed method review may have an added value when compared with previous systematic reviews concerning leadership and an evidence-based approach.

Our mixed methods review fills the gap regarding how nurse leaders themselves use evidence to guide their leadership role and what the measured and perceived impact of evidence-based leadership is in nursing. Although the scarcity of controlled studies on this topic is concerning, the available research data suggest that evidence-based leadership intervention can improve nurse performance, organizational outcomes, and patient outcomes. Leadership problems are also well recognized in healthcare settings. More knowledge and a deeper understanding of the role of nurse leaders, and how they can use evidence in their own managerial leadership decisions, is still needed. Despite the limited number of studies, we assume that this narrative synthesis can provide a good foundation for how to develop evidence-based leadership in the future.

Implications

Based on our review results, several implications can be recommended. First, the future of nursing success depends on knowledgeable, capable, and strong leaders. Therefore, nurse leaders worldwide need to be educated about the best ways to manage challenging situations in healthcare contexts using an evidence-based approach in their decisions. This recommendation was also proposed by nurses and nurse leaders during our discussion meeting with stakeholders.

Second, curriculums in educational organizations and on-the-job training for nurse leaders should be updated to support general understanding how to use evidence in leadership decisions. And third, patients and family members should be more involved in the evidence-based approach. It is therefore important that nurse leaders learn how patients’ and family members’ views as stakeholders are better considered as part of the evidence-based leadership approach.

Future studies should be prioritized as follows: establishment of clear parameters for what constitutes and measures evidence-based leadership; use of theories or models in research to inform mechanisms how to effectively change the practice; conducting robust effectiveness studies using trial designs to evaluate the impact of evidence-based leadership; studying the role of patient and family members in improving the quality of clinical care; and investigating the financial impact of the use of evidence-based leadership approach within respective healthcare systems.

Data availability

The authors obtained all data for this review from published manuscripts.

World Health Organization. State of the world’s nursing 2020: investing in education, jobs and leadership. 2020. https://www.who.int/publications/i/item/9789240003279 . Accessed 29 June 2024.

Hersey P, Campbell R. Leadership: a behavioral science approach. The Center for; 2004.

Cline D, Crenshaw JT, Woods S. Nurse leader: a definition for the 21st century. Nurse Lead. 2022;20(4):381–4. https://doi.org/10.1016/j.mnl.2021.12.017 .

Article Google Scholar

Chen SS. Leadership styles and organization structural configurations. J Hum Resource Adult Learn. 2006;2(2):39–46.

Google Scholar

McKibben L. Conflict management: importance and implications. Br J Nurs. 2017;26(2):100–3.

Article PubMed Google Scholar

Haghgoshayie E, Hasanpoor E. Evidence-based nursing management: basing Organizational practices on the best available evidence. Creat Nurs. 2021;27(2):94–7. https://doi.org/10.1891/CRNR-D-19-00080 .

Majers JS, Warshawsky N. Evidence-based decision-making for nurse leaders. Nurse Lead. 2020;18(5):471–5.

Tichy NM, Bennis WG. Making judgment calls. Harvard Business Rev. 2007;85(10):94.

Sousa MJ, Pesqueira AM, Lemos C, Sousa M, Rocha Á. Decision-making based on big data analytics for people management in healthcare organizations. J Med Syst. 2019;43(9):1–10.

Guo R, Berkshire SD, Fulton LV, Hermanson PM. %J L in HS. Use of evidence-based management in healthcare administration decision-making. 2017;30(3): 330–42.

Liang Z, Howard P, Rasa J. Evidence-informed managerial decision-making: what evidence counts?(part one). Asia Pac J Health Manage. 2011;6(1):23–9.

Hasanpoor E, Janati A, Arab-Zozani M, Haghgoshayie E. Using the evidence-based medicine and evidence-based management to minimise overuse and maximise quality in healthcare: a hybrid perspective. BMJ evidence-based Med. 2020;25(1):3–5.

Shingler NA, Gonzalez JZ. Ebm: a pathway to evidence-based nursing management. Nurs 2022. 2017;47(2):43–6.

Farokhzadian J, Nayeri ND, Borhani F, Zare MR. Nurse leaders’ attitudes, self-efficacy and training needs for implementing evidence-based practice: is it time for a change toward safe care? Br J Med Med Res. 2015;7(8):662.

Article PubMed PubMed Central Google Scholar

American Nurses Association. ANA leadership competency model. Silver Spring, MD; 2018.

Royal College of Nursing. Leadership skills. 2022. https://www.rcn.org.uk/professional-development/your-career/nurse/leadership-skills . Accessed 29 June 2024.

Kakemam E, Liang Z, Janati A, Arab-Zozani M, Mohaghegh B, Gholizadeh M. Leadership and management competencies for hospital managers: a systematic review and best-fit framework synthesis. J Healthc Leadersh. 2020;12:59.

Liang Z, Howard PF, Leggat S, Bartram T. Development and validation of health service management competencies. J Health Organ Manag. 2018;32(2):157–75.

World Health Organization. Global Strategic Directions for Nursing and Midwifery. 2021. https://apps.who.int/iris/bitstream/handle/10665/344562/9789240033863-eng.pdf . Accessed 29 June 2024.

NHS Leadership Academy. The nine leadership dimensions. 2022. https://www.leadershipacademy.nhs.uk/resources/healthcare-leadership-model/nine-leadership-dimensions/ . Accessed 29 June 2024.

Canadian Nurses Association. Evidence-informed decision-making and nursing practice: Position statement. 2018. https://hl-prod-ca-oc-download.s3-ca-central-1.amazonaws.com/CNA/2f975e7e-4a40-45ca-863c-5ebf0a138d5e/UploadedImages/documents/Evidence_informed_Decision_making_and_Nursing_Practice_position_statement_Dec_2018.pdf . Accessed 29 June 2024.

Hasanpoor E, Hajebrahimi S, Janati A, Abedini Z, Haghgoshayie E. Barriers, facilitators, process and sources of evidence for evidence-based management among health care managers: a qualitative systematic review. Ethiop J Health Sci. 2018;28(5):665–80.

PubMed PubMed Central Google Scholar

Collins DB, Holton EF III. The effectiveness of managerial leadership development programs: a meta-analysis of studies from 1982 to 2001. Hum Res Dev Q. 2004;15(2):217–48.

Cummings GG, Lee S, Tate K, Penconek T, Micaroni SP, Paananen T, et al. The essentials of nursing leadership: a systematic review of factors and educational interventions influencing nursing leadership. Int J Nurs Stud. 2021;115:103842.

Clavijo-Chamorro MZ, Romero-Zarallo G, Gómez-Luque A, López-Espuela F, Sanz-Martos S, López-Medina IM. Leadership as a facilitator of evidence implementation by nurse managers: a metasynthesis. West J Nurs Res. 2022;44(6):567–81.

Young SK. Evidence-based management: a literature review. J Nurs Adm Manag. 2002;10(3):145–51.

Williams LL. What goes around comes around: evidence-based management. Nurs Adm Q. 2006;30(3):243–51.

Fraser I. Organizational research with impact: working backwards. Worldviews Evidence-Based Nurs. 2004;1:S52–9.

Roshanghalb A, Lettieri E, Aloini D, Cannavacciuolo L, Gitto S, Visintin F. What evidence on evidence-based management in healthcare? Manag Decis. 2018;56(10):2069–84.

Jaana M, Vartak S, Ward MM. Evidence-based health care management: what is the research evidence available for health care managers? Eval Health Prof. 2014;37(3):314–34.

Tate K, Hewko S, McLane P, Baxter P, Perry K, Armijo-Olivo S, et al. Learning to lead: a review and synthesis of literature examining health care managers’ use of knowledge. J Health Serv Res Policy. 2019;24(1):57–70.

Geerts JM, Goodall AH, Agius S, %J SS. Medicine. Evidence-based leadership development for physicians: a systematic literature review. 2020;246: 112709.

Barends E, Rousseau DM, Briner RB. Evidence-based management: The basic principles. Amsterdam; 2014. https://research.vu.nl/ws/portalfiles/portal/42141986/complete+dissertation.pdf#page=203 . Accessed 29 June 2024.

Stern C, Lizarondo L, Carrier J, Godfrey C, Rieger K, Salmond S, et al. Methodological guidance for the conduct of mixed methods systematic reviews. JBI Evid Synthesis. 2020;18(10):2108–18. https://doi.org/10.11124/JBISRIR-D-19-00169 .

Lancet T. 2020: unleashing the full potential of nursing. Lancet (London, England). 2019. p. 1879.

Välimäki MA, Lantta T, Hipp K, Varpula J, Liu G, Tang Y, et al. Measured and perceived impacts of evidence-based leadership in nursing: a mixed-methods systematic review protocol. BMJ Open. 2021;11(10):e055356. https://doi.org/10.1136/bmjopen-2021-055356 .

The Joanna Briggs Institute. Joanna Briggs Institute reviewers’ manual: 2014 edition. Joanna Briggs Inst. 2014; 88–91.

Pearson A, White H, Bath-Hextall F, Salmond S, Apostolo J, Kirkpatrick P. A mixed-methods approach to systematic reviews. JBI Evid Implement. 2015;13(3):121–31.

Johnson RB, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educational Researcher. 2004;33(7):14–26.

Hong, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Reviews. 2017;6(1):61. https://doi.org/10.1186/s13643-017-0454-2 .

Ramis MA, Chang A, Conway A, Lim D, Munday J, Nissen L. Theory-based strategies for teaching evidence-based practice to undergraduate health students: a systematic review. BMC Med Educ. 2019;19(1):1–13.

Sackett DL, Rosenberg WM, Gray JM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. Bmj. British Medical Journal Publishing Group; 1996. pp. 71–2.

Goodman JS, Gary MS, Wood RE. Bibliographic search training for evidence-based management education: a review of relevant literatures. Acad Manage Learn Educ. 2014;13(3):322–53.

Aromataris E, Munn Z. Chapter 3: Systematic reviews of effectiveness. JBI Manual for Evidence Synthesis. 2020; https://synthesismanual.jbi.global .

Munn Z, Barker TH, Moola S, Tufanaru C, Stern C, McArthur A et al. Methodological quality of case series studies: an introduction to the JBI critical appraisal tool. 2020;18(10): 2127–33.

Hong Q, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed methods Appraisal Tool (MMAT) Version 2018: user guide. Montreal: McGill University; 2018.

McKenna J, Jeske D. Ethical leadership and decision authority effects on nurses’ engagement, exhaustion, and turnover intention. J Adv Nurs. 2021;77(1):198–206.

Maxwell M, Hibberd C, Aitchison P, Calveley E, Pratt R, Dougall N, et al. The TIDieR (template for intervention description and replication) checklist. The patient Centred Assessment Method for improving nurse-led biopsychosocial assessment of patients with long-term conditions: a feasibility RCT. NIHR Journals Library; 2018.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77–101.

Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, et al. Stakeholder involvement in systematic reviews: a scoping review. Syst Reviews. 2018;7:1–26.

Braye S, Preston-Shoot M. Emerging from out of the shadows? Service user and carer involvement in systematic reviews. Evid Policy. 2005;1(2):173–93.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Reviews. 2021;10(1):1–11.

Porta M. Pilot investigation, study. A dictionary of epidemiology. Oxford University Press Oxford; 2014. p. 215.

Kreis J, Puhan MA, Schünemann HJ, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expect. 2013;16(4):323–37.

Joseph ML, Nelson-Brantley HV, Caramanica L, Lyman B, Frank B, Hand MW, et al. Building the science to guide nursing administration and leadership decision making. JONA: J Nurs Adm. 2022;52(1):19–26.

Gifford W, Davies BL, Graham ID, Tourangeau A, Woodend AK, Lefebre N. Developing Leadership Capacity for Guideline Use: a pilot cluster Randomized Control Trial: Leadership Pilot Study. Worldviews Evidence-Based Nurs. 2013;10(1):51–65. https://doi.org/10.1111/j.1741-6787.2012.00254.x .

Hsieh HY, Henker R, Ren D, Chien WY, Chang JP, Chen L, et al. Improving effectiveness and satisfaction of an electronic charting system in Taiwan. Clin Nurse Specialist. 2016;30(6):E1–6. https://doi.org/10.1097/NUR.0000000000000250 .

McAllen E, Stephens K, Swanson-Biearman B, Kerr K, Whiteman K. Moving Shift Report to the Bedside: an evidence-based Quality Improvement Project. OJIN: Online J Issues Nurs. 2018;23(2). https://doi.org/10.3912/OJIN.Vol23No02PPT22 .

Thomas M, Autencio K, Cesario K. Positive outcomes of an evidence-based pressure injury prevention program. J Wound Ostomy Cont Nurs. 2020;47:S24.

Cullen L, Titler MG. Promoting evidence-based practice: an internship for Staff nurses. Worldviews Evidence-Based Nurs. 2004;1(4):215–23. https://doi.org/10.1111/j.1524-475X.2004.04027.x .

DuBose BM, Mayo AM. Resistance to change: a concept analysis. Nursing forum. Wiley Online Library; 2020. pp. 631–6.

Lalleman PCB, Smid GAC, Lagerwey MD, Shortridge-Baggett LM, Schuurmans MJ. Curbing the urge to care: a bourdieusian analysis of the effect of the caring disposition on nurse middle managers’ clinical leadership in patient safety practices. Int J Nurs Stud. 2016;63:179–88.

Article CAS PubMed Google Scholar

Martin E, Warshawsky N. Guiding principles for creating value and meaning for the next generation of nurse leaders. JONA: J Nurs Adm. 2017;47(9):418–20.

Griffiths P, Recio-Saucedo A, Dall’Ora C, Briggs J, Maruotti A, Meredith P, et al. The association between nurse staffing and omissions in nursing care: a systematic review. J Adv Nurs. 2018;74(7):1474–87. https://doi.org/10.1111/jan.13564 .

Lúanaigh PÓ, Hughes F. The nurse executive role in quality and high performing health services. J Nurs Adm Manag. 2016;24(1):132–6.

de Kok E, Weggelaar-Jansen AM, Schoonhoven L, Lalleman P. A scoping review of rebel nurse leadership: descriptions, competences and stimulating/hindering factors. J Clin Nurs. 2021;30(17–18):2563–83.

Warshawsky NE. Building nurse manager well-being by reducing healthcare system demands. JONA: J Nurs Adm. 2022;52(4):189–91.

Paez A. Gray literature: an important resource in systematic reviews. J Evidence-Based Med. 2017;10(3):233–40.

McAuley L, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000;356(9237):1228–31.

Sarah S. Introduction to mixed methods systematic reviews. https://jbi-global-wiki.refined.site/space/MANUAL/4689215/8.1+Introduction+to+mixed+methods+systematic+reviews . Accessed 29 June 2024.

Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546–53.

Download references

Acknowledgements

We want to thank the funding bodies, the Finnish National Agency of Education, Asia Programme, the Department of Nursing Science at the University of Turku, and Xiangya School of Nursing at the Central South University. We also would like to thank the nurses and nurse leaders for their valuable opinions on the topic.

The work was supported by the Finnish National Agency of Education, Asia Programme (grant number 26/270/2020) and the University of Turku (internal fund 26003424). The funders had no role in the study design and will not have any role during its execution, analysis, interpretation of the data, decision to publish, or preparation of the manuscript.

Author information

Authors and affiliations.

Department of Nursing Science, University of Turku, Turku, FI-20014, Finland

Maritta Välimäki, Tella Lantta, Kirsi Hipp & Jaakko Varpula

School of Public Health, University of Helsinki, Helsinki, FI-00014, Finland

Maritta Välimäki

Xiangya Nursing, School of Central South University, Changsha, 410013, China

Shuang Hu, Jiarui Chen, Yao Tang, Wenjun Chen & Xianhong Li

School of Health and Social Services, Häme University of Applied Sciences, Hämeenlinna, Finland

Hunan Cancer Hospital, Changsha, 410008, China

Gaoming Liu

You can also search for this author in PubMed Google Scholar

Contributions

Study design: MV, XL. Literature search and study selection: MV, KH, TL, WC, XL. Quality assessment: YT, SH, XL. Data extraction: JC, MV, JV, WC, YT, SH, GL. Analysis and interpretation: MV, SH. Manuscript writing: MV. Critical revisions for important intellectual content: MV, XL. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Xianhong Li .

Ethics declarations

Ethics approval and consent to participate.

No ethical approval was required for this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Differences between the original protocol

We modified criteria for the included studies: we included published conference abstracts/proceedings, which form a relatively broad knowledge base in scientific knowledge. We originally planned to conduct a survey with open-ended questions followed by a face-to-face meeting to discuss the preliminary results of the review. However, to avoid extra burden in nurses due to COVID-19, we decided to limit the validation process to the online discussion only.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, supplementary material 6, supplementary material 7, supplementary material 8, supplementary material 9, supplementary material 10, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Välimäki, M., Hu, S., Lantta, T. et al. The impact of evidence-based nursing leadership in healthcare settings: a mixed methods systematic review. BMC Nurs 23 , 452 (2024). https://doi.org/10.1186/s12912-024-02096-4

Download citation

Received : 28 April 2023

Accepted : 13 June 2024

Published : 03 July 2024

DOI : https://doi.org/10.1186/s12912-024-02096-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Evidence-based leadership
Health services administration
Organizational development
Quality in healthcare

BMC Nursing

ISSN: 1472-6955

General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 3.7.2024 in Vol 26 (2024)

Design of Digital Mental Health Platforms for Family Member Cocompletion: Scoping Review

Authors of this article:

Ellen T Welsh 1 , BSc, MEng ;
Jennifer E McIntosh 1 , PhD ;
An Vuong 1 , BPsy (Hons) ;
Zoe C G Cloud 1 , PhD ;
Eliza Hartley 1 , DPsych ;
James H Boyd 2 , PhD

1 The Bouverie Centre, School of Psychology and Public Health, La Trobe University, Brunswick, Australia

2 School of Psychology and Public Health, La Trobe University, Bundoora, Australia

Corresponding Author:

Ellen T Welsh, BSc, MEng

The Bouverie Centre

School of Psychology and Public Health

La Trobe University

8 Gardiner Street

Brunswick, 3056

Phone: 61 384814800

Email: [email protected]

Background: The COVID-19 pandemic placed an additional mental health burden on individuals and families, resulting in widespread service access problems. Digital mental health interventions suggest promise for improved accessibility. Recent reviews have shown emerging evidence for individual use and early evidence for multiusers. However, attrition rates remain high for digital mental health interventions, and additional complexities exist when engaging multiple family members together.

Objective: As such, this scoping review aims to detail the reported evidence for digital mental health interventions designed for family use with a focus on the build and design characteristics that promote accessibility and engagement and enable cocompletion by families.

Methods: A systematic literature search of MEDLINE, Embase, PsycINFO, Web of Science, and CINAHL databases was conducted for articles published in the English language from January 2002 to March 2024. Eligible records included empirical studies of digital platforms containing some elements designed for cocompletion by related people as well as some components intended to be completed without therapist engagement. Platforms were included in cases in which clinical evidence had been documented.

Results: Of the 9527 papers reviewed, 85 (0.89%) met the eligibility criteria. A total of 24 unique platforms designed for co-use by related parties were identified. Relationships between participants included couples, parent-child dyads, family caregiver–care recipient dyads, and families. Common platform features included the delivery of content via structured interventions with no to minimal tailoring or personalization offered. Some interventions provided live contact with therapists. User engagement indicators and findings varied and included user experience, satisfaction, completion rates, and feasibility. Our findings are more remarkable for what was absent in the literature than what was present. Contrary to expectations, few studies reported any design and build characteristics that enabled coparticipation. No studies reported on platform features for enabling cocompletion or considerations for ensuring individual privacy and safety. None examined platform build or design characteristics as moderators of intervention effect, and none offered a formative evaluation of the platform itself.

Conclusions: In this early era of digital mental health platform design, this novel review demonstrates a striking absence of information about design elements associated with the successful engagement of multiple related users in any aspect of a therapeutic process. There remains a large gap in the literature detailing and evaluating platform design, highlighting a significant opportunity for future cross-disciplinary research. This review details the incentive for undertaking such research; suggests design considerations when building digital mental health platforms for use by families; and offers recommendations for future development, including platform co-design and formative evaluation.

Introduction

Family mental health.

Normatively, mental health disorders impacted >1 billion people worldwide in 2016 [ 1 ]. The COVID-19 pandemic brought further substantial impact on mental health, placing increased demand on mental health services [ 2 ]. Mental health is inherently relational [ 3 , 4 ], and family members and partners are inevitably impacted by an individual’s mental health challenges [ 5 ]. During the COVID-19 pandemic, markers of heightened family stress included rising rates of family violence [ 6 ]; increased parenting stress [ 7 ]; and observed rates of maladaptive parenting practices, including neglectful, harsh, and coercive parenting [ 8 - 10 ].

There is a strong evidence base for family and systemic interventions for child- and adult-focused mental health challenges. Family participation supports members of the family to safely contribute to individual recovery and improved relationships [ 11 - 13 ] and can be more beneficial than individual work [ 14 - 16 ] and family educational interventions [ 17 ]. In addition, parent involvement in interventions for childhood behavioral [ 18 ] and adolescent anxiety disorders [ 19 ] has been shown to be beneficial and contributes to positive long-term outcomes.

Digital Mental Health

The World Health Organization has emphasized the significant potential of digital mental health interventions (DMHIs) in expanding reach and access to services [ 20 ]. Such DMHIs have shown promise in reaching underserved populations [ 21 ], leading to improved management of symptoms in individuals [ 22 ], particularly youth aged <25 years [ 23 , 24 ]. There is growing meta-analytic evidence for positive mental health outcomes of digitally delivered versus in-person individual treatment, for example, in the field of cognitive behavioral interventions [ 25 ]. With rapid developments in technology, research interest is expanding, with most of the literature so far focused on DMHIs for individuals. For example, a review of systematic reviews of digital interventions for mental health and well-being (with no limitations placed on population) conducted in 2021 identified 246 systematic reviews published between 2016 and 2021, all of which reviewed digitally delivered mental health interventions for individuals [ 26 ].

Beyond DMHIs designed for individuals, 2 first-generation reviews of dyadic (caregiver and care recipient) [ 27 ] and couple-targeted DMHIs [ 28 ] suggest that DMHIs can decrease barriers and improve timely access and outcomes for distressed relationships. However, research into DMHIs for families to access together is as yet undeveloped.

Despite growing evidence, and regardless of the population targeted, retention rates for DMHIs remain low, limiting their ultimate impact [ 29 - 32 ]. Among other factors, interface ease of use has been identified as a barrier to DMHI retention and engagement by individuals [ 25 , 33 ]. It is likely that similar (or possibly even greater) barriers for family engagement in the digital mental health space exist. Given the fundamental differences in the approach and focus for family and relational interventions when compared to interventions designed for individuals [ 3 , 34 ], it is likely that there are unique factors to consider when designing DMHIs for use by families. This might include considerations for individual user privacy and ways in which the platform allows multiple people to contribute to and especially cocomplete activities, such as shared goal setting. Thus, it would be ill-founded to extrapolate results from studies on DMHIs designed for use by individuals and assume similar platform interaction values for families. The need for further research specific to the design of DMHIs for family use is clear.

Design of DMHIs for Families

Therefore, the question arises about what an effective DMHI for family use might look like. Given that computers and tablets are designed for use by individuals, DMHIs intended for cocompletion by family members may use different platform and interface features to support and sustain family engagement. No review to date has examined evidence for design and build characteristics that promote cocompletion usability, including improved engagement and accessibility.

In that light, this review aimed to synthesize the available evidence regarding the build and design characteristics that enable cocompletion and discuss reported indicators of user engagement with platforms designed for such use, namely, usability, satisfaction, acceptability, and feasibility. In the digital mental health literature, these user engagement indicators measure the ability of a platform to engage and sustain users. However, there is a notable lack of agreement on both the definition and measurement of the construct of engagement , which can lead to inappropriate selection, presentation, and interpretation of user engagement indicators across studies [ 35 ]. As such, a scoping review was conducted, and we adopted the definition of user engagement as outlined by Perski et al [ 36 ]: “ Engagement with [Digital Behaviour Change Interventions] is (1) the extent (e.g. amount, frequency, duration, depth) of usage and (2) a subjective experience characterised by attention, interest and affect.”

In this scoping review, we differentiate the term “platform” from the term “intervention.” We define “platform” as the tools, infrastructure, and technical foundation behind the delivery of an intervention, including interface characteristics such as the design, layout, and delivery mode. We define “intervention” as the mental health–related content that is delivered via the platform. This review sought to understand (1) the design and functionality characteristics that enable the effective engagement with and cocompletion of a family-oriented DMHI and (2) whether these elements moderate the effect of the intervention on mental health or relational outcomes. To distinguish effective platform contributors to engagement from elements pertaining to intervention content, we selected only those platforms housing interventions of established clinical efficacy (which we defined as any intervention that had at least one study reporting a significant improvement in a mental health or relational outcome). In addition, it is expected that build characteristics may vary by population, and given that there is no uniform family composition, this review scoped platforms designed for cocompletion by any family relationship type, including couples, family subsystems, and whole families.

Search Strategy

To identify studies reviewing platforms delivering clinical interventions designed for cocompletion by families, a systematic search was conducted following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 37 ]. A comprehensive electronic literature search for articles published in English was conducted in the following databases: MEDLINE, Embase, and PsycINFO via the Ovid platform; CINAHL via the EBSCOhost platform, and Web of Science. In line with developments in digital technology, studies were included if they were published in or since 2002. The search was first conducted on June 24, 2022, and additional searches were conducted on November 24, 2022; April 21, 2023; and March 15, 2024.

Eligibility Criteria

As advised by the Joanna Briggs Institute’s guidelines for conducting scoping reviews [ 38 ], the population, concept, and context framework was used to define eligibility. Textbox 1 shows the inclusion and exclusion criteria in line with the population, concept, and context framework and contains additional study elements relevant to the eligibility criteria.

Studies were not excluded when platforms contained additional components involving practitioner (sometimes referred to in the studies as a coach, professional, therapist, or staff member) engagement. Further to the inclusion and exclusion criteria outlined in Textbox 1 , platforms offering interventions that had no evidence of clinical efficacy (ie, no identified studies that reported any significant improvements in mental health or relational outcomes) were excluded. Provided that at least 1 identified study established clinical efficacy for that platform, all studies on that intervention were then included regardless of whether they reported on clinical outcomes. Platforms that met all the other inclusion criteria but without established clinical efficacy are presented in Multimedia Appendix 1 .

Inclusion criteria

Population: Digital mental health interventions (DMHIs) designed for completion by at least 2 related people together
Concept: Platform design elements of DMHIs (via a web or smartphone interface) containing some component that was intended to be completed without therapist or human intervention (ie, was self-directed by participants)
Context: Open and included all care settings (eg, primary care and community) and all jurisdictions and geographic locations
Study type and design: Empirical studies
Publication date: from January 1, 2002, to March 15, 2024
Publication language: English

Exclusion criteria

Population: DMHIs designed for completion by individuals or designed for use by related people but with no activities completed together (ie, completed separately) and DMHIs where children were the focus and the parent’s role was only in assisting their child to participate
Concept: DMHIs in which the target condition was physical illness, physical activity, and weight management and programs delivered through virtual reality devices, wearable devices, DVD, or other non–web-based approaches
Study type and design: Nonempirical studies and gray literature (ie, non–peer-reviewed or unpublished manuscripts)

Search and Data Extraction Methodology

A total of 3 key search constructs addressed the different elements of the research question: digital intervention, mental or relational health, and population. Results were combined using Boolean operators. The search strategies for each database can be found in Multimedia Appendix 2 . The reference lists of relevant reviews were also screened for potentially relevant studies. Data extraction was completed by 2 researchers trained in systematic search methodology using a standardized template, and discrepancies were resolved through discussion between the 2 researchers. In cases in which it appeared that there could be cocompletion but it was not directly specified, the study authors were contacted, and websites were searched.

Screening and Selection Process

Search results were downloaded into EndNote (Clarivate Analytics) [ 39 ] and imported into Covidence (Veritas Health Innovation) [ 40 ]. Duplicates were first removed in EndNote and again following import into Covidence. In total, 2 researchers screened the identified studies at the title and abstract level, with 20% being double screened. Disagreements were resolved through discussion. A total of 2 researchers screened the articles at the full-text level with 20% double screening to determine eligibility against the inclusion criteria outlined previously. Reasons for exclusion at the full-text level were recorded.

Data Synthesis

Data were synthesized using a narrative approach. Due to high variability in the reporting of outcomes and measurements across studies, a systematic or meta-analytic approach was not possible.

The included articles were grouped by the digital platform used. Information regarding the authors, the year of publication, the country where the study took place, the population, and associated user engagement indicators was extracted. Significant differences in mental health or relational outcomes following the DMHI were indicated. Details about the platforms were extracted into a separate table. Also detailed were the intervention target; the relationship between the participants; components designed to be completed in a self-paced manner, together, individually, or with a professional; tailored components; and any additional key features. Results were categorized and synthesized based on the targeted relationship for the intervention (eg, couples or families).

The combined searches yielded 17,765 results. Following removal of 46.37% (8238/17,765) of duplicates in EndNote and Covidence, 9527 papers were screened at the title and abstract level, resulting in 9184 (96.4%) exclusions. A total of 343 full-text articles were reviewed for inclusion, with 263 (76.7%) exclusions. Reasons for exclusion included the platform being designed for use by individuals (154/263, 58.6%), nonempirical studies (55/263, 20.9%), the platform not containing any self-guided components (36/263, 13.7%), or wrong indication (eg, weight loss intervention; 18/263, 6.8%). A total of 80 studies were included for data extraction. An additional 5 studies were identified through reference scanning and included in data extraction, resulting in a total of 85 studies included in this review. Figure 1 shows the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) diagram [ 37 ].

The following sections first summarize the studies identified and then report on characteristics of and findings related to the included platforms.

Included Studies

Table 1 details the characteristics of the 85 studies, including study type, their population and sample size, usability measures and findings, and an indication of clinical efficacy based on significant improvement in mental health or relational outcomes following completion of the intervention. Among the 85 included studies, data were collected during randomized controlled trials (n=63, 74%), pilot feasibility studies (n=14, 16%), single-arm studies (n=7, 8%), and nonrandomized quasi-experimental studies (n=1, 1%).

A total of 74% (63/85) of the studies were conducted in the United States; 12% (10/85) were conducted in Canada; 5% (4/85) were conducted in Australia; 2% (2/85) were conducted in the United Kingdom; 2% (2/85) were conducted in China; and 1% (1/85) were conducted each in the Netherlands, Sweden, Japan, and Korea. In total, 52% (44/85) of the included studies were published between January 2019 and March 2024, whereas 5% (4/85) of the studies were published in the first 5 years of the search period (2002-2006 inclusive) and the remainder (37/85, 43%) were published in between these periods.

Platform, study, and country	Study characteristics		Measures or outcomes
	Design; comparator	Population; sample size; attrition	Relational and individual	User engagement indicators	Reported findings
4Cs:CRC [ ]; China	Pilot feasibility trial	Heterosexual couples where one member was experiencing colorectal cancer; 24 couples; 16.7%	Dyadic coping; cancer-related communication; self-efficacy; physical and mental health; positive and negative emotions	(1) Postintervention evaluation; (2) feasibility and acceptability	(1) Highly rated usefulness, ease of use, and satisfaction; all mean acceptability ratings >5.2/7; (2) 83.8% retention; 609 session views; mean 29 views per page; mean 3-7 page views per session per dyad
4Cs:CRC [ ]; China	RCT ; web-based, face-to-face, blended, or control	Heterosexual couples where one member had colorectal cancer; 212 couples; 16%	Dyadic coping ; cancer-related communication ; marital satisfaction; self-efficacy ; physical and mental health; positive and negative emotions	Not reported	Not reported
CA-CIFFTA [ ]; United Kingdom	RCT; no treatment	Hispanic (80%) and Black (20%) adolescents and their families; 80 parent-child dyads; 27% (intervention)	Family cohesion ; family conflict; parenting practices; adolescent behavioral problems	Not reported	Not reported
C-MBI for YBCSs [ ]; United States	RCT; MBI completed by YBCSs only (I-MBI )	Female breast cancer survivors and their male partners; 117 couples; 26% (I-MBI) and 38% (C-MBI)	Couple functioning (I-MBI only); individual-level functioning	(1) Feasibility and acceptability of YBCSs (self-report); (2) feasibility and acceptability of partners (self-report)	(1) 39% requested more contact with peers; 63% would recommend it; 77% watched all videos; 90% used the supplemental material; 91% completed some or all of the assignments; rated most useful: mindfulness sessions (80%), yoga (14%), and partner interaction (7%); time constraints were the most cited reason for not recommending the intervention; (2) 93% had no desire to interact with peers; 69% would recommend it; 69% watched all videos; 89% used the supplemental materials; 92% completed some or all of the assignments; time constraints were the most cited reason for not recommending the intervention
Cool Kids Online [ ]; Australia	RCT; waitlist	Children (aged 7-12 years) with anxiety and their parents or caregivers; 95 dyads; 12% at posttreatment time point and 27% at 6-month follow-up	Anxiety diagnosis ; anxiety scale; life interference (parent only); mood and feelings ; strengths and difficulties	(1) Satisfaction; (2) completion	(1) 73% of parents were satisfied or very satisfied, 92% reported it as helpful, and 97% were moderately or very confident recommending the intervention; 64% of children were “happy” with the intervention, 89% reported it as helpful or very helpful, and 70% were moderately or very confident that it would help a friend; (2) 83% accessed all lessons (mean 7.52, SD 1.23; range 3-8); received a mean of 8.8/10 (SD 1.61; range 3-10) calls
Couple HOPES [ ]; Canada	Pilot feasibility trial; pretest-posttest	Couples where one member was a military member, veteran, or first responder with PTSD symptoms; 10 couples; 30%	Relationship satisfaction (partners only); conflict; PTSD symptoms ; partner’s accommodations to PTSD symptoms ; anxiety, distress, and QoL ; AOD use	Satisfaction (CSQ )	PTSD: mean 3.4/4 (SD 0.7); partner: mean 3.7/4 (SD 0.4)
Couple HOPES [ ]; Canada	Single arm	Couples where one member was a military member, veteran, or first responder with PTSD symptoms; 17 couples; 35%	Relationship satisfaction (ineffective arguing); PTSD self-report; partner report of PTSD symptoms; mental health; well-being (perceived health); partner accommodations	Satisfaction (CSQ)	PTSD: mean 3.5/4 (SD 0.6); partner: mean 3.7/4 (SD 0.3)
Couple HOPES [ ]; Canada	Single arm	Couples where one member was a military member, veteran, or first responder with PTSD symptoms; 27 couples; 33%	Relationship functioning (ineffective arguing); mental health; well-being (perceived health and QoL)	Satisfaction (CSQ)	Partner: mean 3.7/4 (SD 0.4)
Couple HOPES [ ]; United States	Pilot feasibility trial; pretest-posttest	Couples where one member was a military veteran with PTSD; 15 couples; 27%	Relationship satisfaction ; relationship quality (negative relationship quality); PTSD symptoms ; depression; QoL; significant other’s response to trauma	(1) Completion; (2) feedback	(1) Mean duration 7.20 (SD 5.56) weeks; n=11 completed; 4 noncompleters ( n=2 completed 4/7 modules, n=1 completed 2/7, and n=1 completed 1/7); n=3 “treatment responders” completed it faster; (2) coach was helpful for processing information, thoughts, and feelings; feedback videos were unrealistic or “cheesy,” others found them helpful for digesting and relating to the material
Couplelinks [ ]; Canada	Pilot feasibility trial	Heterosexual couples where a member had a breast cancer diagnosis; 16 couples; 38%	Not reported	(1) Treatment satisfaction (TSQ ); (2) usability	(1) Mean 4/5 (SD 0.56); (2) mean 4/5 (SD 0.83)
Couplelinks [ ]; Canada	Pilot feasibility trial	Heterosexual couples where a member had a breast cancer diagnosis; 6 couples; not reported	Not reported	Engagement promotion by therapist	Rational model of engagement promotion: friendly and positive yet firm approach, humanizing technology, and inclusive and empathic attitude; empirical model of engagement promotion: fostering couple-facilitator bond, fostering intervention adherence, and fostering within-couple bond
Couplelinks [ ]; Canada	Pilot feasibility trial	Heterosexual couples where a member had a breast cancer diagnosis; 12 couples; not reported	Not reported	Types of engagement	Couple “types”—keen: completed with minimal engagement; compliant: met facilitator deadlines; apologetic: enjoyed it and were committed but had trouble staying on track; straggling: least engaged
Couplelinks [ ]; Canada	Pilot feasibility trial	Heterosexual couples where a member had a breast cancer diagnosis; 13 couples; not reported	Not reported	Perceived benefits and limitations	58% agreed or strongly agreed that it was beneficial; 35% said that it was somewhat beneficial
Couplelinks [ ]; Canada	RCT; waitlist	Heterosexual couples where a member had a breast cancer diagnosis; 67 couples; 20.5% in the intervention group and 0% in the control group	Dyadic coping ; dyadic consensus, cohesion, and satisfaction; marital satisfaction; collective coping ; anxiety and depression	Not reported	Not reported
Couplelinks [ ]; Canada	RCT; waitlist	Heterosexual couples where a member had a breast cancer diagnosis; 57 participants; not reported	Not reported	Treatment satisfaction (TSQ)	Mean 4.3/5 (SD 0.54); female participants’ satisfaction ratings were significantly higher ( =.01); medium effect size (0.57)
eMB [ ]; United States	RCT; control	Couples where one member was pregnant; 30 couples; 0%	Anxiety (pregnant person’s anxiety) and depression symptoms	(1) Satisfaction (CSQ-8 ); (2) completion rates and adherence	(1) Excellent satisfaction: mean 3.42 (SD 0.55); pregnant: mean 3.42 (SD 0.59); and partner: mean 3.43 (SD 0.49); factors perceived to promote engagement included flexibility (independent and joint options and own pace) and focus on the self before talking to their partner; helpful elements included videos, web-based exercises, and activities; factors impacting engagement included video relatability, poor quality, outdated images, simplistic and low-technology visualizations, videos perceived as old or silly, extreme vignettes and illustrations, and videos being overly dramatized and unrelatable; (2) 50% used it alone, 9% used it together with their partner, and 27% were a combination of both; 14% did not engage; 0% completed 1 lesson per week as advised; 83 discrete log-ins; pregnant people visited more (mean 4.17 vs mean [partners] 3.44 visits to the intervention)
Embers the Dragon [ ]; United Kingdom	Pilot feasibility trial; no treatment	Children aged 2-7 years and a parent; 129 families; 7.7% in the intervention group and 20.4% in the control group	Parental responses to childhood behaviors	Not reported	Not reported
ePREP (studies on ePREP and OurRelationship reported separately) [ ]; United States	RCT; IRC	Heterosexual couples in long-term relationships; 77 couples; 0%	Commitment attitudes ; communication ; relationship satisfaction ; psychological aggression and assault ; depression, dysphoria, and well-being ; anxiety	Level of engagement as a moderator of clinical outcomes	Higher engagement (measured via results on quizzes): greater intervention effect for alternative monitoring (β=–.33; =.04), constructive communication (β=.29; =.07), self-reported physical assault (β=–.58; =.11), male relationship satisfaction (β=.48; =.02), and female depression (β=–.37; =.10). Greater time spent completing homework assignments: greater intervention effect for reported couple physical assault (β=–.69; =.06), severe psychological aggression for male (β=–.90; =.02) and female (β=–.09; =.01) individuals, and male-perpetrated physical assault (partner report; β=–1.10; =.02) but an attenuation of the positive effect of ePREP on self-reported minor psychological aggression (male individuals: β=.40 and =.11; female individuals: β=.43 and =.12). Male individuals with higher engagement experienced attenuation of positive impact on anxiety (β=.35; =.01), and female individuals who completed more homework assignments experienced attenuation of positive impact on depression symptoms (β=.45; =.03).
ePREP [ ]; United States	RCT; IRC	Married couples; 52 couples; 4% after the intervention and 92% at the 1-year follow-up (8% in the intervention group and 7.6% in the control group)	Conflict resolution methods ; psychological aggression and assault	Not reported	Not reported
FOCUS [ ]; United States	Single arm; repeated measures	Patient-caregiver dyads; 38 dyads; 14%	Communication; social support; emotional distress ; QoL ; appraisal ; coping resources; self-efficacy	(1) Satisfaction; (2) comfort and skill using computers and the internet; (3) feasibility	(1) Ease of use: mean 6.0/7 (SD 1.1); usefulness: mean 4.4/7 (SD 1.4); general satisfaction: mean 4.8 (SD 1.7); no adverse effects of completing the intervention together; (2) moderate skill level; (3) lower enrollment rate than previous in-person RCTs (51% compared with 68%-80%); retention rate was higher than in-person RCTs (86% compared with 62%-83%)
iCBT [ ]; Sweden	RCT; waitlist	Families where the child (aged 8-12 years) had a mental health diagnosis; 93 families (93 children and 182 parents); 2% in the intervention group and 4% in the control group	Anxiety (parent reported); development and well-being; child depression; primary carer mental health	(1) Satisfaction; (2) compliance	(1) Child satisfaction: mean 3.67; parent satisfaction: mean 3.78; 86% of parents agreed or very much agreed that they would recommend it; 82% of children agreed or very much agreed that the treatment was effective; (2) completed modules: mean 9.7 (SD 1.8; range 4-11); 83% completed the first 9 modules; 4 families did not complete the modules intended for both children and parents
Military Family Foundations [ ]; United States	RCT; no treatment	Heterosexual couples expecting their first child where one member was in the military; 56 couples; 34.5% for mothers and 48.3% for fathers in the intervention group and 7.4% for mothers and 22.2% for fathers in the control group	Interparental relationship (mothers only); parental adjustment ; parent report of child outcomes (sadness)	Completion	Mean 3.93/8 completed modules
MindGuide Couple [ ]; South Korea	Single arm	Korean heterosexual couples; 17 couples; 11%	Couple relationship satisfaction; family relationship ; mental health; positive and negative emotions; satisfaction with life	(1) Satisfaction and acceptability; (2) recruitment, retention, and completion	(1) 100% reported that the content and tasks were helpful; 90% reported that the content was applicable to everyday activities; coaching was most helpful (90%), followed by video lectures (43%) and practical tasks (43%); reported benefits included flexible access (90%), being less burdensome than face-to-face interventions (86.3%), and no geographic limitations (76.7%); reported drawbacks included being too long (33.3%) and time burden (76.7%); 93.4% were satisfied; 100% were satisfied with the level of coaching; (2) 94.1% completed
MR [ ]; United States	RCT; MR plus PREP , PREP alone or waitlist	Veteran-partner dyads; 320 individuals (160 couples); 1.2% for MR, 2.5% for MR plus PREP, 1.2% for PREP alone, and 0% for waitlist	Perceived social support; dyadic adjustment; stress ; depression ; PTSD symptoms ; self-compassion ; response to stressful experiences ; sleep quality; physical pain	(1) Intervention use; (2) satisfaction	(1) Mean 2.5 hours of use per week; at 16-week follow-up: mean 90 minutes per week; (2) likely to recommend: mean (veterans) 8.7/10 and mean (partners) 9.1/10
Mother-daughter program [ ]; United States	RCT; waitlist	Girls aged 10-13 years and their mothers; 202 dyads; 0% between pre- and posttest, and 2% in the intervention group and 1% in the control group lost between postintervention time point and follow-up	Mother communication ; conflict management ; daughter communication ; perceived rules ; parental monitoring; normative beliefs ; self-efficacy ; alcohol use ; drinking intention ; refusal skills; parental rules ; parental monitoring	Anonymous program rating	Improved mother-daughter relationship: mean (girls) 4.14/5 (SD 0.35) and mean (mothers) 4.25/5 (SD 0.29); learned useful information: mean (girls) 4.16/5 (SD 0.38) and mean (mothers) 4.13/5 (SD 0.34); enjoyed the intervention: mean (girls) 4.07/5 (SD 0.39); mean (mothers) 4.20/5 (SD 0.34); found time to complete it together: mean (girls) 3.04/5 (SD 0.37); mean (mothers) 3.24/5 (SD 0.33)
Mother-daughter program [ ]; United States	RCT; no treatment	Girls aged 11-13 years and their mothers; 591 dyads; 3.2% in the intervention group	Mother-daughter communication ; substance use ; family rules ; parental monitoring ; normative beliefs ; depression; problem-solving skills; body esteem; drug refusal self-efficacy ; intentions	Not reported	Not reported
Mother-daughter program [ ]; United States	RCT; no treatment	Girls aged 11-13 years and their mothers; 916 dyads; 5.7% from baseline to 1-year follow-up and 4.2% between 1- and 2-year follow-up	Communication ; mother-daughter closeness ; family rules ; parental monitoring ; body esteem; depression; coping ability ; normative beliefs ; refusal self-efficacy ; substance use ; intentions ; family rituals	Not reported	Not reported
Mother-daughter program [ ]; United States	RCT; no treatment	Asian American girls aged 11-14 years and their mothers; 108 dyads; 3.5% in the intervention group and 3.8% in the control group	Mother-daughter closeness ; mother-daughter communication ; substance use ; intentions; depression ; self-efficacy ; refusal skills ; parental monitoring ; family rules	Not reported	Not reported
Mother-daughter program [ ]; United States	RCT; no treatment	Asian American girls aged 11-14 years and their mothers; 108 dyads; 89.2% completed the 2-year measure	Mother-daughter closeness (girls only); mother-daughter communication ; parental monitoring (girls only); family rules (girls only); depressive symptoms; body esteem; self-efficacy ; refusal skills ; normative beliefs; substance use ; intentions	Completion	96.4% completed the entire intervention; 94.6% completed the booster session; participants completed initial 9 sessions (mean 175, SD 68.9 days)
Mother-daughter program [ ]; United States	RCT; no treatment	Black and Hispanic girls aged 10-13 years and their mothers; 564 dyads; 6.6% in the intervention group and 3.3% in the control group	Mother-daughter closeness; mother-daughter communication ; substance use ; normative beliefs ; intentions ; depression ; self-efficacy ; refusal skills; parental monitoring ; family rules ; body esteem	Not reported	Not reported
Mother-daughter program [ ]; United States	RCT; no treatment	Mother-daughter dyads in public housing; 36 dyads; 3%	Mother-daughter closeness ; mother-daughter communication ; parental monitoring ; substance use; fruit and vegetable intake ; physical activity ; perceived stress ; drug refusal skills	Fidelity	97% completed all 3 sessions
OFPS [ ]; United States	Pilot feasibility trial; pretest-posttest	Children (aged 5-16 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child); 19 participants in 6 families; 0%	Child-parent relationship ; sibling relationship ; therapeutic alliance	(1) Feasibility; (2) ease of use; (3) helpfulness and satisfaction (WEQ )	(1) All web sessions completed without therapist assistance; families completed a mean of 10.3 web sessions; (2) ease of use: mean 3.59/5; (3) website helpfulness: mean 4.12/5; videoconferencing helpfulness: mean 4.35/5; 94.7% would recommend the intervention to others
OFPS [ ]; United States	Pilot feasibility trial; pretest-posttest	Children (aged 5-16 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child); 19 participants in 6 families; 0%	Injury-related family stress and burden ; therapeutic alliance ; parental distress, depression, and anxiety ; child adjustment	Not reported	Not reported
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 5-16 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child); 46 families; 12% in the intervention group and 0% in the IRC group	Family problem-solving, communication, and behavior management; parental problem-solving; parental distress, depression, and anxiety	Website use and caregiver satisfaction (WEQ)	100% of parents indicated that they would recommend it to others; 33% indicated that they would prefer to meet in person; 94.4% reported that the website was moderately to extremely easy to use
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 5-16 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child); 46 families; 12% in the intervention group and 0% in the IRC group	Child adjustment (self-control and compliance only)	(1) Child’s self-reported website use; (2) satisfaction (WEQ)	(1) Strong negative correlations between number of sessions completed and child behavioral problems (–0.59) and parental distress (–0.60) at baseline, suggesting families with more problems at baseline completed fewer sessions; (2) 88% rated the website as at least moderately easy to use; 26% rated it as hardly or not easy to use relative to other sites; all children rated the website content as at least moderately helpful; 94% reported feeling at least moderate support and understanding when using the website; 31% reported feeling angry when using the website; 25% reported feeling moderately to extremely worried when using the website
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 5-16 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child); 46 families; 12% in the intervention group and 0% in the IRC group	Therapeutic alliance (no moderation by previous technology use); parental depression (moderated by previous technology use) and anxiety	(1) Parents’ self-reported website use; (2) satisfaction (WEQ); (3) previous computer use; (4) computer equipment comfort rating	(1) Both groups reported spending equivalent amounts of time on the website; (2) satisfaction did not differ by previous technology use; (3) significant effect of technology at home for improvements in depression (t =2.24; =.04); trend in the same direction for anxiety; non–technology users more likely to miss sessions (mean 16.33 missed sessions, SD 11.29; t =2.43; =.03); (4) technology users became more comfortable with the technology over time
OFPS [ ]; United States	Pilot feasibility study; pretest-posttest	Teenagers (aged 11-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 9 families; 0%	Family functioning ; adolescent adjustment; parental distress and depression	Feasibility	All families completed the 10 core sessions; 6 families completed one or more supplemental sessions
OFPS [ ]; United States	Pilot feasibility study; pretest-posttest	Teenagers (aged 11-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 9 families; 0%	Not reported	(1) Self-reported website use; (2) satisfaction (WEQ and OSS )	(1) In addition to parents and teenagers, 9 siblings participated in at least some of the sessions; (2) father satisfaction was generally high; 4/9 teenagers and 2/7 mothers reported a preference for face-to-face meetings; feedback provided support for acceptability and helpfulness of the intervention
OFPS [ ]; United States	RCT; usual care plus IRC	Teenagers (aged 11-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 41 families; 20% in the intervention group and 5% in the IRC group	Executive functioning (teenagers with severe TBI)	Not reported	Not reported
OFPS [ ]; United States	RCT; usual care plus IRC	Teenagers (aged 11-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 41 families; 20% in the intervention group and 5% in the IRC group	Family conflict ; adolescent adjustment	Self-reported website use and satisfaction	Families completed an average of 10 sessions; 95% completed all 10 sessions; 87% of parents reported meeting their goals, learning ways to improve their child’s behavior, and understanding their child better ( <.05 relative to IRC)
OFPS [ ]; United States	RCT; usual care plus IRC	Teenagers (aged 11-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 41 families; 20% in the intervention group and 5% in the IRC group	Parental distress and depression (lower SES only); social problem-solving (lower SES only)	Website use, ease of use, and satisfaction (WEQ and OSS)	93% rated it as moderately or extremely helpful compared to other sites; parents’ suggestions for change included fewer questionnaires; 20% of parents agreed that the intervention was too short
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 12-17 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 132 children and their families; 12.3% in the intervention group and 5.9% in the control group	Teenager executive function (older adolescents)	Not reported	Not reported
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 12-17 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 132 children and their families; 12.3% in the intervention group and 5.9% in the control group	Caregiver depression and distress (intention-to-treat analysis); caregiver efficacy	(1) Previous technology use; (2) completion	(1) Previous computer use did not moderate reductions in depression and distress; nonfrequent computer users in the intervention group reported significantly higher levels of caregiver efficacy ( =7.15; =.01); (2) 43% of parents reported spending <30 minutes per week on CAPS ; 50% reported spending 30 minutes-2 hours per week; 88% completed ≥4 sessions
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 12-17 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 132 children and their families; 12.3% in the intervention group and 5.9% in the control group	Child behavioral outcomes (older adolescents)	Completion	43% of parents reported spending <30 minutes per week on CAPS; 50% reported spending 30 minutes-2 hours per week; 88% completed ≥4 sessions; 93% rated the website as moderately to extremely helpful
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 12-17 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 132 children and their families; 12.3% in the intervention group and 5.9% in the control group	Parent-teenager conflict; parent-teenager interactions; structural, organizational, and transactional characteristics of families	Completion	43% of parents reported spending <30 minutes per week on CAPS; 50% reported spending 30 minutes-2 hours per week; 88% completed ≥4 sessions
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 12-17 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 132 children and their families; 12.3% in the intervention group and 5.9% in the control group (final assessment: 13.4% in the intervention group and 11.4% in the control group)	Long-term caregiver depression and distress (distress only); long-term perceived parenting efficacy	Not reported	Not reported
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 12-17 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 132 children and their families; 12.3% in the intervention group and 5.9% in the control group (final assessment: 30.8% in the intervention group and 19.4% in the control group)	Long-term child behavioral outcomes (internalizing behaviors of older adolescents)	Completion	Number of sessions completed unrelated to improvements in internalizing symptoms over time; those who completed more sessions reported less improvement in externalizing symptoms over time ( =.007)
OFPS [ ]; United States	RCT; usual care plus IRC	Children (aged 12-17 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 132 children and their families; 25% in the intervention group and 21% in the control group	Adolescent emotional and behavioral functioning; adolescent mood and behavior (as a function of parent marital status)	Not reported	Not reported
OFPS [ ]; United States	RCT; face-to-face F-PST , therapist-guided F-PST, or self-guided web-based F-PST	Adolescents (aged 14-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 149 parents and caregivers; 18%	Parent depression (therapist-guided group only); parent psychological distress (therapist-guided group only)	Computer use before and during	Parents with less comfort with technology improved more with therapist-guided treatment when compared to self-guided treatment ( =3.80; =.05)
OFPS [ ]; United States	RCT; face-to-face F-PST, therapist-guided F-PST, or self-guided web-based F-PST	Adolescents (aged 14-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 149 parents and caregivers; at the 9-month assessment: 35.3% in the face-to-face group, 21.5% in the therapist-guided group, and 20% in the self-guided group	Behavioral outcomes	(1) Patient-perceived preference for treatment (before the intervention); (2) adherence; (3) satisfaction; (4) computer use	(1) 71% of parents agreed or strongly agreed that self-guided F-PST was most convenient; 54% of parents agreed or strongly agreed that self-guided and therapist-guided web-based F-PST would be most beneficial; 55% of teenagers agreed or strongly agreed that self-guided F-PST was most convenient; (2) median 5 hours per week; parents assigned to their preferred group completed a mean of 5.29 sessions, and those assigned to their nonpreferred group completed a mean of 6.37 sessions; adolescents in their preferred group completed a mean of 6.12 sessions, and those in their nonpreferred group completed a mean of 5.17 sessions; adolescent treatment preference was significantly related to attrition (χ =4.2, 95% CI 1.03–5.44; =.04); (3) parents in the face-to-face group rated the intervention more favorably than those in the therapist-guided (Cohen =0.67, 95% CI 0.10-1.15; =–2.49; <.04) or self-guided (Cohen =1.18, 95% CI 0.56-1.62; =–4.36; <.001) group; parents in the face-to-face group reported higher satisfaction than parents in the self-guided group (Cohen =0.63, 95% CI 0.09-1.11; =–2.51; =.04); (4) no significant association with treatment preference
OFPS [ ]; United States	RCT; face-to-face F-PST, therapist-guided F-PST, or self-guided web-based F-PST	Adolescents (aged 14-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 149 parents and caregivers; at the 9-month assessment: 35.3% in the face-to-face group, 21.5% in the therapist-guided group, and 20% in the self-guided group	Adolescent QoL ; brain injury symptoms	Not reported	Not reported
OFPS [ ]; United States	RCT; TOPS with family, TOPS with teenagers only, or IRC	Teenagers (aged 11-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 152 teenagers and their families; 31% in the TOPS with family group, 24% in the TOPS with teenagers only group, and 23% in the IRC group	Child behavioral outcomes (TOPS with family)	Completion	Completion: mean sessions completed (TOPS with family) 8.00 (SD 2.90) and mean sessions completed (TOPS with teenagers only) 8.40 (SD 2.80); completed supplemental sessions: 14.29% for TOPS with family and 13.46% for TOPS with teenagers only
OFPS [ ]; United States	RCT; TOPS with family, TOPS with teenagers only, or IRC	Teenagers (aged 11-18 years) with moderate to severe TBI and families (all family members could participate; outcomes reported for one parent and child only); 152 teenagers and their families; 31% in the TOPS with family group, 24% in the TOPS with teenagers only group, and 23% in the IRC group	Family functioning; family cohesion (TOPS with family and 2-parent households); parent-adolescent conflict; parental psychological distress and depression (TOPS with family and 2-parent households)	Not reported	Not reported
OurRelationship [ ]; United States	RCT; waitlist	Heterosexual couples; 300 couples; 8%	Relationship satisfaction ; positive and negative relationship quality (reducing negative relationship quality); relationship confidence ; depression ; anxiety ; perceived health ; work functioning ; QoL	(1) Evaluation (Client Evaluation of Services Questionnaire); (2) completion rates; (3) coach engagement	(1) Mean 26.81 (SD 4.44), nearly equivalent to in-person individual therapy (Cohen =–0.07) and high-quality couple therapy (Cohen =–0.18); 94% were mostly or very satisfied with the services received; 97% would recommend it to a friend; (2) 86% completed the entire intervention; an additional 5% completed up to the “Understand” phase; (3) coaches spent a mean of 51.32 (SD 17.11) minutes with the couples; individuals received a mean of 5.11 (SD 1.7) scripted chat reminders and no tailored chat messages
OurRelationship [ ]; United States	RCT; waitlist	Heterosexual couples; 300 couples; 8%	Relationship satisfaction (no moderation by LI-IPV )	Not reported	Not reported
OurRelationship [ ]; United States	RCT; waitlist	Heterosexual couples; 300 couples; 8%	Relationship satisfaction; relationship confidence; positive and negative relationship quality (moderated by rurality); depression; anxiety; perceived health (moderated by race); work functioning; QoL	(1) Evaluation (Client Evaluation of Services Questionnaire); (2) participant predictors of completion	(1) Couples were generally satisfied with the intervention (mean 26.81, SD 4.44); service evaluation was not moderated by race, ethnicity, income, educational level, or rural status; (2) Hispanic couples (OR 0.24; =.009; Cohen =0.79) and low-income couples (OR 0.21; =.002; Cohen =0.85) were more likely to drop out
OurRelationship [ ]; United States	RCT; waitlist	Heterosexual couples; 300 couples; 8%	Long term: relationship satisfaction; positive and negative relationship quality; relationship confidence (Hispanic couples); depression ; anxiety ; perceived health ; work functioning ; QoL	Not reported	Not reported
OurRelationship [ ]; United States	RCT; waitlist	Heterosexual couples; 300 couples; 8%	Relationship satisfaction; coparenting conflict (not maintained at follow-up); child functioning	Not reported	Not reported
OurRelationship [ ]; United States	RCT; waitlist	Heterosexual couples; 300 couples; 8%	Relationship satisfaction ; communication ; emotional intimacy ; relationship problem confidence ; relationship problem acceptance ; self-protective orientation	Not reported	Not reported
OurRelationship [ ]; United States	RCT; waitlist	Heterosexual couples; 300 couples; 8%	Relationship satisfaction (moderated by neuroticism); relationship confidence ; depression (moderated by neuroticism and conscientiousness); personality	Not reported	Not reported
OurRelationship [ ]; United States	RCT; low coach support or high coach support	Heterosexual couples; 356 couples; 34% in the group with high coach support and 64% in the group with low coach support	Relationship satisfaction (both groups); depression (both groups); anxiety (both groups; significantly greater in the high-support group)	Platform predictors of completion	Participants in the high-support group were significantly more likely to complete the entire intervention (66% vs 36%; χ =32.8, <.001); participants in the high-support group were more likely to complete two-thirds of the intervention (69% vs 45%; =20.4, <.001); no significant differences in first phase completion; completion did not differ by race, ethnicity, or household income
OurRelationship [ ]; United States	RCT; low coach support, high coach support, or no coach support	Heterosexual couples; 529 couples; 93.9% in the group with no coach support, 34% in the group with high support, and 64% in the group with low support	Relationship satisfaction; relationship confidence; depression; anxiety	Platform predictors of completion	6.1% of participants in the group with no coach, 66.1% of participants in the high-support group, and 36% of participants in the low-support group completed the intervention; substantial and immediate dropout when compared with the high-support ( =–2.68; SE 0.35; =–7.65; OR 0.07, 95% CI 0.04-0.14; <.001) and low-support ( =–1.98; SE 0.34; =–5.76; OR 0.14, 95% CI 0.07-0.27; <.001; neither was significant) groups; Hispanic individuals were less likely to complete the intervention without a coach than non-Hispanic individuals ( =–3.99; <.001); higher levels of depressive symptoms predicted less drop-off with no coach ( =0.08; =.04)
OurRelationship [ ]; United States	RCT; brief OurRelationship with coach, brief OurRelationship without coach, or waitlist	Heterosexual couples; 104 couples; 40.4% at midintervention, 25% at end of intervention, and 17.4% at follow-up in the arm with a coach and 56% at midintervention, 26% at end of intervention, and 26% at follow-up in the arm without a coach	Relationship satisfaction; positive and negative relationship quality (positives); relationship confidence; communication; anxiety; depression; perceived health and QoL; work functioning	Platform predictors of completion	Dropout rate was 9.3% for the full OurRelationship and 28.8% for the brief OurRelationship with a coach ( =12.1; <.001); 71.2% completion in the coach condition and 42.3% completion in the no-coach condition ( ; =.003)
OurRelationship [ ]; United States	Pilot; pretest-posttest	Veterans and their partners; 13 couples; 15%	Relationship satisfaction and distress; relationship conflict; depression symptoms; probable PTSD; QoL	(1) Intervention satisfaction (CSQ-8); (2) completion	(1) Mean (veterans) 3.4/4 (SD 0.4) and mean (partners) 3.2/4 (SD 0.6); 91% were mostly or very satisfied; 96% would recommend it; positive qualitative feedback included structure, videos of similar couples, and reminder calls; negative qualitative feedback included repetition, length of some content, and technical and logistic frustrations; couples preferred the coach calls; (2) completion rate was 85%; median completion time was 52 (range 29-73) days; couples received clinical contact ranging from 52 to 95 minutes in total
OurRelationship [ ]; United States	Single arm; pretest-posttest	Coparenting couples; 136 couples; 20%	Relationship satisfaction and distress ; coparenting satisfaction ; gatekeeping and gate closing behaviors ; perception of partner’s gatekeeping and gate closing behaviors	Not reported	Not reported
OurRelationship [ ]; United States	RCT; OurRelationship or OurRelationship+ with greater therapist engagement	Couples; 314 couples; 64.3%	Individual use, joint use and perception of partner’s pornography use; arguments surrounding self-, joint, and partner’s pornography consumption; individual pornography use ; problematic pornography use; lifestyle changes due to the COVID-19 pandemic	Not reported	Not reported
OurRelationship [ ]; United States	RCT; full coach, automated coach, contingent coach, or waitlist	Couples; 740 couples; 30%	Relationship satisfaction (comparable across all types of coach support)	Completion	Completion comparable across all conditions; posterior distributions indicated that the probability of full-coach couples having higher odds of completing phases 1, 2, and 3 relative to automated-coach couples was 28.4%, 43.9%, and 77.4%, respectively; probability of full-coach couples having higher odds of completing phases 1, 2, and 3 relative to contingent-coach couples was 65%, 70%, and 92.7%, respectively; probability of contingent-coach couples having higher odds of completing phases 1, 2, and 3 relative to automated-coach couples was 15.6%, 22.6%, and 21.7%, respectively
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Romantic couples; 742 couples; 10.3% at posttreatment time point, 12.5% at 2-month follow-up, and 13% at 4-month follow-up	Relationship satisfaction ; communication conflict ; emotional support ; intimate partner violence ; breakup potential	(1) Evaluation (Client Evaluation of Services Questionnaire); (2) completion	(1) Participants rated the intervention positively (mean 9.9/11); 96% would recommend it to a friend; 93% were satisfied; no significant difference between OurRelationship and ePREP in satisfaction ( =–0.058; SE 0.148; =.70); (2) 69% in both ePREP and OurRelationship completed all content
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP or waitlist	Romantic couples; 742 couples; 10.3% at posttreatment time point, 12.5% at 2-month follow-up, and 13% at 4-month follow-up	Relationship satisfaction ; breakup potential ; negative communication ; positive communication ; relationship problem intensity ; relationship problem confidence ; emotional support	Not reported	Not reported
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Romantic couples; 742 couples; 10.3% at posttreatment time point, 12.5% at 2-month follow-up, and 13% at 4-month follow-up	Psychological distress ; perceived stress ; anger ; problematic alcohol use ; perceived health ; insomnia ; exercise	Not reported	Not reported
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Romantic couples; 742 couples; 10.3% at posttreatment time point, 12.5% at 2-month follow-up, and 13% at 4-month follow-up	Cooperative parenting; parenting stress; parenting nurturance (OurRelationship); physical and harsh verbal discipline (OurRelationship)	Not reported	Not reported
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Romantic couples; 742 couples; 10.3% at posttreatment time point, 12.5% at 2-month follow-up, 13% at 4-month follow-up, and 18.6% at 12-month follow-up	Long term: relationship satisfaction ; breakup potential ; positive communication ; communication conflict ; emotional support ; intimate partner violence; psychological distress ; perceived stress ; anger ; alcohol use ; perceived health ; insomnia	Not reported	Not reported
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Military and nonmilitary couples; 90 military couples; 43% for military couples	Relationship satisfaction ; communication conflict ; emotional support ; breakup potential ; intimate partner violence; psychological distress; perceived stress; anger; substance use; perceived health	(1) Evaluation (Client Evaluation of Services Questionnaire); (2) completion	(1) Evaluation ratings were similarly positive ( =0.470; =.07); (2) 57% of military couples completed the entire intervention (compared with 71% of civilian couples), 8% completed two-thirds, 18% completed one-third, and 18% completed none
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Low-income couples; 671 couples; 36% for OurRelationship and 31% for ePREP	Relationship satisfaction ; communication conflict ; emotional support ; intimate partner violence; breakup potential (not maintained long term for ePREP)	(1) Evaluation (Client Evaluation of Services Questionnaire); (2) completion	(1) Participants’ satisfaction: mean (OurRelationship) 9.51/11; mean (ePREP) 9.6/11; >95% of participants indicated that the intervention helped them; 97% indicated that they would recommend the intervention; 90% were satisfied with the intervention; no reliable differences in satisfaction between the 2 interventions ( =0.07, 95% CI –0.07 to 0.21); (2) 64% completed OurRelationship, and 69% completed ePREP
OurRelationship and ePREP [ ]; United States	RCT, OurRelationship, ePREP, or waitlist	Low-income perinatal couples; 180 couples; 32.8% for OurRelationship and 36.1% for ePREP	Relationship satisfaction ; perceived likelihood of breakup ; communication conflict ; sexual intimacy ; emotional support ; experience of intimate partner violence; psychological distress ; perceived stress (OurRelationship only)	Not reported	Not reported
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Low-income couples; 659 couples; 16.8%	Relationship satisfaction	Not reported	Not reported
OurRelationship and ePREP [ ]; United States	RCT; OurRelationship, ePREP, or waitlist	Low-income couples; 615 couples; not reported	Perceived gratitude from partner ; relationship satisfaction ; relationship instability ; communication skills ; destructive communication ; partner emotional support	Not reported	Not reported
ParentSTRONG [ ]; United States	RCT; waitlist	Early adolescent male individuals and a parent or guardian; 119 dyads; 8.5%	Dating violence behaviors ; parent-child communication ; attitudes supporting dating violence; aggression; emotional regulation	Acceptability and fidelity	90% of families completed all 6 modules; 87% of parents rated helpfulness as >4/5, and 99% of parents rated helpfulness as >3/5; 65% of teenagers rated helpfulness as >4/5, and 96% of teenagers rated helpfulness as >3/5; intervention did not allow participants to progress without completing all activities
PACT [ ] Australia	RCT; waitlist	Parent-child dyads in which the child (aged 2-10 years) had cerebral palsy; 67 dyads; 24.4%	Emotional availability ; child involvement ; QoL ; parental mindfulness ; parental acceptance ; adjustment	Not reported	Not reported
ParentWorks [ ]; Australia	Single arm; pretest-posttest measures	Parent or caregiver of a child aged 2-16 years; 388 families; 92.7% (nonstarters included)	Dysfunctional parenting ; interparental conflict ; child behavioral difficulties ; parental mental health	Satisfaction (CSQ)	Mean 5.49 (SD 0.95); no significant sex differences (t =0.41; >.05), indicating that mothers and fathers were equally satisfied
ParentWorks [ ]; Australia	Single arm; pretest-posttest measures	Parent or caregiver of a child aged 2-16 years; 388 families; 92.7% (nonstarters included)	Parent and family functioning; parenting conflict; child behavioral difficulties; parental mental health	(1) Completion; (2) dropout characteristics; (3) participant predictors of completion	(1) For partial completers, mean 2.4/5 (SD 1.2) modules completed; for full completers, mean 5.58/6 (SD 0.76) modules completed (including 1 optional module); (2) mothers in the full completer and partial completer groups reported higher levels of conduct problems than nonstarters =3.99; <.05); (3) relative to full completers, nonstarters were more likely to have older children, be married or in a de facto relationship, have higher levels of psychological difficulties, and have lower levels of child conduct problems; relative to full completers, partial completers were more likely to be married or in a de facto relationship and have higher levels of dysfunctional parenting
PERC [ ]; United States	Single arm; pretest-posttest	Couples where one member had a prostate cancer diagnosis; 26 couples; 15%	Dyadic communication; relationship satisfaction; QoL ; symptom distress ; general symptoms	(1) Feasibility and acceptability; (2) web activity; (3) ease of use	(1) 96% completed the intervention; (2) 37% of couples always logged in together, and 23% always logged in individually; mean 3.64 (SD 1.68) log-ins per couple; mean time spent on the platform per couple: 56.96 (SD 39.74) minutes; 83% used audio-enhanced slides; 94% visited the assignment and exercise section; (3) participants rated PERC as easy to use, engaging, and of high quality
Resilient Living [ ]; the Netherlands	Pilot feasibility trial	Patients with stroke or brain tumor and their caregivers; 16 participants; 68.75%	Dyadic coping; resilience; stress; caregiver role overload; QoL; fatigue ; physical function ; anxiety ; sleep	(1) Intervention evaluation; (2) WiWi	(1) Mean 2.6/5 for “Do you think the skills you learned enhanced your resilience?” and mean 4.4/5 for “did you find the online intervention easy to use?”; remaining mean scores ranged between 3.3 and 4.2/5; length of modules and ability to complete them in their own time were identified as facilitators to use; finding time to complete them as a dyad was challenging; (2) 4/5 indicated that it was worthwhile participating in the study, 4/5 indicated that it was as expected, and 1 indicated it was better than expected
Web-based partnership support program [ ]; Japan	Quasi-experimental design (nonrandomized); control	Infertile couples; 151 couples; 20.4%	QoL ; distress	Not reported	Not reported
Web-based PREP program [ ]; United States	RCT; IRC	Heterosexual foster or adoptive couples; 32 couples; 35%	Negative communication; knowledge acquisition ; use of PREP skills	Intervention feedback	Participants responded favorably to the intervention

a 4Cs:CRC: Caring for Couples Coping With Colorectal Cancer.

b RCT: randomized controlled trial.

c Indicates significance, or that the intervention was superior to the comparator, at the postintervention time point for the outcome measure.

d CA-CIFFTA: Computer-Assisted, Culturally Informed, and Flexible Family-Based Treatment for Adolescents.

e C-MBI: couple mindfulness-based intervention.

f YBCS: young breast cancer survivor.

g MBI: mindfulness-based intervention.

h I-MBI: mindfulness-based intervention for individuals.

i HOPES: Helping Overcome Posttraumatic Stress Disorder and Enhance Satisfaction.

j PTSD: posttraumatic stress disorder.

k QoL: quality of life.

l AOD: alcohol and other drug.

m CSQ: Client Satisfaction Questionnaire.

n TSQ: Treatment Satisfaction Questionnaire.

o eMB: mothers and babies online course.

p CSQ-8: 8-item Client Satisfaction Questionnaire.

q ePREP: computer-based Prevention and Relationship Enhancement Program.

r IRC: internet resource comparison.

s FOCUS: family involvement, optimistic outlook, coping effectiveness, uncertainty reduction, and symptom management.

t iCBT: internet-delivered cognitive behavioral therapy.

u MR: Mission Reconnect.

v PREP: Prevention and Relationship Enhancement Program.

w OFPS: Online Family Problem-Solving Therapy.

x TBI: traumatic brain injury.

y WEQ: Website Evaluation Questionnaire.

z OSS: Online Satisfaction Survey.

aa SES: socioeconomic status.

ab CAPS: counselor-assisted problem-solving.

ac F-PST: family-problem-solving therapy.

ad TOPS: teen online problem-solving.

ae LI-IPV: low-intensity intimate partner violence.

af OR: odds ratio.

ag PACT: Parenting Acceptance and Commitment Therapy.

ah PERC: Prostate Cancer Education and Resources for Couples.

ai WiWi: Was It Worth It questionnaire.

The Platforms

A total of 24 unique platforms were identified from the 85 studies. Table 2 shows the characteristics of the 24 platforms, including the intervention target; relationship targeted; duration of intervention participation; components designed for cocompletion, individual completion, and therapist engagement; any tailoring offered; and additional reported features.

Most interventions (14/24, 58%) were designed for cocompletion by couples, with some identified interventions for parent-child dyads (6/24, 25%), families (2/24, 8%), and caregiver–care recipient dyads (2/24, 8%). Given that it was expected that build characteristics might differ according to the population (eg, number of participating family members and their ages), platform results are grouped and reported by the relationship structure targeted by the platform (ie, couples, parent-child dyads, families, and caregiver–care recipient dyads).

Data from Table 2 are synthesized based on the features of the platforms and detail reported user engagement indicators. As platforms were included only in cases in which at least one study had demonstrated clinical efficacy of the intervention, mental health and relational outcomes are not reported in this table (and are, instead, indicated in Table 1 ).

Platform	Target relationship; intervention target; intervention duration	Self-paced components	Cocompletion versus individual completion	Practitioner engagement components	Tailored platform components and additional key features
4Cs:CRC	Couples; patient–partner coping with cancer; 6 weeks	6 intervention sections including dyadic learning sessions, health information, cancer news, web-based counseling, sharing circle, and personal center	Content intended to be completed by couples together	Face-to-face or web-based synchronous counseling sessions delivered biweekly to revisit content and provide additional support (some study conditions)	Weekly reminders to complete web-based sessions
CA-CIFFTA	Parent-child; treat behavioral problems and family conflict in young minority adolescents and their families; 12 weeks	4-6 computer-based modules; links to academic websites	Parents watched videos independently first, then rewatched with the adolescent; individual log-ins; role-appropriate videos	6-10 face-to-face sessions; fortnightly phone calls; asynchronous communication	Modular format for families to select content most relevant to the family’s clinical and cultural needs and preferences; custom links
Cool Kids Online	Parent-child; psychoeducation and CBT -based anxiety management skills for children and their parents; 10 weeks	8 web-based lessons—first 6 released weekly and final 2 released biweekly	Web-based lessons completed together; parent trained as a “coach” for their child; additional web-based information provided to caregivers at the end of each lesson	Parents completed weekly phone calls with clinician—reinforce success, clarify questions, assist with barriers and skill implementation, reinforce practice, and normalize experience	Automated reminder emails—emails reinforced content, skill practice, and engagement
C-MBI for YBCSs	Couples; relationship distress for couples where one member is a breast cancer survivor; 8 weeks	8 weekly, prerecorded videos delivered via the web; video links and reminders emailed to participants weekly	All videos watched together	Participants encouraged to email or call research staff regarding questions or content during participation	None
Couple HOPES	Couples; relationship functioning when one partner has PTSD ; 8 weeks	7 web-based modules containing videos, exercises, and practice assignments completed sequentially	Videos and module exercises completed together; partners had separate, linked accounts where they independently completed assignments; assignment entries and scores could be seen by both partners	4 scheduled calls with a coach after modules 1, 3, 5, and 7 plus 1 additional call as needed; engagement and adherence facilitated through platform messaging; coaches’ role involved reviewing symptom change, reinforcing successes, enhancing motivation for engagement, and troubleshooting barriers	Automated feedback graph depicted reported symptom change over time; progress bar and module menu communicated and incentivized progress; web-based application and smartphone app
Couplelinks	Couples; relationship functioning after cancer diagnosis; 8 weeks	6 modules; each module begins with an informational component followed by instructions for interactive exercises; couples reflect after each module; additional articles and video resources available	Modules completed together	Asynchronous platform-based messaging; introductory telephone call and 2 brief “check-ins” to reinforce alliance and promote adherence	Additional noncompulsory content
eMB	Couples; increase partner’s understanding of perinatal mood and anxiety disorders and therapeutic approaches to managing associated symptoms; 8 weeks	Recommended completion of 1 lesson per week in any order, with revisits as needed; psychoeducational modules containing YouTube videos, vignettes, interactive quizzes, homework, guided meditation, and downloadable resources	Participants could choose whether to complete separately or together	None	Could be completed in any order
Embers the Dragon	Parent-child; supporting emotional development and parental responses to child behavior; 8 weeks	Two 6-minute animated episodes and accompanying videos and activities	Parent and child watch the episodes and complete postvideo activities together; following the episodes, parents watch explanation videos	None	None
ePREP	Couples; preventative intervention to enhance relationship satisfaction and mental health; 6 weeks	6 hours of web-based modules and approximately 1-2 hours of homework	Couples completed modules and homework together	Four 15-minute appointments with coach practicing skills; weekly reminder emails to complete content and links to resources	Computer based, could be completed from mobile or tablet
FOCUS	Caregiver–care recipient (family); psychosocial health of patients with cancer and their family caregivers; 6 weeks	3 sessions delivered sequentially, with time to practice skills learned in between	Dyads completed the sessions together	Asynchronous “help” function that generated an email to the project director	Tailored, app-generated messages provided web links addressing the dyad’s specific concerns; offered a choice of tailored activities to complete between web sessions; tailoring based on baseline information provided
iCBT	Families; family functioning when a child has an anxiety disorder diagnosis; 10 weeks	11 modules, including reading materials, film, animations, and illustrations	Parents worked on their modules first so that they could then work with the children; 7 modules aimed at parents only	Platform-based messages; tailored feedback after exercise completion; 3 telephone calls during treatment and additional ones as needed to clarify content, increase motivation, and solve problems	None
Military Family Foundations	Couples; military couples in the transition to parenthood; not specified	5 prenatal and 3 postnatal modules	Modules completed together	Email reminders sent to couples if they stopped engaging for >10 days	None
MindGuide Couples	Couples; preventative intervention centered on vulnerability to Korean middle adulthood depression, “Hwa-Byung,” and couple relationships; 5-7 weeks	4 modules over 16 sessions, maximum 60 minutes each; sessions included audio-recorded mindfulness activities, video lectures, practical tasks, and case-based scenarios	Modules 3 and 4 were joint sessions, including creating a shared vision; performed practical tasks together; modules 1 and 2 were completed individually	Coaching sessions after each module to promote participation via reflective dialogue and provide feedback on participants’ responses	None
MR	Couples; relationship functioning when a member is a veteran with a history of deployment in a post-9/11 combat operation; 16 weeks	11 activities delivered via instructional videos, guided audio, and written manuals	Sessions on “Connecting with Partner” could be completed alone or together; the remaining sessions were completed independently	None	Accessible through website and mobile apps
Mother-daughter program	Parent-child; mother-daughter relationship quality and reduced risk of underage drinking; 10 weeks (4 weeks for the brief version)	9-14 modules; different adaptations were developed; animated characters portrayed the adolescent girl and her mother	Modules completed together; participants independently logged in to complete questions about content; participants could not advance until both mother and daughter had completed this	None	None
OFPS (including CAPS and TOPS )	Families; family functioning when the child, adolescent, or teenager has a TBI ; 6 months	7-11 sessions; core sessions and additional supplementary sessions provided based on identified need; web-based content included problem-solving skills, video clips, exercises, and assignments	Website used by multiple family members together	Initial face-to-face session completed in the family’s home; telehealth session following web-based sessions to review exercises	Supplementary sessions provided based on personal need; family members selected their picture to indicate that they were present; when required, the platform would prompt particular family members to respond, and other times, the whole family was asked to respond together
OR	Couples; relationship distress; 6 weeks (brief OR=2 weeks)	3 sections including video examples and psychoeducation	Content completed separately; couple completed guided conversation together at the end of each section	4 phone calls during the intervention; asynchronous chat feature	Tailored report on improvement provided; in some studies, automated tailored emails were provided
ParentSTRONG	Parent-child; adolescent boy domestic violence prevention intervention; 4 weeks	6 modules comprising 4-6 activities; parents and teenagers progress through alternate reality as avatars	After module 1, all modules are completed by the parent and child together; module 1 (introduction) completed by parents only	Staff could be contacted to troubleshoot technology	None
PACT	Parent-child; emotional availability and parent and child adjustment when the child has cerebral palsy; 10 weeks (enforced break in the middle)	3 modules and a final review module after a short break	Some exercises were designed for individual completion	Fortnightly check-in (phone, SMS text message, or email) to monitor completion and check understanding of content	None
ParentWorks	Couples; father-inclusive parenting intervention; 4 weeks	5-8 modules	Participants accessed via a shared account; participants had the option to complete it independently	None	Feedback provided based on participant responses; formatted for mobile, laptop, and tablet viewing
PERC	Couples; relationship distress following a prostate cancer diagnosis; 8 weeks	7 modules—5 core and 2 optional	Encouraged to view and complete everything together	None	Optional modules; users could select text- or audio-based slides depending on preference
Resilient Living	Caregiver–care recipient (family); building dyadic resilience skills for patients with stroke or brain tumor and their family caregivers; 8 weeks	4 web-based video modules and participant journal	Option to complete individually	Telehealth session before commencement of web-based modules	None
Web-based partnership support program	Couples; support intervention to prevent quality of life deterioration and reduce emotional distress in men undergoing fertility treatment; 2 weeks	30-minute self-paced content over 10 days	Watched information together; discussion between couples using the communication form; couples individually completed their communication form, which was subsequently used to guide their discussion	None	None
Web-based PREP program	Couples; couple relationship education for foster or adoptive parents; 1 week	4 chapters plus additional resources	Entire intervention completed together	None	None

b CA-CIFFTA: Computer-Assisted, Culturally Informed, and Flexible Family-Based Treatment for Adolescents.

c CBT: cognitive behavioral therapy.

d C-MBI: couple mindfulness-based intervention.

e YBCS: young breast cancer survivor.

f HOPES: Helping Overcome Posttraumatic Stress Disorder and Enhance Satisfaction.

g PTSD: posttraumatic stress disorder.

h eMB: mothers and babies online course.

i ePREP: computer-based Prevention and Relationship Enhancement Program.

j FOCUS: family involvement, optimistic outlook, coping effectiveness, uncertainty reduction, and symptom management.

k iCBT: internet-delivered cognitive behavioral therapy.

l MR: Mission Reconnect.

m OFPS: Online Family Problem-Solving Therapy.

n CAPS: counselor-assisted problem-solving.

o TOPS: teen online problem-solving.

p TBI: traumatic brain injury.

q OR: OurRelationship.

r PACT: Parenting Acceptance and Commitment Therapy.

s PERC: Prostate Cancer Education and Resources for Couples.

t PREP: Prevention and Relationship Enhancement Program.

Features of Platforms for Couples

Of the platforms requiring cocompletion, platforms designed for couples were the most common. A total of 58% (14/24) of the identified platforms were for couples. The intervention targets included relationship distress when a member has a cancer diagnosis (2/14, 14%); relationship functioning when a member has a cancer diagnosis (2/14, 14%), has posttraumatic stress disorder (1/14, 7%), or is a veteran (1/14, 7%); parenting-focused interventions, including a father-inclusive parenting intervention (1/14, 7%), education for foster and adoptive parents (1/14, 7%), and an intervention for military couples transitioning to parenthood (1/14, 7%); partnership support interventions for cases in which the male partner is undergoing treatment for infertility (1/14, 7%) or a member is pregnant (1/14, 7%); general relational distress (1/14, 7%); and preventative interventions to enhance relationship satisfaction and mental health (1/14, 7%) and reduce vulnerability to middle adulthood depression (1/14, 7%).

Structure and Duration of Engagement

Duration of participation varied from 1 to 16 weeks, with the most common duration being 8 weeks (5/14, 36%) followed by 6 weeks (4/14, 29%), including 1 intervention described as taking 5 to 7 weeks. The intended duration of 7% (1/14) of the interventions was not specified. One intervention offered a brief version that was completed by couples in 2 weeks as opposed to the 6-week full version. As per the inclusion criteria for this review, all interventions involved some web-based self-paced component completed on the platform. Most appeared to require at least weekly engagement, although it was not always specified or prescribed. One platform was designed such that participants could complete the intervention modules in any order but advised participants to access 1 module per week and complete all modules. For all the remaining interventions, it appeared that intervention content or modules were designed to be completed in a defined order and over a specified period.

Coparticipation and Contact With Practitioners

A total of 43% (6/14) of the interventions contained elements that were intended for individual completion (ranging from completing assessments to completion of entire sections of content), 50% (7/14) of the interventions required couples to cocomplete the whole intervention, and 7% (1/14) of the interventions gave participants the choice to complete some or all of the intervention together. In total, 57% (8/14) of the interventions included an element of practitioner engagement, including asynchronous platform-based messaging or scheduled synchronous counseling sessions.

Tailoring and Additional Features

Beyond personalization through contact with practitioners, 29% (4/14) of the platforms provided tailored content or options for personalization. A total of 14% (2/14) of the platforms provided supplementary content that could be accessed based on need, and 14% (2/14) of the platforms provided personalized feedback and reporting based on responses to questionnaires. In total, 29% (4/14) of the platforms specified that they were formatted for both web and mobile or tablet use, and 7% (1/14) of the platforms allowed participants to select either audio-enhanced or text-based presentation of content. Finally, 7% (1/14) of the platforms included an automated graph depicting reported symptom change over time and a progress bar to incentivize participation.

Reported User Engagement Indicators of Platforms for Couples

A total of 56% (48/85) of the studies examined the 14 couple-focused platforms. Of those 48 studies, 30 (62%) reported on user engagement indicators, including 23 (77%) studies that reported on satisfaction, feedback, usability, participant evaluation, feasibility, and acceptability and 18 (60%) that reported on completion rates and website use. The remaining 38% (18/48) of the studies did not report on any user engagement data or findings.

Measures used to collect participant satisfaction, feedback, usability, and evaluation varied. A total of 10% (5/48) of the studies administered the Client Evaluation of Services Questionnaire [ 126 ], and 15% (7/48) used the Client Satisfaction Questionnaire [ 127 ]. The remaining studies reported on satisfaction, feedback, and participant evaluation through nonvalidated measures. Satisfaction ratings were generally high across all studies.

The impact of video content on user engagement appeared mixed. Participants in 8% (4/48) of the studies provided feedback that the content and examples presented in the videos were helpful; however, in another 4% (2/48) of the studies, participants reported that the videos were unhelpful or that they negatively impacted engagement as they were not relatable, overly dramatized, or appeared outdated. In addition, participant qualitative feedback reported in another study suggested that outdated imagery and low-technology visualizations also negatively impacted engagement. Other factors that were reported to be important based on qualitative feedback included one study that reported on the structured nature of the intervention and reminder calls and another where participants reported that they were more likely to access audio-enhanced slides than text-based content. Feedback provided by participants in one study also noted that the flexibility of the web-based format facilitated engagement. However, in general, satisfaction, feedback, usability, and evaluation data were reported as average values on rating scales.

Reporting of completion rates and website use rates varied. They were reported as combinations of the following: the average number of participants who completed the entire intervention, the average number of modules or sessions completed by individuals or couples, the average time to completion, the number of discrete log-ins or page views, and the amount of time spent accessing the platform. Feasibility and acceptability data were reported similarly, with completion statistics often used as an indication of an intervention’s feasibility or acceptability. In addition, 5 studies reported on predictors of noncompletion, including 3 (60%) studies that reported higher levels of support from a practitioner as predictors of completion. The remaining 40% (2/5) of the studies reported on participant baseline characteristics as predictors of noncompletion.

Finally, 4% (2/48) of the studies on the same platform identified different couple “types” with regard to their enthusiasm and engagement (eg, “keen completers” or “stragglers”) and therapists’ role in engagement promotion. One study reported that higher levels of engagement (measured using participants’ correct responses to quiz questions) led to greater intervention effect on a number of clinical outcomes, and another found that those with the shortest time frame between commencement and completion (ie, completed the intervention faster) were more likely to be classified as “treatment responders” (identified by significant improvement on outcomes) at the postintervention assessment.

No studies of couple-based platforms identified build or design characteristics as moderators of intervention effect. No studies performed a formative evaluation of the platforms, and no studies reported design and build characteristics that enabled coparticipation beyond participant qualitative feedback.

Parent-Child Dyads

Features of platforms for parent-child dyads.

Platforms designed for co-use by parent-child dyads were the second most common, accounting for 25% (6/24) of the platforms identified in this review. The intervention targets included behavioral problems and conflict in young minority adolescents and their families (1/6, 17%), emotional development and parental responses to child behavior (1/6, 17%), mother-daughter relationship quality and risk of underage drinking (1/6, 17%), adolescent male domestic violence prevention (1/6, 17%), emotional availability and parent-child adjustment when a child has cerebral palsy (1/6, 17%), and anxiety management skills and psychoeducation for parents and children (1/6, 17%). A total of 67% (4/6) of the platforms were developed for adolescents and a parent, 17% (1/6) were for young children aged 2 to 7 years and a parent, and 17% (1/6) were for children aged 7 to 12 years and their parents. In all cases, only 1 parent was asked to participate. In the following sections, we summarize the reported features of the platforms as detailed in the included studies.

Duration of intervention use varied from 4 to 12 weeks, with the most common duration being 10 weeks (3/6, 50%). One intervention of a 10-week duration in total enforced an extended break in the middle of intervention engagement, and another offered a brief version of only 4 weeks (compared with the 10-week full version). Participation varied from once a fortnight to 2 web-based sessions or modules a week. A total of 83% (5/6) of the interventions appeared to involve completion in a structured manner following a predetermined order. One platform presented intervention content in a modular format that allowed participants to select the content that was relevant to their needs and cultural preferences in any order. However, the intended duration and number of modules accessed appeared to be prescribed.

The amount and method of coparticipation varied greatly. A total of 33% (2/6) of the platforms required parents to watch the intervention content or preparatory materials before engaging with their adolescent child. In total, 33% (2/6) of the platforms required the parent to complete explanation videos or additional content following cocompletion with their young child. A total of 17% (1/6) of the platforms involved cocompletion of all intervention modules and independent completion of questions about content, with both the parent and adolescent required to complete these questions before the dyad could progress to the next module. Finally, studies on 17% (1/6) of the platforms reported that “some exercises” were designed for cocompletion but did not specify the extent of cocompletion.

A total of 67% (4/6) of the interventions included contact with a practitioner, whereas 33% (2/6) were entirely self-guided. One of those offering contact with a practitioner only offered this to parents and not the participating child. A total of 33% (2/6) of the interventions included scheduled sessions with a practitioner to discuss content, with 17% (1/6) also supporting asynchronous communication with a practitioner via the platform. Finally, in 17% (1/6) of the interventions, participants could contact practitioners via the platform for technical troubleshooting as required.

In total, 17% (1/6) of the platforms allowed participants to select content based on their clinical and cultural needs. Content was selected from a list of available modules, although the process through which the dyads selected this content was not described. This same platform offered dyads links to external sources of information based on their responses to questionnaires.

Reported User Engagement Indicators of Platforms for Parent-Child Dyads

A total of 14% (12/85) of the studies evaluated 6 different interventions designed for use by parent-child dyads. Of those 12 studies, 5 (42%) reported on user engagement indicators, including completion or fidelity (4/5, 80%) and satisfaction or acceptability (3/5, 60%). The remaining 58% (7/12) of the studies reported on mental health or relational outcomes and did not report on user engagement indicators.

The 33% (4/12) of the studies reporting on completion or fidelity documented the number of participants who completed the entire intervention as prescribed. One study also reported on the average time it took participants to complete the intervention, and another reported on the number of dyads who accessed all sessions and received calls from a practitioner. No studies reported on participants’ interaction with the platform or any predictors of noncompletion.

All studies reporting on satisfaction and acceptability did so using nonvalidated measures. Mean satisfaction ratings were high. One study asked participants to indicate how easily they found time to complete the activities together, with a mean rating of 3.04/5 (SD 0.37) for daughters and a mean rating of 3.24/5 (SD 0.33) for mothers. In no study did the satisfaction and acceptability data distinguish between platform and intervention satisfaction.

No studies on parent-child interventions identified platform build or design characteristics as moderators of intervention effect. No studies performed formative evaluations of the platforms, and no studies reported on design and build characteristics that enabled coparticipation.

Features of Platforms for Families

Among the 24 platforms, 2 (8%) designed for cocompletion by families were identified. The intervention targets included family functioning when a child has an anxiety diagnosis (1/2, 50%) and family functioning when a child, adolescent, or teenager has a traumatic brain injury (1/2, 50%). Both platforms were intended for use by a child, adolescent, or teenager with a presenting clinical concern and any family members, including parents and siblings. Though siblings and other family members were invited to participate, the studies detailed outcomes and engagement for a single parent and child only.

Intervention participation on one platform extended for 10 weeks over 11 web-based chapters, and the other delivered 7 to 11 sessions over 6 months. Both were designed for sequential completion of module content.

One platform asked family members to complete the entire intervention together. The other asked parents to complete sections themselves before working with their children on a small number of modules intended for cocompletion. Both included scheduled telehealth sessions with a practitioner during intervention participation. In addition, one platform also included a platform-based message system for contacting practitioners asynchronously. In this same platform, practitioners also provided reports to participants following exercise completion.

One platform provided no tailoring beyond engagement with and feedback provided by practitioners. The other platform included supplementary sessions that could be completed by families should they wish to. In addition, this platform supported cocompletion by asking family members to select their picture when they were present. The platform would then either prompt individual family members to respond or ask all family members to respond together.

Reported User Engagement Indicators of Platforms for Families

A total of 27% (23/85) of the studies examined the 2 family-based interventions. Of these 23 studies, 16 (70%) reported user engagement indicators including satisfaction and ease of use (n=9, 56%); completion rates, compliance, adherence, and website use (n=13, 81%); and feasibility (n=2, 12%). The remaining 30% (7/23) of the studies did not report on satisfaction, completion, or feasibility data or findings.

Of the 9 studies reporting satisfaction and ease of use, 6 (67%) used an adaptation of the Website Evaluation Questionnaire [ 128 ] to measure participant satisfaction with the intervention. The remaining 33% (3/9) of the studies administered nonvalidated measures developed for the studies. Satisfaction ratings were high across all studies. In 33% (2/6) of the studies in which the Website Evaluation Questionnaire was administered, participants were asked to rate the website’s ease of use, generally reporting that the website was “moderately easy” to “easy” to use. Participants in one study reported a preference for meeting in person. Other than this, satisfaction ratings either were relevant to content or did not distinguish between platform and intervention satisfaction.

Completion rates, compliance, adherence, and website use were all reported as combinations of the following: the number of participants who completed the entire intervention, the average number of modules completed, time spent on the platform, and the number of families who completed supplemental sessions. Feasibility was reported similarly, with one study also reporting that families were able to complete all sessions without practitioner assistance. In addition, one study reported on number of sessions completed as a predictor of symptom change (with inconsistent effect), and another reported on participant characteristics at baseline as predictors of completion.

A total of 13% (3/23) of the studies also measured participants’ technology use and comfort with technology before the commencement of the intervention and examined this as a predictor of intervention effect. Results were inconsistent. In addition, one study identified whether participants’ preference for treatment modality before the intervention, that is, face-to-face, web self-paced, or therapist-guided modality, impacted treatment outcomes. It was found that adolescent treatment preference was significantly related to attrition, but there were no other links with treatment effect or satisfaction.

No studies on family-based platforms identified build or design characteristics as moderators of the intervention effect. No studies performed a formative evaluation of the platforms, and aside from one study describing how participants identified that they were present, no studies reported design and build characteristics that enabled coparticipation.

Caregiver–Care Recipient (Family) Dyads

Features of platforms for caregiver–care recipient dyads.

Among the 24 platforms, 2 (8%) were identified for family caregiver–care recipient dyads. The targets of the interventions on the platforms included dyadic resilience for patients with stroke or brain tumor and their family caregivers (1/2, 50%) and the psychosocial health of patients with cancer and their family caregivers (1/2, 50%).

One of the 2 platforms involved intervention participation over 6 weeks, with 3 sessions delivered sequentially. The other platform contained 4 web-based modules and a participant journal completed over 8 weeks. While not explicitly reported, it appeared that this platform also required sequential completion of intervention content.

Both platforms were designed to be completed by members of the dyad together; however, one had the option of completing the entire intervention independently if desired. One platform contained an asynchronous help function that generated an email to the project director. The other included a telehealth session before commencement of the web-based component.

One platform contained several tailored elements, whereas the other did not offer any personalization. Tailoring included platform-generated messages that provided web links addressing the dyad’s concerns and supplementary activities offered between web sessions. Both were generated from self-reported baseline information.

Reported User Engagement Indicators of Platforms for Caregiver–Care Recipient Dyads

The 2 interventions were each evaluated in 1% (1/85) of the studies. Both studies reported on satisfaction, and one reported on feasibility, with both reporting high satisfaction ratings. One study reported that there were no adverse effects of participants completing the intervention on the web-based platform together, and the other identified the length of the modules and the ability to complete the intervention in the users’ own time as facilitators to use. In one study, participants noted that finding time to complete the intervention as a dyad was sometimes challenging. Where feasibility was reported, the study found lower enrollment rates than those for previous in-person randomized controlled trials but higher retention rates.

Neither study identified build or design characteristics as moderators of intervention effect. No study performed a formative evaluation of the platforms, and neither reported on design nor build characteristics that enabled coparticipation.

Principal Findings

This review details build, design, and user engagement characteristics of platforms that enable cocompletion of clinical interventions by related people. To distinguish effective platform contributors to engagement from elements pertaining to intervention content, we selected only those platforms housing interventions of established clinical efficacy (ie, previously reported significant improvement of at least one mental health or relational outcome). Some common design features were identified; however, in contrast to expected findings, specific design characteristics enabling cocompletion were rarely reported, and evidence for engaging families was underexplored.

Common Platform Features

This review identified platform design features that were common across the included studies. Regardless of the relationship targeted, most platforms delivered a structured intervention that required engagement over a prescribed duration with content completed sequentially. A total of 8% (2/24) of the platforms allowed participants to access content in a nonsequential manner, and a handful (4/24, 17%) offered supplementary content based on identified need. Retention rates remain low for DMHIs [ 29 ], and there are further complexities when family members participate together [ 50 ]. As such, consideration might be given to ways in which families’ time on the platform can be optimized.

Single Session Thinking is one process through which therapists treat each encounter as if it were the sole session, encouraging the participants to make the most of the time [ 129 ]. Adaptation to web delivery of family therapy sessions already holds promise [ 130 ], and digital single-session interventions have been trialed in college student settings with positive preliminary findings [ 131 - 133 ]. Therefore, there is emerging evidence suggesting that Single Session Thinking principles could be readily applied to DMHIs, mimicking single, stand-alone sessions that address the family’s present needs as they identify them. Check-in prompts and invitations to return as needed could be automated from the platform to encourage return visits as required or desired by the family. A platform designed to deliver content in this way would likely reduce the burden on families and provide greater flexibility in how they access content.

Minimal tailoring was offered in 29% (7/24) of the DMHIs identified in this review, providing more or less the same intervention to all participants. A total of 67% (16/24) of the interventions included interaction (either synchronous or asynchronous) with a practitioner. Evidence for personalized mental health care is growing rapidly, acknowledging the complexity and diversity of individuals and families [ 134 , 135 ]. Higher levels of engagement are reported for guided interventions (ie, those where participants have some contact with a practitioner) than for self-guided interventions; however, incorporating human contact can be costly and can limit the flexibility and accessibility associated with DMHIs [ 136 ]. Research suggests that, compared with targeted or generic feedback, personalization can be used to improve engagement and subsidize personal contact and contributes to positive attitudes toward a DMHI [ 134 , 135 ]. Beyond this, several studies included in this review (9/85, 11%) identified baseline characteristics that moderated participants’ responses to the intervention. These included characteristics such as age, relationship status, and previous comfort with technology. Understanding how baseline measures might impact participants’ ability or desire to engage with platforms and providing options for personalization accordingly would likely result in greater engagement. A family-based platform might include tailored design options such as color and font choices, preferences for video- or text-based content, and preferences for receipt of prompts and reminders. In addition, if children are present, families could have the option to access content that has been adapted for younger readers. In a world where artificial intelligence is supporting personalization across the internet, it would be remiss not to consider personalization in family- and relational-based DMHIs.

Platform Features for Enabling Cocompletion

By their nature, computers and mobile devices are designed for use by individuals. Given obvious complexities involved in having multiple people participate in a web-based intervention together, it was expected that platforms designed for such use may contain features for enabling cocompletion across the life span. It was also expected that the way in which participants engage may differ from that in platforms designed for individual use. This could include considerations about privacy of individual participants’ data, methods for encouraging participants to work together, and design choices to allow all participants to contribute to activities. One platform requested participants to select their image when they were in attendance, and this was then used to prompt individuals to respond and participate in activities. Other than this, no study identified platform characteristics that were included to specifically enable cocompletion. In general, studies detailed participants’ engagement with the intervention but not with the platform. Reporting on platform engagement might include details on how participants navigated the interface, how they identified and accessed content, or the modes through which content was delivered. On the other hand, intervention reporting was found to delve into factors such as attrition rate and measurement completion. Crucially, it is important to distinguish between intervention trial attrition (ie, dropout or loss to follow-up) and platform disengagement (ie, nonuse attrition), as recommended in a previous review [ 137 ]. These 2 forms of attrition are influenced by distinct factors [ 138 , 139 ], and failure to differentiate between them could potentially lead to misinterpretation of platform engagement dynamics.

It was also expected that studies would provide insights into the build and design considerations related to individual user privacy and safety within a shared web-based space. This encompasses considerations such as determining when an individual’s information can or should be shared with other members of the family and effectively identifying and responding to safety risks. From our perspective, these design aspects are essential considerations when developing a family-based DMHI. However, none of the studies identified in this review reported or discussed how they tackled or addressed these privacy and safety considerations. To further ensure the adequate addressing of not only these concerns and anticipate other potential considerations, rigorous co-design processes are essential. This co-design strategy would significantly contribute to the refinement of family-based DMHIs, ensuring that they meet the nuanced needs of users.

Engagement With Practitioners

The varied nature of engagement with guided tools (ie, involving interaction with practitioners, structured sessions, and feedback loops) stands in stark contrast to the self-guided use and consistent participation characterizing engagement with tools lacking contact with practitioners. Recognizing challenges intrinsic to self-guided tools, such as user motivation and adherence, becomes paramount, particularly given that the absence of practitioner involvement is likely to make the sustainability of user interest more demanding. The role of technology in promoting engagement with practitioners is multifaceted, encompassing communication facilitation through asynchronous methods and data-driven insights that enhance personalized interactions. Moreover, exploring hybrid models and incorporating periodic check-ins or teletherapy sessions within self-guided platforms presents a promising balance between autonomy and professional support.

Addressing challenges in technology engagement involves prioritizing user-centered design; integrating behavioral science principles; and leveraging feedback mechanisms, either automated or through clinician input, to ensure continuous support and guidance. Looking forward, suggested avenues for future research are many, including the long-term effectiveness of guided and self-guided tools, understanding the impact of different engagement strategies, and developing sophisticated technology-assisted therapeutic approaches.

Evidence for Enabling Cocompletion

We faced constraints in reporting evidence on platform features that engaged and enabled cocompletion by families because no study conducted a direct evaluation of the platform design. This limitation hindered our ability to provide comprehensive insights into the effectiveness of features promoting cocompletion among participating family members. In addition, while several studies (13/85, 15%) evaluated practitioner support, family member coparticipation, population characteristics, and baseline scores on mental health or relational measures as moderators of intervention outcomes, no study evaluated design features as potential moderators of intervention outcomes.

Of the 85 included studies, 66 (78%) reported on user engagement indicators. Of those, most (48/66, 73%) used custom, nonvalidated measures, and the remaining studies used validated measures that were intervention specific and gave no information about platform engagement. Given this measurement heterogeneity, little is possible by way of cross-study comparison. In addition, without evaluation of platform design strategies, no conclusions can be drawn about enabling or disabling features. The capacity for real-world translation and understanding of how to overcome known barriers is constrained.

A Need for Cohesive Platform Evaluation and Reporting

Platform user experience design, including ease of use, navigation, screen layout, readability, gamification, feedback, and attractiveness, plays a large role in a participant’s perception of and engagement with a website and, ultimately, a site’s usability [ 29 , 140 , 141 ]. In addition, individual participant characteristics such as age, literacy level, level of disability, and mental health conditions may impact their engagement with and ability to use a platform as designed. When a family presents on a web-based platform, more than one person’s needs must be catered to.

There is a lack of consensus and shared understanding of how to usefully conceptualize and measure engagement with and accessibility of digital mental health platforms [ 35 , 36 ]. This variability is not unique to the context of family-based mental health platforms, with reviews of engagement in digital mental health reporting similar heterogeneity [ 35 , 134 , 142 ]. Studies tend to report on measures such as completion or attrition rates, usability, user satisfaction, acceptability, and feasibility as indicators of how well the application engaged users. Often, these data are self-reported. Given the high attrition rates for self-guided platforms [ 102 ] and additional complexities involved in requiring family members to cocomplete activities [ 50 ], understanding platform characteristics that enable co-use and promote engagement is vital to informing the future development of such platforms. There is limited direct evidence to support practitioners, developers, and designers in understanding why engagement levels remain low, and there remains a limited understanding of how to design a DMHI to optimize engagement for families.

Assessment of user engagement indicators such as completion data alone is likely insufficient to measure how well a platform engaged its users. For example, reporting on duration of participation and sessions completed neglects factors impacting how families navigate the website, such as interface design and organization, and user characteristics. Analysis of platform use patterns and baseline characteristics in addition to these completion statistics would provide greater insights into how families engage with a platform. Formative as opposed to summative evaluations of usability are conducted to inform the redesign and improvement of a web interface. Formative evaluations consider multiple factors and involve building a deep understanding of user perceptions and use patterns of platforms. In addition to self-reported measures and completion rates, formative evaluations often also consider website analytics such as bounce rate, pages per session, top exit pages, and the pathways that users take to get to pages where they ultimately spend most of their time. It is a recommendation of this review that formative evaluations of web-based mental health platforms become common practice for DMHIs, particularly for novel and complex applications such as family-based platforms.

Finally, a systematic review of evaluations of usability of mobile mental health technologies [ 143 ] recommended closer collaboration between health care and computer science experts when evaluating DMHIs, suggesting that this would increase the quality of interpretation of the evaluation. A summary of learnings from the ParentWorks trial identified an expected benefit of having involved a web agency during the early stages of content translation to optimize user experience [ 144 ]. An interdisciplinary approach might enhance knowledge sharing, too, through detailed reporting of DMHI design decisions and their interactions with platform elements and clinical outcomes.

Clearly, there remains a need for coherent reporting and evaluation practices in the field to inform guidelines and policy on effective strategies for engaging families on the web in mental health–related interventions. Until rigorous co-design with families and an interdisciplinary approach between content experts and user experience designers is taken to formative evaluations, the growth and expansion of efficacious mental health platforms for family use will lag.

Study Strengths, Limitations, and Future Research Directions

This study represents the first of its kind. Using a replicable search strategy over 4 periods, we synthesized in this study the state of the published evidence regarding platform design and build characteristics enabling successful engagement of related parties with digitally delivered mental health interventions. Given that gray literature was not searched for this review, it is possible that emerging evidence for new multiuser digital platforms was missed. Our findings are limited by the technical reporting of the studies. Principally, many studies did not provide details about their platform build or the way in which participants engaged with the platform, including whether coparticipation was expected. Where this information was not provided, the study authors were contacted, and websites were searched to retrieve the relevant information. It is likely that examination of some relevant functionality was precluded when this information was not provided or was insufficient.

Many studies (24/85, 28%) explicitly excluded participants when those other than the identified person had a mental illness. Whether through caregiving burden, stigma, or familial shared conditions, it is rare for a family presenting for therapy to have only 1 member experiencing mental health stress or significant challenges [ 145 , 146 ]. Given the potential of these platforms to aid family therapy, further research with families in which multiple members experience stress or mental health challenges is needed. Until then, it is difficult to generalize the evidence reported in this review to the real-world experience of families who may present for family therapy.

In addition, diversity in populations was limited, with most studies including White, heterosexual, and middle-class participants. There was a lack of evidence from low- and middle-income countries (LMICs), with all studies conducted in more high-income countries. The technological experiences and needs of families in LMICs will likely vary significantly from those in more high-income countries given, among other factors, the varying degree of ease of access to technology. Digital interventions have the potential to expand reach and access to services; however, until participants from LMICs are included in studies of digital platforms for families, findings cannot be generalized to these populations and ultimate reach will be limited.

As this is a new and novel field, language and terminology are still being defined, and means of measuring and defining engagement and feasibility are not well established [ 29 ]. Of the included studies, 52% (44/85) were published in the last 5 years, reflecting rapid developments in technology and associated applications.

Conclusions

While there is emerging evidence suggesting that DMHIs are clinically effective, there remains a large evidence gap in the literature on the extent to which platform-specific design and build elements may also contribute to timely access, user experience, safe cocompletion by family members, and clinical outcomes. In the service of improved mental and relational health outcomes, our findings point to a significant opportunity for meaningful cross-disciplinary research, development, and evaluation of family-based mental health platforms. Findings from the next era of research will be central to enabling policy and practice advancements in equitable access to effective mental health care support for families.

Data Availability

All data generated or analyzed during this study are included in this published article (and its supplementary information files).

Conflicts of Interest

None declared.

Excluded studies and platforms.

Search strategies.

Rehm J, Shield KD. Global burden of disease and the impact of mental and addictive disorders. Curr Psychiatry Rep. Mar 07, 2019;21(2):10. [ CrossRef ] [ Medline ]
Duden GS, Gersdorf S, Stengler K. Global impact of the COVID-19 pandemic on mental health services: a systematic review. J Psychiatr Res. Oct 2022;154:354-377. [ FREE Full text ] [ CrossRef ] [ Medline ]
Foster K, Maybery D, Reupert A, Gladstone B, Grant A, Ruud T, et al. Family-focused practice in mental health care: an integrative review. Child Youth Serv. Mar 23, 2016;37(2):129-155. [ FREE Full text ] [ CrossRef ]
Anderson H. Rethinking family therapy: a delicate balance. J Marital Fam Ther. Jun 08, 2007;20(2):145-149. [ FREE Full text ] [ CrossRef ]
Whisman MA. Marital distress and DSM-IV psychiatric disorders in a population-based national survey. J Abnorm Psychol. Aug 2007;116(3):638-643. [ CrossRef ] [ Medline ]
Department of Health and Human Services. Police-recorded crime trends in Victoria during the COVID-19 pandemic: update to end of December. Crime Statistics Agency. 2020. URL: https://www.crimestatistics.vic.gov.au/research-and-eva luation/publications/police-recorded-crime-trends-in-victoria-during-the-covid-19-1 [accessed 2024-04-29]
Painter FL, Booth AT, Letcher P, Olsson CA, McIntosh JE. The lived experience of stress for parents in the context of COVID-19–related disruption. Fam Relat. Mar 07, 2023;72(4):1511-1531. [ FREE Full text ] [ CrossRef ]
Chung G, Lanier P, Wong PY. Mediating effects of parental stress on harsh parenting and parent-child relationship during coronavirus (COVID-19) pandemic in Singapore. J Fam Violence. 2022;37(5):801-812. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wolf JP, Freisthler B, Chadwick C. Stress, alcohol use, and punitive parenting during the COVID-19 pandemic. Child Abuse Negl. Jul 2021;117:105090. [ FREE Full text ] [ CrossRef ] [ Medline ]
Connell CM, Strambler MJ. Experiences with COVID-19 stressors and parents' use of neglectful, harsh, and positive parenting practices in the Northeastern United States. Child Maltreat. Aug 2021;26(3):255-266. [ FREE Full text ] [ CrossRef ] [ Medline ]
Bartle-Haring S, Slesnick N, Murnan A. Benefits to children who participate in family therapy with their substance-using mother. J Marital Fam Ther. Oct 2018;44(4):671-686. [ FREE Full text ] [ CrossRef ] [ Medline ]
Horigian VE, Feaster DJ, Robbins MS, Brincks AM, Ucha J, Rohrbaugh MJ, et al. A cross-sectional assessment of the long term effects of brief strategic family therapy for adolescent substance use. Am J Addict. Oct 2015;24(7):637-645. [ FREE Full text ] [ CrossRef ] [ Medline ]
Aldersey HM, Whitley R. Family influence in recovery from severe mental illness. Community Ment Health J. May 2015;51(4):467-476. [ CrossRef ] [ Medline ]
Carr A. The effectiveness of family therapy and systemic interventions for adult‐focused problems. J Fam Ther. Dec 23, 2008;31(1):46-74. [ FREE Full text ] [ CrossRef ]
Carr A. The evidence base for family therapy and systemic interventions for child‐focused problems. J Fam Ther. Jan 29, 2014;36(2):107-157. [ FREE Full text ] [ CrossRef ]
Carr A. How and why do family and systemic therapies work? Aust N Z J Fam Ther. Apr 05, 2016;37(1):37-55. [ FREE Full text ] [ CrossRef ]
Hartmann M, Bäzner E, Wild B, Eisler I, Herzog W. Effects of interventions involving the family in the treatment of adult patients with chronic physical diseases: a meta-analysis. Psychother Psychosom. 2010;79(3):136-148. [ CrossRef ] [ Medline ]
O'Brien M, Daley D. Self-help parenting interventions for childhood behaviour disorders: a review of the evidence. Child Care Health Dev. Sep 2011;37(5):623-637. [ CrossRef ] [ Medline ]
Cardy JL, Waite P, Cocks F, Creswell C. A systematic review of parental involvement in cognitive behavioural therapy for adolescent anxiety disorders. Clin Child Fam Psychol Rev. Dec 2020;23(4):483-509. [ FREE Full text ] [ CrossRef ] [ Medline ]
WHO guideline: recommendations on digital interventions for health system strengthening. World Health Organisation. URL: https://iris.who.int/bitstream/handle/10665/311941/9789241550505-eng.pdf?sequence=31 [accessed 2024-04-29]
Naslund JA, Aschbrenner KA, Araya R, Marsch LA, Unützer J, Patel V, et al. Digital technology for treating and preventing mental disorders in low-income and middle-income countries: a narrative review of the literature. Lancet Psychiatry. Jun 2017;4(6):486-500. [ FREE Full text ] [ CrossRef ] [ Medline ]
Sherifali D, Ali MU, Ploeg J, Markle-Reid M, Valaitis R, Bartholomew A, et al. Impact of internet-based interventions on caregiver mental health: systematic review and meta-analysis. J Med Internet Res. Jul 03, 2018;20(7):e10668. [ FREE Full text ] [ CrossRef ] [ Medline ]
Babbage CM, Jackson GM, Davies EB, Nixon E. Self-help digital interventions targeted at improving psychological well-being in young people with perceived or clinically diagnosed reduced well-being: systematic review. JMIR Ment Health. Aug 26, 2022;9(8):e25716. [ FREE Full text ] [ CrossRef ] [ Medline ]
Richardson T, Stallard P, Velleman S. Computerised cognitive behavioural therapy for the prevention and treatment of depression and anxiety in children and adolescents: a systematic review. Clin Child Fam Psychol Rev. Sep 2010;13(3):275-290. [ CrossRef ] [ Medline ]
Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J Anxiety Disord. Apr 2018;55:70-78. [ FREE Full text ] [ CrossRef ] [ Medline ]
De Witte NA, Joris S, Van Assche E, Van Daele T. Technological and digital interventions for mental health and wellbeing: an overview of systematic reviews. Front Digit Health. Dec 23, 2021;3:754337. [ FREE Full text ] [ CrossRef ] [ Medline ]
Shaffer KM, Tigershtrom A, Badr H, Benvengo S, Hernandez M, Ritterband LM. Dyadic psychosocial eHealth interventions: systematic scoping review. J Med Internet Res. Mar 04, 2020;22(3):e15509. [ FREE Full text ] [ CrossRef ] [ Medline ]
Knopp K, Schnitzer JS, Khalifian C, Grubbs K, Morland LA, Depp C. Digital interventions for couples: state of the field and future directions. Couple Family Psychol Res Pract. Dec 2023;12(4):201-217. [ FREE Full text ] [ CrossRef ]
Nwosu A, Boardman S, Husain MM, Doraiswamy PM. Digital therapeutics for mental health: is attrition the Achilles heel? Front Psychiatry. 2022;13:900615. [ FREE Full text ] [ CrossRef ] [ Medline ]
Boardman S. Creating wellness apps with high patient engagement to close the intention-action gap. Int Psychogeriatr. Jun 2021;33(6):551-552. [ CrossRef ] [ Medline ]
Lattie EG, Stiles-Shields C, Graham AK. An overview of and recommendations for more accessible digital mental health services. Nat Rev Psychol. Mar 26, 2022;1(2):87-100. [ FREE Full text ] [ CrossRef ] [ Medline ]
Patel S, Akhtar A, Malins S, Wright N, Rowley E, Young E, et al. The acceptability and usability of digital health interventions for adults with depression, anxiety, and Somatoform disorders: qualitative systematic review and meta-synthesis. J Med Internet Res. Jul 06, 2020;22(7):e16228. [ FREE Full text ] [ CrossRef ] [ Medline ]
Usability 101: introduction to usability. Nielsen Norman Group. 2012. URL: https://www.nngroup.com/articles/usability-101-introduction-to-usability/ [accessed 2023-04-23]
Falloon IR. Family interventions for mental disorders: efficacy and effectiveness. World Psychiatry. Mar 2003;2(1):20-28. [ FREE Full text ] [ Medline ]
Ng MM, Firth J, Minen M, Torous J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv. Jul 01, 2019;70(7):538-544. [ FREE Full text ] [ CrossRef ] [ Medline ]
Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. Jun 13, 2017;7(2):254-267. [ FREE Full text ] [ CrossRef ] [ Medline ]
Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. Oct 02, 2018;169(7):467-473. [ FREE Full text ] [ CrossRef ] [ Medline ]
Peters MD, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. Oct 2020;18(10):2119-2126. [ CrossRef ] [ Medline ]
Endnote Team. Endnote. Clarivate. 2013. URL: https://clarivate.com/innovation-exchange/solution/endnote/ [accessed 2024-04-29]
Babineau J. Product review: Covidence (systematic review software). J Can Health Libr Assoc. Aug 01, 2014;35(2):68. [ CrossRef ]
Luo X, Li J, Cao Q, Sun L, Chen Y, Zhao J, et al. A feasibility study of an integrated couples-based supportive programme for Chinese couples living with colorectal cancer. Nurs Open. Mar 26, 2021;8(2):920-926. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wang Z, Chen M, Cao Q, Gong J, Zhao J, Lin C, et al. An integrated intervention programme for couples facing colorectal cancer: a randomized controlled trial. Clin Psychol Psychother (Forthcoming). Nov 10, 2023. [ CrossRef ] [ Medline ]
Santisteban DA, Czaja SJ, Nair SN, Mena MP, Tulloch AR. Computer informed and flexible family-based treatment for adolescents: a randomized clinical trial for at-risk racial/ethnic minority adolescents. Behav Ther. Jul 2017;48(4):474-489. [ CrossRef ] [ Medline ]
Price-Blackshear MA, Pratscher SD, Oyler DL, Armer JM, Cheng A, Cheng MX, et al. Online couples mindfulness-based intervention for young breast cancer survivors and their partners: a randomized-control trial. J Psychosoc Oncol. 2020;38(5):592-611. [ CrossRef ] [ Medline ]
McLellan LF, Woon S, Hudson JL, Lyneham HJ, Karin E, Rapee RM. Treating child anxiety using family-based internet delivered cognitive behavior therapy with brief therapist guidance: a randomized controlled trial. J Anxiety Disord. Jan 2024;101:102802. [ FREE Full text ] [ CrossRef ] [ Medline ]
Fitzpatrick S, Wagner AC, Crenshaw AO, Varma S, Whitfield KM, Valela R, et al. Initial outcomes of couple HOPES: a guided online couple intervention for PTSD and relationship enhancement. Internet Interv. Sep 2021;25:100423. [ FREE Full text ] [ CrossRef ] [ Medline ]
Monson CM, Wagner AC, Crenshaw AO, Whitfield KM, Newnham CM, Valela R, et al. An uncontrolled trial of couple HOPES: a guided online couple intervention for PTSD and relationship enhancement. J Fam Psychol. Sep 2022;36(6):1036-1042. [ CrossRef ] [ Medline ]
Crenshaw AO, Whitfield KM, Collins A, Valela R, Varma S, Landy MS, et al. Partner outcomes from an uncontrolled trial of couple HOPES: a guided online couple intervention for posttraumatic stress disorder and relationship enhancement. J Trauma Stress. Mar 2023;36(1):230-238. [ CrossRef ] [ Medline ]
Morland LA, Wachsman T, Webster K, Fitzpatrick S, Valela R, Crenshaw AO, et al. A pilot of couple HOPES within the U.S. veterans affairs healthcare system: PTSD and relationship outcomes in veteran couples. Psychol Serv (Forthcoming). Jun 19, 2023. [ CrossRef ] [ Medline ]
Fergus KD, McLeod D, Carter W, Warner E, Gardner SL, Granek L, et al. Development and pilot testing of an online intervention to support young couples' coping and adjustment to breast cancer. Eur J Cancer Care (Engl). Jul 2014;23(4):481-492. [ CrossRef ] [ Medline ]
Ianakieva I, Fergus K, Ahmad S, Pos A, Pereira A. A model of engagement promotion in a professionally facilitated online intervention for couples affected by breast cancer. J Marital Fam Ther. Oct 2016;42(4):701-715. [ CrossRef ] [ Medline ]
Ianakieva I, Fergus K, Ahmad S, Pereira A, Stephen J, McLeod D, et al. Varying levels and types of engagement in an online relationship enhancement program for couples following breast cancer. J Couple Relatsh Ther. Jan 22, 2019;18(1):22-43. [ FREE Full text ] [ CrossRef ]
Fergus K, Ahmad S, Ianakieva I, McLeod D, Carter W. Metaphor and meaning in an online creative expression exercise to promote dyadic coping in young couples affected by breast cancer. Arts Health. Aug 08, 2016;9(2):139-153. [ FREE Full text ] [ CrossRef ]
Fergus K, Ahmad S, Gardner S, Ianakieva I, McLeod D, Stephen J, et al. Couplelinks online intervention for young couples facing breast cancer: a randomised controlled trial. Psychooncology. Mar 2022;31(3):512-520. [ CrossRef ] [ Medline ]
Fergus K, Tanen A, Ahmad S, Gardner S, Warner E, McLeod D, et al. Treatment satisfaction with couplelinks online intervention to promote dyadic coping in young couples affected by breast cancer. Front Psychol. 2022;13:862555. [ FREE Full text ] [ CrossRef ] [ Medline ]
Canfield SM, Canada KE, Rolbiecki AJ, Petroski GF. Feasibility and acceptability of an online mental health intervention for pregnant women and their partners: a mixed method study with a pilot randomized control trial. BMC Pregnancy Childbirth. Oct 18, 2023;23(1):739. [ FREE Full text ] [ CrossRef ] [ Medline ]
Selby E, Allabyrne C, Keenan JR. Delivering clinical evidence-based child-parent interventions for emotional development through a digital platform: a feasibility trial. Clin Child Psychol Psychiatry. Oct 2021;26(4):1271-1283. [ FREE Full text ] [ CrossRef ] [ Medline ]
Braithwaite SR, Fincham FD. Computer-based dissemination: a randomized clinical trial of ePREP using the actor partner interdependence model. Behav Res Ther. Mar 2011;49(2):126-131. [ CrossRef ] [ Medline ]
Braithwaite SR, Fincham FD. Computer-based prevention of intimate partner violence in marriage. Behav Res Ther. Mar 2014;54:12-21. [ FREE Full text ] [ CrossRef ] [ Medline ]
Northouse L, Schafenacker A, Barr KL, Katapodi M, Yoon H, Brittain K, et al. A tailored Web-based psychoeducational intervention for cancer patients and their family caregivers. Cancer Nurs. 2014;37(5):321-330. [ FREE Full text ] [ CrossRef ] [ Medline ]
Vigerland S, Ljótsson B, Thulin U, Öst LG, Andersson G, Serlachius E. Internet-delivered cognitive behavioural therapy for children with anxiety disorders: a randomised controlled trial. Behav Res Ther. Jan 2016;76:47-56. [ FREE Full text ] [ CrossRef ] [ Medline ]
Feinberg ME, Boring J, Le Y, Hostetler ML, Karre J, Irvin J, et al. Supporting military family resilience at the transition to parenthood: a randomized pilot trial of an online version of family foundations. Family Relations. Dec 10, 2019;69(1):109-124. [ FREE Full text ] [ CrossRef ]
Gil M, Kim SS, Kim D, Kim S. Online coaching blended couple-oriented intervention for preventing depression among Korean middle adulthood: a feasibility study. Fam Process. Dec 2023;62(4):1478-1505. [ CrossRef ] [ Medline ]
Kahn JR, Collinge W, Soltysik R. Post-9/11 veterans and their partners improve mental health outcomes with a self-directed mobile and web-based wellness training program: a randomized controlled trial. J Med Internet Res. Sep 27, 2016;18(9):e255. [ FREE Full text ] [ CrossRef ] [ Medline ]
Schinke SP, Cole KC, Fang L. Gender-specific intervention to reduce underage drinking among early adolescent girls: a test of a computer-mediated, mother-daughter program. J Stud Alcohol Drugs. Jan 2009;70(1):70-77. [ FREE Full text ] [ CrossRef ] [ Medline ]
Schinke SP, Fang L, Cole KC. Preventing substance use among adolescent girls: 1-year outcomes of a computerized, mother-daughter program. Addict Behav. Dec 2009;34(12):1060-1064. [ FREE Full text ] [ CrossRef ] [ Medline ]
Schinke SP, Fang L, Cole KC. Computer-delivered, parent-involvement intervention to prevent substance use among adolescent girls. Prev Med. Nov 2009;49(5):429-435. [ FREE Full text ] [ CrossRef ] [ Medline ]
Fang L, Schinke SP, Cole KC. Preventing substance use among early Asian-American adolescent girls: initial evaluation of a web-based, mother-daughter program. J Adolesc Health. Nov 2010;47(5):529-532. [ FREE Full text ] [ CrossRef ] [ Medline ]
Fang L, Schinke SP. Two-year outcomes of a randomized, family-based substance use prevention trial for Asian American adolescent girls. Psychol Addict Behav. Sep 2013;27(3):788-798. [ FREE Full text ] [ CrossRef ] [ Medline ]
Schinke SP, Fang L, Cole KC, Cohen-Cutler S. Preventing substance use among black and Hispanic adolescent girls: results from a computer-delivered, mother-daughter intervention approach. Subst Use Misuse. 2011;46(1):35-45. [ FREE Full text ] [ CrossRef ] [ Medline ]
Schwinn TM, Schinke S, Fang L, Kandasamy S. A web-based, health promotion program for adolescent girls and their mothers who reside in public housing. Addict Behav. Apr 2014;39(4):757-760. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Wolfe CR, Pestian JP. A web-based family problem-solving intervention for families of children with traumatic brain injury. Behav Res Methods Instrum Comput. May 2004;36(2):261-269. [ CrossRef ] [ Medline ]
Wade SL, Wolfe C, Brown TM, Pestian JP. Putting the pieces together: preliminary efficacy of a web-based family intervention for children with traumatic brain injury. J Pediatr Psychol. 2005;30(5):437-442. [ CrossRef ] [ Medline ]
Wade SL, Carey J, Wolfe CR. An online family intervention to reduce parental distress following pediatric brain injury. J Consult Clin Psychol. Jun 2006;74(3):445-454. [ CrossRef ] [ Medline ]
Wade SL, Carey J, Wolfe CR. The efficacy of an online cognitive-behavioral family intervention in improving child behavior and social competence following pediatric brain injury. Rehabil Psychol. Aug 2006;51(3):179-189. [ FREE Full text ] [ CrossRef ]
Carey JC, Wade SL, Wolfe CR. Lessons learned: the effect of prior technology use on web-based interventions. Cyberpsychol Behav. Apr 2008;11(2):188-195. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Walz NC, Carey JC, Williams KM. Preliminary efficacy of a web-based family problem-solving treatment program for adolescents with traumatic brain injury. J Head Trauma Rehabil. 2008;23(6):369-377. [ CrossRef ] [ Medline ]
Wade SL, Walz NC, Carey JC, Williams KM. Brief report: description of feasibility and satisfaction findings from an innovative online family problem-solving intervention for adolescents following traumatic brain injury. J Pediatr Psychol. Jun 2009;34(5):517-522. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Walz NC, Carey J, Williams KM, Cass J, Herren L, et al. A randomized trial of teen online problem solving for improving executive function deficits following pediatric traumatic brain injury. J Head Trauma Rehabil. 2010;25(6):409-415. [ CrossRef ] [ Medline ]
Wade SL, Walz NC, Carey J, McMullen KM, Cass J, Mark E, et al. Effect on behavior problems of teen online problem-solving for adolescent traumatic brain injury. Pediatrics. Oct 2011;128(4):e947-e953. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Walz NC, Carey J, McMullen KM, Cass J, Mark E, et al. A randomized trial of teen online problem solving: efficacy in improving caregiver outcomes after brain injury. Health Psychol. Nov 2012;31(6):767-776. [ CrossRef ] [ Medline ]
Kurowski BG, Wade SL, Kirkwood MW, Brown TM, Stancin TM, Taylor HG. Online problem-solving therapy for executive dysfunction after child traumatic brain injury. Pediatrics. Jul 2013;132(1):e158-e166. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Karver CL, Taylor HG, Cassedy A, Stancin T, Kirkwood MW, et al. Counselor-assisted problem solving improves caregiver efficacy following adolescent brain injury. Rehabil Psychol. Mar 2014;59(1):1-9. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Stancin T, Kirkwood M, Brown TM, McMullen KM, Taylor HG. Counselor-assisted problem solving (CAPS) improves behavioral outcomes in older adolescents with complicated mild to severe TBI. J Head Trauma Rehabil. 2014;29(3):198-207. [ FREE Full text ] [ CrossRef ] [ Medline ]
Narad ME, Minich N, Taylor HG, Kirkwood MW, Brown TM, Stancin T, et al. Effects of a web-based intervention on family functioning following pediatric traumatic brain injury. J Dev Behav Pediatr. 2015;36(9):700-707. [ FREE Full text ] [ CrossRef ] [ Medline ]
Petranovich CL, Wade SL, Taylor HG, Cassedy A, Stancin T, Kirkwood MW, et al. Long-term caregiver mental health outcomes following a predominately online intervention for adolescents with complicated mild to severe traumatic brain injury. J Pediatr Psychol. Aug 2015;40(7):680-688. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Taylor HG, Cassedy A, Zhang N, Kirkwood MW, Brown TM, et al. Long-term behavioral outcomes after a randomized, clinical trial of counselor-assisted problem solving for adolescents with complicated mild-to-severe traumatic brain injury. J Neurotrauma. Jul 01, 2015;32(13):967-975. [ FREE Full text ] [ CrossRef ] [ Medline ]
Raj SP, Zhang N, Kirkwood MW, Taylor HG, Stancin T, Brown TM, et al. Online family problem solving for pediatric traumatic brain injury: influences of parental marital status and participation on adolescent outcomes. J Head Trauma Rehabil. 2018;33(3):158-166. [ FREE Full text ] [ CrossRef ] [ Medline ]
Wade SL, Cassedy AE, McNally KA, Kurowski BG, Kirkwood MW, Stancin T, et al. A randomized comparative effectiveness trial of family-problem-solving treatment for adolescent brain injury: parent outcomes from the coping with head injury through problem solving (CHIPS) study. J Head Trauma Rehabil. 2019;34(6):E1-E9. [ CrossRef ] [ Medline ]
Wade SL, Cassedy AE, Sklut M, Taylor HG, McNally KA, Kirkwood MW, et al. The relationship of adolescent and parent preferences for treatment modality with satisfaction, attrition, adherence, and efficacy: the coping with head injury through problem-solving (CHIPS) study. J Pediatr Psychol. Apr 01, 2019;44(3):388-401. [ CrossRef ] [ Medline ]
Wade SL, Cassedy AE, Taylor HG, McNally KA, Kirkwood MW, Stancin T, et al. Adolescent quality of life following family problem-solving treatment for brain injury. J Consult Clin Psychol. Nov 2019;87(11):1043-1055. [ CrossRef ] [ Medline ]
Wade SL, Taylor HG, Yeates KO, Kirkwood M, Zang H, McNally K, et al. Online problem solving for adolescent brain injury: a randomized trial of 2 approaches. J Dev Behav Pediatr. 2018;39(2):154-162. [ CrossRef ] [ Medline ]
Narad ME, Raj S, Yeates KO, Taylor HG, Kirkwood MW, Stancin T, et al. Randomized controlled trial of an online problem-solving intervention following adolescent traumatic brain injury: family outcomes. Arch Phys Med Rehabil. May 2019;100(5):811-820. [ CrossRef ] [ Medline ]
Doss BD, Cicila LN, Georgia EJ, Roddy MK, Nowlan KM, Benson LA, et al. A randomized controlled trial of the web-based OurRelationship program: effects on relationship and individual functioning. J Consult Clin Psychol. Apr 2016;84(4):285-296. [ FREE Full text ] [ CrossRef ] [ Medline ]
Roddy MK, Georgia EJ, Doss BD. Couples with intimate partner violence seeking relationship help: associations and implications for self-help and online interventions. Fam Process. Jun 2018;57(2):293-307. [ CrossRef ] [ Medline ]
Georgia Salivar EJ, Roddy MK, Nowlan KM, Doss BD. Effectiveness of the online OurRelationship program for underserved couples. Couple Family Psychol Res Pract. Sep 2018;7(3-4):212-226. [ FREE Full text ] [ CrossRef ]
Doss BD, Roddy MK, Nowlan KM, Rothman K, Christensen A. Maintenance of gains in relationship and individual functioning following the online OurRelationship program. Behav Ther. Jan 2019;50(1):73-86. [ FREE Full text ] [ CrossRef ] [ Medline ]
Doss BD, Roddy MK, Llabre MM, Georgia Salivar E, Jensen-Doss A. Improvements in coparenting conflict and child adjustment following an online program for relationship distress. J Fam Psychol. Mar 2020;34(1):68-78. [ FREE Full text ] [ CrossRef ] [ Medline ]
Roddy MK, Stamatis CA, Rothman K, Doss BD. Mechanisms of change in a brief, online relationship intervention. J Fam Psychol. Mar 2020;34(1):57-67. [ FREE Full text ] [ CrossRef ] [ Medline ]
Biesen J, Roddy MK, Doss BD. The role of five-factor model personality traits in a web-based relationship improvement program. Couple Family Psychol Res Pract. Apr 04, 2022. [ FREE Full text ] [ CrossRef ]
Roddy MK, Rothman K, Doss BD. A randomized controlled trial of different levels of coach support in an online intervention for relationship distress. Behav Res Ther. Nov 2018;110:47-54. [ CrossRef ] [ Medline ]
Rothman K, Roddy MK, Doss BD. Completion of a stand‐alone versus coach‐supported trial of a web‐based program for distressed relationships. Fam Relat. Aug 07, 2019;68(4):375-389. [ FREE Full text ] [ CrossRef ]
Roddy MK, Nowlan KM, Doss BD. A randomized controlled trial of coach contact during a brief online intervention for distressed couples. Fam Process. Dec 2017;56(4):835-851. [ CrossRef ] [ Medline ]
Knopp K, Rashkovsky K, Khalifian CE, Grubbs KM, Doss BD, Depp CA, et al. Pilot open trial of the OurRelationship online couples’ program in a Veterans Affairs Medical Center. Couple Fam Psychol Res Pract. Mar 2022;11(1):33-41. [ FREE Full text ] [ CrossRef ]
Le Y, Hatch SG, Goodman ZT, Doss BD. Does coparenting improve during the OurRelationship program? Explorations within a low-income sample during the COVID-19 pandemic. J Fam Psychol. Sep 2022;36(6):1030-1035. [ FREE Full text ] [ CrossRef ] [ Medline ]
Hatch SG, Goodman ZT, Hatch HD, Le Y, Guttman S, Doss BD. Web-based relationship education and pornography-related behaviors: a single-group design during the COVID-19 pandemic. Arch Sex Behav. May 04, 2023;52(4):1841-1853. [ FREE Full text ] [ CrossRef ] [ Medline ]
Le Y, Roddy MK, Rothman K, Salivar EG, Guttman S, Doss BD. A randomized controlled trial of the online OurRelationship program with varying levels of coach support. Internet Interv. Dec 2023;34:100661. [ FREE Full text ] [ CrossRef ] [ Medline ]
Doss BD, Knopp K, Roddy MK, Rothman K, Hatch SG, Rhoades GK. Online programs improve relationship functioning for distressed low-income couples: results from a nationwide randomized controlled trial. J Consult Clin Psychol. Apr 2020;88(4):283-294. [ CrossRef ] [ Medline ]
Le Y, Roddy MK, Hatch SG, Doss BD. Mechanisms of improvements and maintenance in online relationship programs for distressed low-income couples. J Consult Clin Psychol. Dec 2020;88(12):1091-1104. [ CrossRef ] [ Medline ]
Roddy MK, Rhoades GK, Doss BD. Effects of ePREP and OurRelationship on low-income couples' mental health and health behaviors: a randomized controlled trial. Prev Sci. Aug 2020;21(6):861-871. [ CrossRef ] [ Medline ]
Le Y, O'Reilly Treter M, Roddy MK, Doss BD. Coparenting and parenting outcomes of online relationship interventions for low-income couples. J Fam Psychol. Oct 2021;35(7):1033-1039. [ CrossRef ] [ Medline ]
Roddy MK, Knopp K, Georgia Salivar E, Doss BD. Maintenance of relationship and individual functioning gains following online relationship programs for low-income couples. Fam Process. Mar 30, 2021;60(1):102-118. [ FREE Full text ] [ CrossRef ] [ Medline ]
Georgia Salivar E, Knopp K, Roddy MK, Morland LA, Doss BD. Effectiveness of online OurRelationship and ePREP programs for low-income military couples. J Consult Clin Psychol. Oct 2020;88(10):899-906. [ CrossRef ] [ Medline ]
Hatch SG, Knopp K, Le Y, Allen MO, Rothman K, Rhoades GK, et al. Online relationship education for help-seeking low-income couples: a bayesian replication and extension of the OurRelationship and ePREP programs. Fam Process. Sep 2022;61(3):1045-1061. [ CrossRef ] [ Medline ]
Mitchell EA, Le Y, Hatch SG, Guttman S, Doss BD. Effects of online relationship programs for low-income couples during the perinatal period. Behav Res Ther. Aug 2023;167:104337. [ CrossRef ] [ Medline ]
Le Y, Xia M, Roddy MK, Hatch SG, Doss BD. Profiles of low-income help-seeking couples and implications for intervention gains: a couple-centered approach. Behav Ther. May 2024;55(3):443-456. [ CrossRef ] [ Medline ]
Barton AW, Gong Q, Guttman S, Doss BD. Trajectories of perceived gratitude and change following relationship interventions: a randomized controlled trial with lower-income, help-seeking couples. Behav Ther. Mar 2024;55(2):401-411. [ CrossRef ] [ Medline ]
Rizzo CJ, Houck C, Barker D, Collibee C, Hood E, Bala K. Project STRONG: an online, parent-son intervention for the prevention of dating violence among early adolescent boys. Prev Sci. Mar 2021;22(2):193-204. [ FREE Full text ] [ CrossRef ] [ Medline ]
Whittingham K, Sheffield J, Mak C, Wright A, Boyd RN. Parenting acceptance and commitment therapy: an RCT of an online course with families of children with CP. Behav Res Ther. Aug 2022;155:104129. [ CrossRef ] [ Medline ]
Piotrowska PJ, Tully LA, Collins DA, Sawrikar V, Hawes D, Kimonis ER, et al. ParentWorks: evaluation of an online, father-inclusive, universal parenting intervention to reduce child conduct problems. Child Psychiatry Hum Dev. Aug 24, 2020;51(4):503-513. [ FREE Full text ] [ CrossRef ] [ Medline ]
Dadds MR, Sicouri G, Piotrowska PJ, Collins DA, Hawes DJ, Moul C, et al. Keeping parents involved: predicting attrition in a self-directed, online program for childhood conduct problems. J Clin Child Adolesc Psychol. 2019;48(6):881-893. [ CrossRef ] [ Medline ]
Song L, Rini C, Deal AM, Nielsen ME, Chang H, Kinneer P, et al. Improving couples' quality of life through a web-based prostate cancer education intervention. Oncol Nurs Forum. Mar 2015;42(2):183-192. [ FREE Full text ] [ CrossRef ] [ Medline ]
Rhudy LM, Hines EA, Farr EM, Esterov D, Chesak SS. Feasibility and acceptability of the resilient living program among persons with stroke or brain tumor and their family caregivers. NeuroRehabilitation. 2023;52(1):123-135. [ CrossRef ] [ Medline ]
Asazawa K, Jitsuzaki M, Mori A, Ichikawa T. Effectiveness of a web-based partnership support program for preventing decline in the quality of life of male patients undergoing infertility treatment: a quasi-experimental study. Jpn J Nurs Sci. Jul 2023;20(3):e12536. [ CrossRef ] [ Medline ]
Loew B, Rhoades G, Markman H, Stanley S, Pacifici C, White L, et al. Internet delivery of PREP-based relationship education for at-risk couples. J Couple Relatsh Ther. Oct 2012;11(4):291-309. [ FREE Full text ] [ CrossRef ]
Nguyen TD, Attkisson CC, Stegner BL. Assessment of patient satisfaction: development and refinement of a service evaluation questionnaire. Eval Program Plann. 1983;6(3-4):299-313. [ CrossRef ] [ Medline ]
Attkisson CC, Zwick R. The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plann. 1982;5(3):233-237. [ CrossRef ] [ Medline ]
Rotondi AJ, Sinkule J, Spring M. An interactive web-based intervention for persons with TBI and their families: use and evaluation by female significant others. J Head Trauma Rehabil. 2005;20(2):173-185. [ CrossRef ] [ Medline ]
Hoyt MF, Young J, Rycroft P. Single session thinking 2020. Aust N Z J Fam Ther. Oct 07, 2020;41(3):218-230. [ FREE Full text ] [ CrossRef ]
Hartley E, Moore L, Knuckey A, von Doussa H, Painter F, Story K, et al. Walk-in together: a pilot study of a walk-in online family therapy intervention. Aust N Z J Fam Ther. May 26, 2023;44(2):127-144. [ CrossRef ]
Osborn TL, Rodriguez M, Wasil AR, Venturo-Conerly KE, Gan J, Alemu RG, et al. Single-session digital intervention for adolescent depression, anxiety, and well-being: outcomes of a randomized controlled trial with Kenyan adolescents. J Consult Clin Psychol. Jul 2020;88(7):657-668. [ CrossRef ] [ Medline ]
Wasil AR, Park SJ, Gillespie S, Shingleton R, Shinde S, Natu S, et al. Harnessing single-session interventions to improve adolescent mental health and well-being in India: development, adaptation, and pilot testing of online single-session interventions in Indian secondary schools. Asian J Psychiatr. May 2020;50:101980. [ CrossRef ] [ Medline ]
Wasil AR, Taylor ME, Franzen RE, Steinberg JS, DeRubeis RJ. Promoting graduate student mental health during COVID-19: acceptability, feasibility, and perceived utility of an online single-session intervention. Front Psychol. 2021;12:569785. [ FREE Full text ] [ CrossRef ] [ Medline ]
Saleem M, Kühne L, De Santis KK, Christianson L, Brand T, Busse H. Understanding engagement strategies in digital interventions for mental health promotion: scoping review. JMIR Ment Health. Dec 20, 2021;8(12):e30000. [ FREE Full text ] [ CrossRef ] [ Medline ]
Musiat P, Hoffmann L, Schmidt U. Personalised computerised feedback in E-mental health. J Ment Health. Aug 2012;21(4):346-354. [ CrossRef ] [ Medline ]
Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res. Mar 24, 2021;23(3):e24387. [ FREE Full text ] [ CrossRef ] [ Medline ]
Alkhaldi G, Hamilton FL, Lau R, Webster R, Michie S, Murray E. The effectiveness of prompts to promote engagement with digital interventions: a systematic review. J Med Internet Res. Jan 08, 2016;18(1):e6. [ FREE Full text ] [ CrossRef ] [ Medline ]
Eysenbach G. The law of attrition. J Med Internet Res. Mar 31, 2005;7(1):e11. [ FREE Full text ] [ CrossRef ] [ Medline ]
Murray E, White IR, Varagunam M, Godfrey C, Khadjesari Z, McCambridge J. Attrition revisited: adherence and retention in a web-based alcohol trial. J Med Internet Res. Aug 30, 2013;15(8):e162. [ FREE Full text ] [ CrossRef ] [ Medline ]
Szinay D, Jones A, Chadborn T, Brown J, Naughton F. Influences on the uptake of and engagement with health and well-being smartphone apps: systematic review. J Med Internet Res. May 29, 2020;22(5):e17572. [ FREE Full text ] [ CrossRef ] [ Medline ]
Garett R, Chiu J, Zhang L, Young SD. A literature review: website design and user engagement. Online J Commun Media Technol. Jul 2016;6(3):1-14. [ FREE Full text ] [ Medline ]
Shim M, Mahaffey B, Bleidistel M, Gonzalez A. A scoping review of human-support factors in the context of internet-based psychological interventions (IPIs) for depression and anxiety disorders. Clin Psychol Rev. Dec 2017;57:129-140. [ CrossRef ] [ Medline ]
Inal Y, Wake JD, Guribye F, Nordgreen T. Usability evaluations of mobile mental health technologies: systematic review. J Med Internet Res. Jan 06, 2020;22(1):e15337. [ FREE Full text ] [ CrossRef ] [ Medline ]
Collins DA, Tully LA, Piotrowska PJ, Hawes DJ, Dadds MR. Perspectives on ParentWorks: learnings from the development and national roll-out of a self-directed online parenting intervention. Internet Interv. Mar 2019;15:52-59. [ FREE Full text ] [ CrossRef ] [ Medline ]
Corrigan PW, Miller FE. Shame, blame, and contamination: a review of the impact of mental illness stigma on family members. J Ment Health. Jul 06, 2009;13(6):537-548. [ FREE Full text ] [ CrossRef ]
Ennis E, Bunting BP. Family burden, family health and personal mental health. BMC Public Health. Mar 21, 2013;13(1):255. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

digital mental health intervention

low- and middle-income countries

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews

Edited by A Mavragani; submitted 29.05.23; peer-reviewed by M Ali, M Loades, F Tong; comments to author 01.12.23; revised version received 13.12.23; accepted 04.05.24; published 03.07.24.

©Ellen T Welsh, Jennifer E McIntosh, An Vuong, Zoe C G Cloud, Eliza Hartley, James H Boyd. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 03.07.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

The Role of Fintech in Promoting Financial Inclusion to Achieve Sustainable Development: An Integrated Bibliometric Analysis and Systematic Literature Review

Published: 03 July 2024

Cite this article

Kriti Kishor ORCID: orcid.org/0009-0006-2808-1633 1 ,
Sanjeev K. Bansal ORCID: orcid.org/0009-0009-8585-0429 1 &
Roshan Kumar 2

Fintech’s ability to enhance efficiency and reduce costs in financial services can promote greater financial inclusion (FI), which in turn serves as a foundation for sustainable and equitable development. Due to the dearth of thorough summaries in the body of existing literature, this systematic review and bibliometric analysis aim to present quantitative and qualitative information about the comprehensive relationship between fintech, FI, and sustainability development in an organised way. The review includes 189 publications from peer-reviewed journals of Scopus and Web of Science (WoS) databases up to 2023. The article was compiled based on the Scientific Procedures and Rationales for Systematic Literature Reviews (SPAR‐4‐SLR) protocol and the theory-context-characteristics-methodology (TCCM) framework. Bibliometric analysis has identified the leading journals, authors, nations, articles, and themes. A conceptual model has been designed to illustrate the entire scope, following which potential study areas have been proposed. This study aims to provide academic researchers, policymakers, and regulators with a detailed understanding of the relationship between fintech, financial inclusion, and sustainable development. The analysis demonstrates that FI is an essential requirement of our society and a vital pathway to achieve sustainable development. In the content analysis, we identify an integrative framework of four variables on this nexus. We found a very few conceptual, qualitative, and mixed method papers on this interaction, which provide potential avenues for further research. We recommend that scholars consider adopting a multi-theory perspective. We propose a comprehensive framework on this nexus. It will also pinpoint specific areas that require further investigation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

Get 10 units per month
Download Article/Chapter or Ebook
1 Unit = 1 Article or 1 Chapter
Cancel anytime

Price excludes VAT (USA) Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data Availability

The data that supports the findings of this study is available on request.

Adegbite, O. O., & Machethe, C. L. (2020). Bridging the financial inclusion gender gap in smallholder agriculture in Nigeria: An untapped potential for sustainable development. World Development, 127 , 104755.

Article Google Scholar

Afjal, M. (2023). Bridging the financial divide: A bibliometric analysis on the role of digital financial services within FinTech in enhancing financial inclusion and economic development. Humanities and Social Sciences Communications, 10 (1), 1–27.

Ahelegbey, D., Giudici, P., & Pediroda, V. (2023). A network based fintech inclusion platform. Socio-Economic Planning Sciences, 87 , 101555.

Akolgo, I. A. (2023). On the contradictions of Africa’s fintech boom: evidence from Ghana. Review of International Political Economy , 30 (5), 1639–1659. https://doi.org/10.1080/09692290.2023.2225142

Alshater, M. M., Saba, I., Supriani, I., & Rabbani, M. R. (2022). Fintech in Islamic finance literature: A review. Heliyon , 8(9). https://doi.org/10.1016/j.heliyon.2022.e10385

Ameen, N., Sharma, G. D., Tarba, S., Rao, A., & Chopra, R. (2022). Toward advancing theory on creativity in marketing and artificial intelligence. Psychology & Marketing, 39 (9), 1802–1825.

Arner, D. W., Buckley, R. P., Zetzsche, D. A., & Veidt, R. (2020). Sustainability, FinTech and financial inclusion. European Business Organization Law Review, 21 , 7–35.

Banna, H., Mia, M. A., Nourani, M., & Yarovaya, L. (2022). Fintech-based financial inclusion and risk-taking of microfinance institutions (MFIs): Evidence from Sub-Saharan Africa. Finance Research Letters, 45 , 102149.

Ben Slimane, S., Coeurderoy, R., & Mhenni, H. (2022). Digital transformation of small and medium enterprises: A systematic literature review and an integrative framework. International Studies of Management & Organization, 52 (2), 96–120.

Bhatt, A., Joshipura, M., & Joshipura, N. (2022). Decoding the trinity of Fintech, digitalization and financial services: An integrated bibliometric analysis and thematic literature review approach. Cogent Economics & Finance, 10 (1), 2114160.

Carè, R., Boitan, I. A., & Fatima, R. (2023). How do FinTech companies contribute to the achievement of SDGs? Insights from case studies. Research in International Business and Finance, 66 , 102072.

Chinoda, T., & Mashamba, T. (2021). Fintech, financial inclusion and income inequality nexus in Africa. Cogent Economics & Finance, 9 (1), 1986926. https://doi.org/10.1080/23322039.2021.1986926

Chowdhury, E. K., & Chowdhury, R. (2023). Role of financial inclusion in human development: Evidence from Bangladesh, India and Pakistan. Journal of the Knowledge Economy , 1–26. https://doi.org/10.1007/s13132-023-01366-x

Chung, S., Kim, K., Lee, C. H., & Oh, W. (2023). Interdependence between online peer-to-peer lending and cryptocurrency markets and its effects on financial inclusion. Production and Operations Management, 32 (6), 1939–1957.

Coffie, C. P. K., & Hongjiang, Z. (2023). FinTech market development and financial inclusion in Ghana: The role of heterogeneous actors. Technological Forecasting and Social Change, 186 , 122127.

Comerio, N., & Strozzi, F. (2019). Tourism and its economic impact: A literature review using bibliometric tools. Tourism Economics, 25 (1), 109–131. https://doi.org/10.1177/1354816618793762

Danladi, S., Prasad, M. S. V., Modibbo, U. M., Ahmadi, S. A., & Ghasemi, P. (2023). Attaining Sustainable Development Goals through financial inclusion: Exploring collaborative approaches to fintech adoption in developing economies. Sustainability, 15 (17), 13039.

David-West, O., Iheanachor, N., & Umukoro, I. (2020). Sustainable business models for the creation of mobile financial services in Nigeria. Journal of Innovation & Knowledge, 5 (2), 105–116.

Di Vaio, A., Hassan, R., & Palladino, R. (2023). Blockchain technology and gender equality: A systematic literature review. International Journal of Information Management, 68 , 102517.

Dong, Y., Chung, M., Zhou, C., & Venkataraman, S. (2018). Banking on “mobile money”: The implications of mobile money services on the value chain. Manufacturing & Service Operations Management , 21(2). https://doi.org/10.1287/msom.2018.0717

Ellili, N. O. D. (2023). Is there any association between FinTech and sustainability? Evidence from bibliometric review and content analysis. Journal of Financial Services Marketing, 28 (4), 748–762.

Fu, J., & Mishra, M. (2022). Fintech in the time of COVID− 19: Technological adoption during crises. Journal of Financial Intermediation, 50 , 100945.

Gálvez-Sánchez, F. J., Lara-Rubio, J., Verdú-Jóver, A. J., & Meseguer-Sánchez, V. (2021). Research advances on financial inclusion: A bibliometric analysis. Sustainability, 13 (6), 3156.

GPFI. (2010). G20 Principles for innovative financial inclusion - executive brief. Accessed 18 November 2023. Available at: http://www.gpfi.org/publications/g20-principles-innovative-financial-inclusion-executive-brief

Gulati, A., & Singh, S. (2024). Financial self-efficacy of consumers: A review and research agenda. International Journal of Consumer Studies, 48 (2), e13024.

Gupta, S., Yun, H., Xu, H., & Kim, H. W. (2017). An exploratory study on mobile banking adoption in Indian metropolitan and urban areas: A scenario-based experiment. Information Technology for Development, 23 (1), 127–152.

Han, H., & Gu, X. (2021). Linkage between inclusive digital finance and high-tech enterprise innovation performance: Role of debt and equity financing. Frontiers in Psychology, 12 , 814408.

Hasan, M., Le, T., & Hoque, A. (2021). How does financial literacy impact on inclusive finance? Financial Innovation, 7 (1), 1–23.

Hasan, M., Noor, T., Gao, J., Usman, M., & Abedin, M. Z. (2023). Rural consumers’ financial literacy and access to FinTech services. Journal of the Knowledge Economy, 14 (2), 780–804.

Hashemizadeh, A., Ashraf, R. U., Khan, I., & Zaidi, S. A. H. (2023). Digital financial inclusion, environmental quality, and economic development: the contributions of financial development and investments in OECD countries. Environmental Science and Pollution Research, 30 (54), 116336–116347. https://doi.org/10.1007/s11356-023-30275-4

Hulland, J., & Houston, M. B. (2020). Why systematic review papers and meta-analyses matter: An introduction to the special issue on generalizations in marketing. Journal of the Academy of Marketing Science, 48 , 351–359. https://doi.org/10.1007/s11747-020-00721-7

Hwa, G. (2019). Global FinTech Adoption Index 2019. Accessed on 1 October 2023. Available at https://assets.ey.com/content/dam/ey-sites/ey-com/en_gl/topics/banking-and-capital-markets/ey-global-fintech-adoption-index.pdf

Iheanachor, N., David-West, Y., & Umukoro, I. O. (2021). Business model innovation at the bottom of the pyramid–A case of mobile money agents. Journal of Business Research, 127 , 96–107.

Karim, Z. A., Nizam, R., Law, S. H., & Hassan, M. K. (2022). Does financial inclusiveness affect economic growth? New evidence using a dynamic panel threshold regression. Finance Research Letters, 46 , 102364.

Kemal, A. A. (2019). Mobile banking in the government-to-person payment sector for financial inclusion in Pakistan. Information Technology for Development, 25 (3), 475–502.

Khando, K., Islam, M. S., & Gao, S. (2022). The emerging technologies of digital payments and associated challenges: A systematic literature review. Future Internet, 15 (1), 21.

Kim, M., Zoo, H., Lee, H., & Kang, J. (2018). Mobile financial services, financial inclusion, and development: A systematic review of academic literature. The Electronic Journal of Information Systems in Developing Countries, 84 (5), e12044.

Koomson, I., Martey, E., & Etwire, P. M. (2023). Mobile money and entrepreneurship in East Africa: The mediating roles of digital savings and access to digital credit. Information Technology & People, 36 (3), 996–1019.

Lagna, A., & Ravishankar, M. N. (2022). Making the world a better place with fintech research. Information Systems Journal, 32 (1), 61–102.

Latif, N., Safdar, N., Liaquat, M., Younas, K., Nazeer, N., & Rafeeq, R. (2023). The role of institutional quality in assessing the environmental externality of financial inclusion: A DCCE approach. Frontiers in Environmental Science, 11 , 65.

Lee, C. C., Lou, R., & Wang, F. (2023). Digital financial inclusion and poverty alleviation: Evidence from the sustainable development of China. Economic Analysis and Policy, 77 , 418–434.

Li, J., Wei, R., & Guo, Y. (2022). How can the financing constraints of SMEs be eased in China?-Effect analysis, heterogeneity test and mechanism identification based on digital inclusive finance. Frontiers in Environmental Science, 10 , 949164.

Lim, W. M., Yap, S.-F., & Makkar, M. (2021). Home sharing in marketing and tourism at a tipping point: What do we know, how do we know, and where should we be heading? Journal of Business Research, 122 (September 2020), 534–566. https://doi.org/10.1016/j.jbusres.2020.08.051

Liu, X., Zhan, F. B., Hong, S., Niu, B., & Liu, Y. (2012). A bibliometric study of earthquake research: 1900–2010. Scientometrics, 92 (3), 747–765. https://doi.org/10.1007/s11192-011-0599-z

Liu, S., Gao, L., Latif, K., Dar, A. A., Zia-UR-Rehman, M., & Baig, S. A. (2021). The behavioral role of digital economy adaptation in sustainable financial literacy and financial inclusion. Frontiers in Psychology, 12 , 742118.

Liu, A., Urquía-Grande, E., López-Sánchez, P., & Rodríguez-López, Á. (2023). Research into microfinance and ICTs: A bibliometric analysis. Evaluation and Program Planning, 97 , 102215.

Louman, B., Girolami, E. D., Shames, S., Primo, L. G., Gitz, V., Scherr, S. J., & Brady, M. (2022). Access to landscape finance for small-scale producers and local communities: A literature review. Land, 11 (9), 1444.

Mapanje, O., Karuaihe, S., Machethe, C., & Amis, M. (2023). Financing sustainable agriculture in sub-Saharan Africa: A review of the role of financial technologies. Sustainability, 15 (5), 4587.

Michael, B., Koroleska, N., Tai, A., & Wong, D. W. H. (2022). A critical look at using financial technology policy to promote the sustainable development goals. Sustainable Development, 30 (6), 1911–1920.

Mishra, V., & Bisht, S. S. (2013). Mobile banking in a developing economy: A customer-centric model for policy formulation. Telecommunications Policy, 37 (6–7), 503–514.

Morgan, P. J. (2022). Fintech and financial inclusion in Southeast Asia and India. Asian Economic Policy Review, 17 (2), 183–208.

Mpofu, F. Y. (2022). Industry 4.0 in financial services: Mobile money taxes, revenue mobilisation, financial inclusion, and the realisation of sustainable development goals (SDGs) in Africa. Sustainability, 14 (14), 8667.

N’dri, L. M., & Kakinaka, M. (2020). Financial inclusion, mobile money, and individual welfare: The case of Burkina Faso. Telecommunications Policy, 44 (3), 101926.

Niankara, I. (2023). The impact of financial inclusion on digital payment solution uptake within the Gulf Cooperation Council Economies. International Journal of Innovation Studies, 7 (1), 1–17.

Ozili, P. K. (2018). Impact of digital finance on financial inclusion and stability. Borsa Istanbul Review, 18 (4), 329–340.

Pal, A., De’, R., & Herath, T. (2020). The role of mobile payment technology in sustainable and human-centric development: Evidence from the post-demonetization period in India. Information Systems Frontiers, 22 , 607–631.

Paul, J., & Barari, M. (2022). Meta-analysis and traditional systematic literature reviews—What, why, when, where, and how? Psychology & Marketing, 39 (6), 1099–1115. https://doi.org/10.1002/mar.21657

Paul, J., & Criado, A. R. (2020). The art of writing literature review: What do we know and what do we need to know? International Business Review, 29 (4), 101717. https://doi.org/10.1016/j.ibusrev.2020.101717

Paul, J., & Rosado-Serrano, A. (2019). Gradual Internationalization vs Born-Global/International new venture models: A review and research agenda. International Marketing Review, 36 (6), 830–858. https://doi.org/10.1108/IMR-10-2018-0280

Paul, J., Lim, W. M., O’Cass, A., Hao, A. W., & Bresciani, S. (2021a). Scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR). International Journal of Consumer Studies, 45 (45), 1–16. https://doi.org/10.1111/ijcs.12695

Paul, J., Merchant, A., Dwivedi, Y. K., & Rose, G. (2021b). Writing an impactful review article: What do we know and what do we need to know? Journal of Business Research, 133 , 337–340. https://doi.org/10.1016/j.jbusres.2021.05.005

Paul, J., Khatri, P., & Kaur Duggal, H. (2023). Frameworks for developing impactful systematic literature reviews and theory building: What, Why and How?. Journal of Decision Systems , 1–14. https://doi.org/10.1080/12460125.2023.2197700

Pittaway, L., Holt, R., & Broad, J. (2014). Synthesising knowledge in entrepreneurship research-The role of systematic literature reviews. In Handbook of research on small business and entrepreneurship (pp. 83–105). Edward Elgar Publishing. https://doi.org/10.4337/9781849809245.00014

Pradhan, R. P., Arvin, M. B., Nair, M. S., Hall, J. H., & Bennett, S. E. (2021). Sustainable economic development in India: The dynamics between financial inclusion, ICT development, and economic growth. Technological Forecasting and Social Change, 169 , 120758.

Puschmann, T. (2017). Fintech. Business & Information Systems Engineering, 59 , 69–76. https://doi.org/10.1007/s12599-017-0464-6

Raksmey, U., Lin, C. Y., & Kakinaka, M. (2022). Macroprudential regulation and financial inclusion: Any difference between developed and developing countries? Research in International Business and Finance, 63 , 101759.

Rohman, P. S., Fianto, B. A., Shah, S. A. A., Kayani, U. N., Suprayogi, N., & Supriani, I. (2021). A review on literature of Islamic microfinance from 2010–2020: Lesson for practitioners and future directions. Heliyon, 7 (12). https://doi.org/10.1016/j.heliyon.2021.e08549

Roy, P., & Patro, B. (2022). Financial inclusion of women and gender gap in access to finance: A systematic literature review. Vision, 26 (3), 282–299.

Sahabuddin, M., Sakib, M. N., Rahman, M. M., Jibir, A., Fahlevi, M., Aljuaid, M., & Grabowska, S. (2023). The evolution of FinTech in scientific research: A bibliometric analysis. Sustainability, 15 (9), 7176.

Schilling, L., & Seuring, S. (2023). Mobile financial service-enabled micro-businesses driving sustainable value creation in emerging markets. Technological Forecasting and Social Change, 192 , 122596.

Senyo, P. K., & Osabutey, E. L. (2020). Unearthing antecedents to financial inclusion through FinTech innovations. Technovation, 98 , 102155.

Senyo, P. K., Karanasios, S., Gozman, D., & Baba, M. (2022). FinTech ecosystem practices shaping financial inclusion: The case of mobile money in Ghana. European Journal of Information Systems, 31 (1), 112–127.

Setiawan, B., Phan, T. D., Medina, J., Wieriks, M., Nathan, R. J., & Fekete-Farkas, M. (2023). Quest for financial inclusion via digital financial services (Fintech) during COVID-19 pandemic: Case study of women in Indonesia. Journal of Financial Services Marketing , 1–15. https://doi.org/10.1057/s41264-023-00217-9

Shaikh, A. A., Glavee-Geo, R., Karjaluoto, H., & Hinson, R. E. (2023). Mobile money as a driver of digital financial inclusion. Technological Forecasting and Social Change, 186 , 122158.

Sharma, R., Kamble, S., Gupta, S., Belhadi, A., Rana, N. P., & Kumar, K. (2023). Interlinkages between digital-social entrepreneurship and technological capabilities for sustainable value creation. Journal of Global Information Management (JGIM), 31 (1), 1–26.

Siddik, A. B., Rahman, M. N., & Yong, L. (2023). Do fintech adoption and financial literacy improve corporate sustainability performance? The mediating role of access to finance. Journal of Cleaner Production , 421 , 137658. https://doi.org/10.1016/j.jclepro.2023.137658

Soetan, T. O., Mogaji, E., & Nguyen, N. P. (2021). Financial services experience and consumption in Nigeria. Journal of Services Marketing, 35 (7), 947–961.

Sultana, N., Chowdhury, R. S., & Haque, A. (2023). Gravitating towards Fintech: A study on undergraduates using extended UTAUT model. Heliyon, 9 (10). https://doi.org/10.1016/j.heliyon.2023.e2073

Tay, L. Y., Tai, H. T., & Tan, G. S. (2022). Digital financial inclusion: A gateway to sustainable development. Heliyon 8 (6). https://doi.org/10.1016/j.heliyon.2022.e09766

Tepe, G., Geyikci, U. B., & Sancak, F. M. (2021). Fintech companies: A bibliometric analysis. International Journal of Financial Studies, 10 (1), 2.

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14 (3), 207–222.

Truby, J. (2020). Fintech and the city: Sandbox 2.0 policy and regulatory reform proposals. International Review of Law, Computers & Technology , 34 (3), 277–309. https://doi.org/10.1080/13600869.2018.1546542

Úbeda, F., Mendez, A., & Forcadell, F. J. (2023). The sustainable practices of multinational banks as drivers of financial inclusion in developing countries. Finance Research Letters, 51 , 103278.

UNSGSA. (2018) Igniting SDG progress through digital financial inclusion. Accessed on 7 October 2023. Available online: https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=2655&menu=1515

Vasile, V., Panait, M., & Apostu, S. A. (2021). Financial inclusion paradigm shift in the post pandemic period. Digital-divide and gender gap. International Journal of Environmental Research and Public Health, 18 (20), 10938.

Wang, L., Wu, Y., Huang, Z., & Wang, Y. (2022). How big data drives green economic development: Evidence from China. Frontiers in Environmental Science, 10 , 1055162.

Xue, L., Dong, J., & Zha, Y. (2023). How does digital finance affect firm environmental, social and governance (ESG) performance?—Evidence from Chinese listed firms. Heliyon, 9 (10). https://doi.org/10.1016/j.heliyon.2023.e20800

Yang, L., Chen, Z., Liu, T., Gong, Z., Yu, Y., & Wang, J. (2013). Global trends of solid waste research from 1997 to 2011 by using bibliometric analysis. Scientometrics, 96 (1), 133–146. https://doi.org/10.1007/s11192-012-0911-6

Zerucha, T. (2023, May 05). “ The key factors driving financial inclusion”: Fintech Nexus. Accessed on May 28, 2024. https://www.fintechnexus.com/the-key-factors-driving-financial-inclusion/

Download references

Author information

Authors and affiliations.

I.K. Gujral Punjab Technical University, Kapurthala, Punjab, India

Kriti Kishor & Sanjeev K. Bansal

Banaras Hindu University, Varanasi, India

Roshan Kumar

You can also search for this author in PubMed Google Scholar

Contributions

All authors made contributions to the conceptualisation and design of this study. The authors collaborated as a team to carry out material preparation, SLR data collection, and analysis. The initial version of the paper had been written by the collective team of authors, who actively engaged in providing feedback. The final manuscript was read and approved by every author.

Corresponding author

Correspondence to Kriti Kishor .

Ethics declarations

Conflict of interest.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Kishor, K., Bansal, S.K. & Kumar, R. The Role of Fintech in Promoting Financial Inclusion to Achieve Sustainable Development: An Integrated Bibliometric Analysis and Systematic Literature Review. J Knowl Econ (2024). https://doi.org/10.1007/s13132-024-02168-5

Download citation

Received : 03 April 2024

Accepted : 14 June 2024

Published : 03 July 2024

DOI : https://doi.org/10.1007/s13132-024-02168-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Financial Inclusion
Sustainability
Bibliometric
Systematic Literature Review
Find a journal
Publish with us
Track your research

Flip ’N Jacks

Hours updated 2 weeks ago

Today is a holiday! Business hours may be different today.

Location & Hours

Suggest an edit

3221 Adventureland Dr

Altoona, IA 50009

Mon

Tue

Wed

Thu

Fri

Sat

Sun




	Closed now

Amenities and More

Ask the Community

Ask a question

Is this a kid friendly olace?

Yes! I see kids in there and I think they have a children's menu

Do you take reservations?

Not sure if you can call ahead to make a reservation or not

Recommended Reviews

To read this content please select one of the options below:

Please note you do not have access to teaching notes, sustainability in (engineering) education through reclaiming and reusing electronic components from e-waste: a last decade research review.

International Journal of Sustainability in Higher Education

ISSN : 1467-6370

Article publication date: 5 July 2024

This paper aims to investigate the e-waste aspect of sustainability in education, with a specific interest in engineering education. Specifically, it focuses on recycling through reclaiming electronic components from e-waste and reusing them in repairs or in the design and construction of new devices.

Design/methodology/approach

A systematic literature review is performed according to the PRISMA methodology. In total, 27 articles are analysed as to publication parameters, characteristics and evaluation of educational interventions on e-waste and evaluation results across major domains of learning (cognitive, affective and 21st century skills).

The reviewed subject is under-research; publications are rare and mostly in conference proceedings. The majority of interventions take place at university level, in face-to-face mode, using a practical approach in hands-on labs. Educational methods draw from modern, learner-centred pedagogies such as collaborative learning and constructionism. Topics focus on innovative design and construction, while interventions tend to become embedded in engineering curricula/courses. Evaluation of learners’ gains across domains of learning is rare and follows informal procedures that shake the reliability of results. Domains other than the cognitive are scarcely and subjectively evaluated.

Originality/value

Contrary to other aspects of sustainability, the aspect of e-waste has not been reviewed. The applied, hands-on approach and the analytic, synthetic, collaboration and creativity skills it requires are all much valued in education. The current review, therefore, comes to inform, inspire and guide educators and researchers in planning and implementing activities on this subject.

Sustainability
Engineering education
Electronic components
Educational intervention
Domains of learning
Systematic literature review

Nikoloudakis, N. and Rangoussi, M. (2024), "Sustainability in (engineering) education through reclaiming and reusing electronic components from e-waste: a last decade research review", International Journal of Sustainability in Higher Education , Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/IJSHE-09-2023-0405

Emerald Publishing Limited

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

IMAGES

Structure of the literature review on Service Design
Sample of Research Literature Review
(PDF) A visual information of the systematic literature review on
50 Smart Literature Review Templates (APA) ᐅ TemplateLab
How to write a literature review: Tips, Format and Significance
(PDF) A literature review of service research since 1993

VIDEO

How to explain .. service design on a party
Chapter 5 Design of Goods and Services
Demystify Service Design Award at the SDN Next Gen Conference
"What is Service Design?"
الحلقة 2
Research Methodology

COMMENTS

Leveraging service design as a multidisciplinary approach to service
The first phase involved a literature review on service design. The selection of publications for this preliminary literature review was based on references selected by the multidisciplinary research team. The sample criterion was the relevance of the publication for service design, in terms of concepts, processes and approaches (e.g. service ...
Service design activities in health services: A systematic literature
Service Design (SD) represents a breakthrough in searching for solutions to health systems challenges, but the activities that support these solutions remain underexplored. This research investigates how SD has been applied in the healthcare sector based on two conceptual models: multilevel ecosystem perspective and SD transformative approach.
Wicked problems in service design
Research method: systematic literature review of wicked problems in service design. This systematic literature review aims to identify the literature that connects wicked problems with service design. Service design is a field that often works with social issues, which are frequently also sensitive and wicked (Miettinen and Kuure Citation 2013 ...
(PDF) Service design and Design for wellbeing: A Literature Review
This systematic literature review aims to investigate the relationship between service design and wellbeing, identifying bibliometric characteristics of the studies (publication year, origin ...
Service design activities in health services: A systematic literature
Service Design (SD) represents a breakthrough in searching for solutions to health systems challenges, but the activities that support these solutions remain underexplored. ... First, we conducted a systematic literature review in eight comprehensive databases in March 2021. Eligibility criteria returned 990 articles filtered by a search ...
Service Design Handover to user experience design
To study the Service Design Handover, i.e., knowledge transfer and knowledge utilisation in Service Creation Projects, a selection of knowledge transfer related articles is used to frame the phenomena; and then a systematic literature review is used to study the phenomena in Service Creation Projects. This study contributes to Agile UX research ...
Service design for the transformation of healthcare systems: A
There is a growing interest in applying the Service Design (SD) approach to innovate and transform healthcare systems. However, comprehensive studies are scarce. ... Hernández RJ, Cooper R, Tether B, et al. Design, the language of innovation: a review of the design studies literature. She Ji: The Journal of Design, Economics, and Innovation ...
20569 PDFs
Explore the latest full-text research PDFs, articles, conference papers, preprints and more on SERVICE DESIGN. Find methods information, sources, references or conduct a literature review on ...
Service design activities in health services: A systematic literature
Download Citation | Service design activities in health services: A systematic literature review based on ecosystem perspective and transformative approach | Service Design (SD) represents a ...
Towards a Method for Service Design
According to the literature review provided in this paper, no single service design approach covers all the aspects that are needed for the implementation and deployment of a service-oriented architecture. Beside the literature review this paper provides a service design approach that combines the existing methods and approaches.
Service Design Handover to user experience design
To provide an overview of effective knowledge transfer, frameworks of Service Creation Project information flow and Service Design Handover are proposed. Skip Conclusion: Section Conclusion: The existing knowledge transfer literature is voluminous, but this literature review is the first to study knowledge transfer in Service Creation Project ...
Data science for service design: An introductory overview of methods
The first phase (exploration) focused on exploring theoretical and tacit knowledge about Service Design and data science.The academic literature on Service Design and data science provided a foundation. This theoretical basis was complemented by empirical incursions in the field - namely, shadowing and interviewing - to extend the comprehension of service designers' ways of working.
Servitization in Digital Age: : A Systematic Literature Review
A systematic literature review was conducted to uncover successful practices and approaches in implementing servitization in the current technological setting. ... I. Legarda, M. Uranga, M. Nazabal, A. Atxa, Service design for digital servitization: Facilitating manufacturers' advanced services value proposition design in the context of ...
How HCI Adopts Service Design
How HCI Adopts Service Design CHI '22, April 29-May 05, 2022, New Orleans, LA, USA. that service design could help HCI encompass economic models as a core part of its design process [31]. Yoo et al. [. 114] echoed this view and demonstrated the contribution of service design to propose a new value co-creation model in HCI for the viable ...
Service design in higher education: a literature review
This literature review discusses management approaches British higher education institutions (HEIs) have adopted in their attempt to survive the turbulence of the last 20 years. From substantial changes in regulatory and financial frameworks to changing fundamental perceptions of whether higher education is a public or private good, HEIs are ...
How HCI Interprets Service Design: A Systematic Literature Review
According to the present literature review, service design work in HCI has not increased since the first years of 2004-2006, and the peak in publications in 2012-2013 is largely dependent on four research groups publishing in the top HCI venues. One reason behind the slow adoption may be the deeply rooted user-centered mindset of HCI.
Service design in the healthcare space with a special focus on non
After analyzing the articles and reviewing the Service design in healthcare literature, the following are our main contributions: (i) clarification of the service design concept and the developments that appears in the literature review of service design in the healthcare sector; (ii) classification of the service design tools and techniques ...
What is Service Design?
What is Service Design? Service design is a process where designers create sustainable solutions and optimal experiences for both customers in unique contexts and any service providers involved. Designers break services into sections and adapt fine-tuned solutions to suit all users' needs in context—based on actors, location and other factors.
Service design approaches and applications in higher education: A
Service design has gained ground in the field of education. This article aims to reveal current approaches of service design applied to higher education pedagogy. The methodological approach is thematic literature review. Great variation in the application of service design can be found through review of selected literature. Three key categories were used for analysis: service, method and ...
Context-awareness for the design of Smart-product service systems
A systematic literature review on context-awareness for the design of S-PSS. •. S-PSS digital capabilities allows a data-driven value co-creation where users and consumers have multiple roles. •. Context-awareness capability allows S-PSS to react to multiple stakeholders and use environments. •.
A systematic review of experimentally tested implementation strategies
Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies ...
The impact of evidence-based nursing leadership in healthcare settings
Design. In this review, we used a mixed methods approach [].A mixed methods systematic review was selected as this approach has the potential to produce direct relevance to policy makers and practitioners [].Johnson and Onwuegbuzie [] have defined mixed methods research as "the class of research in which the researcher mixes or combines quantitative and qualitative research techniques ...
How HCI Interprets Service Design: A Systematic Literature Review
To examine how HCI has adopted service design, this paper. presents results from a systematic literature review on 52 papers from the most. cited HCI publication venues. Our findings show that the ...
Design of Digital Mental Health Platforms for Family Member
Background: The COVID-19 pandemic placed an additional mental health burden on individuals and families, resulting in widespread service access problems. Digital mental health interventions suggest promise for improved accessibility. Recent reviews have shown emerging evidence for individual use and early evidence for multiusers. However, attrition rates remain high for digital mental health ...
TCCM Framework-Based Review of the Studies
Fintech's ability to enhance efficiency and reduce costs in financial services can promote greater financial inclusion (FI), which in turn serves as a foundation for sustainable and equitable development. Due to the dearth of thorough summaries in the body of existing literature, this systematic review and bibliometric analysis aim to present quantitative and qualitative information about ...
A contemporary systematic literature review of equestrian tourism
Design/methodology/approach. Using a systematic literature review (SLR) method, pertinent journal articles published over the past 3 decades were retrieved and analyzed. Based on the review process, 44 papers were identified and analyzed by publication year, journal distribution, research method, and lead author.
Lean thinking and risk management in healthcare organizations: a
This study aims to analyse the state of the art in the literature centred on the Lean approach in the context of risk management in healthcare organizations, and to identify new research opportunities, highlighting possible lines of future research.,Following a systematic literature review approach, 51 papers were considered relevant for this ...
Updated July 2024
12 reviews and 23 photos of FLIP 'N JACKS "Food was great, service was impeccable. Super cute decor & design. Will definitely be a new favorite!" Yelp. ... service was impeccable. Super cute decor & design. Will definitely be a new favorite! Over the border skillet & the butcher omelet. Helpful 3. Helpful 4. Thanks 0.
Service design approaches and applications in higher education: A
Great variation in the application of service design can be found through review of selected literature. Three key categories were used for analysis: service, method and value.
Sustainability in (engineering) education through reclaiming and
Design/methodology/approach. A systematic literature review is performed according to the PRISMA methodology. In total, 27 articles are analysed as to publication parameters, characteristics and evaluation of educational interventions on e-waste and evaluation results across major domains of learning (cognitive, affective and 21st century skills).

Save citation to file

Add to My Bibliography

Service design activities in health services: A systematic literature review based on ecosystem perspective and transformative approach

Similar articles

Publication types

Related information

Service Design

Service Design is about Designing for the Biggest Picture

How to Do Service Design Best

Do Service Design for the Complete Experience

Learn More about Service Design

Questions related to Service Design

Answer a Short Quiz to Earn a Gift

Better luck next time!

Congratulations! You did amazing

Check Your Inbox

Literature on Service Design

Learn more about Service Design

All open-source articles on Service Design

Service Design - Design is Not Just for Products

The Moment of Truth: Build Desirable Relationships with Users and Customers

Product-Service Hybrids – When Products and Services Become One

10 Ideas to Help You Sell UX Work

Making sense of new UX words: A first dictionary for UX Ecosystem Design

Understand the Service Design Process

Keep These Goals in Mind to Create Successful Service Designs

Learn the Language of Service Design

Top Service Blueprint Templates

Start Your UX Journey: Essential Insights for Success

Open Access—Link to us!

Privacy Settings

Enhanced Functionality

Referral Program

Error Reporting

Analytics Storage

Advertising Storage

Personalization Storage

Advertising Personalization

LinkedIn Insights

LinkedIn CAPI

Google Ads Tag

Share Knowledge, Get Respect!

Cite according to academic standards

New to UX Design? We’re Giving You a Free ebook!

Service design approaches and applications in higher education: A thematic literature review

4 Citations

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

Contributions to the literature

Registration

Eligibility criteria

Information sources

Search strategy

Title and abstract screening process

Full text screening

Review of study rigor

Outside expert consultation

Systematic reviews

Data abstraction and coding

Assessment and minimization of bias

Search results

Descriptive results

Control arm

Experimental arm

Tested strategies

Implications for implementation science: four recommendations

Limitations

Conclusions

Availability and materials

Abbreviations

Acknowledgements

Author information

Authors and Affiliations

Contributions

Corresponding author

Ethics declarations

Consent for publication

Competing interests

Supplementary Information

About this article

Share this article