Article Text
PDF
Abstract
Introduction Traditional Chinese medicine (TCM) is commonly used alongside Western medicine for stroke management in China. However, there is significant variation in TCM practice, and the utilisation of evidence-based clinical practice guidelines is inadequate. This study aims to evaluate the effectiveness of three popular frameworks—Consolidated Framework for Implementation Research (CFIR), Theoretical Domains Framework (TDF) and Normalization Process Theory (NPT)—in improving implementation outcomes for the integrated TCM and Western medicine clinical practice guideline for stroke management.
Methods and analysis This study employs a hybrid type III design with a factorial randomised controlled trial, where 45 TCM hospitals will be randomly assigned to one of eight experimental conditions based on the use or non-use of each framework (CFIR, TDF, NPT). The factorial design allows for the evaluation of the main effects of each framework and their two-way and three-way interactions, offering insights into which combination of frameworks is most effective in enhancing implementation outcomes. The factorial design provides greater efficiency compared with traditional designs by enabling the simultaneous testing of multiple interventions and their combinations with the same sample size, which increases statistical power. Implementation facilitators will be trained to support the guideline adoption process, with interventions aligned to specific framework components (eg, CFIR for identifying barriers and facilitators, TDF for understanding behavioural influences and NPT for normalising practices within organisational routines). Outcomes will be evaluated using the RE-AIM framework (reach, effectiveness, adoption, implementation and maintenance). Hierarchical logistic regression models will test the study hypotheses, and qualitative methods, such as interviews and focus groups, will provide contextual understanding. Additionally, a cost-effectiveness analysis will be conducted to assess the economic feasibility of the implementation strategies.
Ethics and dissemination This trial has been approved by the Institutional Review Board of Southern Medical University (approval number: #202261) and follows all relevant ethical guidelines for research involving human participants. On completion, the findings will be shared with patients, healthcare providers and stakeholders through various dissemination activities, including workshops and presentations within relevant TCM and stroke management networks. The results will be published in peer-reviewed academic journals and presented at national and international conferences to inform future practice and policy on the integration of TCM and Western medicine for stroke management.
Trial registration details This study has been registered on the Open Science Framework with the DOI: 10.17605/OSF.IO/NJEVB.
- Stroke
- Clinical Trial
- Clinical Decision-Making
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This study addresses a critical gap in implementation research by empirically evaluating the effectiveness of three widely used frameworks, both individually and in combination.
In addition to factorial analysis, the study incorporates qualitative comparative analysis and cost-effectiveness evaluation, providing a comprehensive exploration of implementation outcomes.
A hybrid type III design enables the simultaneous assessment of implementation strategies and clinical outcomes in real-world settings.
Qualitative data collection (interviews and focus groups) enhances understanding of contextual factors influencing implementation outcomes.
As the study is conducted in a single country (China), findings may have limited generalisability to other healthcare systems.
Background
Adherence to credible and evidence-based clinical practice guidelines is a critical indicator of high-quality care. There has been a plethora of clinical practice guidelines for cardiovascular conditions worldwide. But in many countries, including China, guidelines are largely not implemented.1 Stroke is the leading burden of disease in China. Inpatient stroke management has often featured a combined use of Western and traditional Chinese medicine (TCM) in China. A cross-sectional survey involving 48 general hospitals across China showed that about one-third of patients with ischaemic stroke received a wide range of proprietary Chinese medicine during their hospital care.2 Another prospective observational study found that the use of TCM in new patients with ischaemic stroke was as high as 83.1%, even exceeding the use of antithrombotic drugs.3 Furthermore, the most important clinical indication for acupuncture is the rehabilitation of poststroke sequelae.4 5 The clinical practice guideline to promote evidence-based practice (EBP) in managing stroke with integrated Western and TCM has been developed through a rigorous evidence-based medicine approach.5 However, many of these studies have not adequately addressed the challenges in guideline adherence, particularly in the context of TCM practices, where there is considerable variability in clinical application. Previous research has often focused on clinical outcomes without a strong emphasis on the implementation process, resulting in a lack of understanding regarding how best to ensure the consistent application of evidence-based guidelines in practice. Few studies in China have been conducted to develop and test the effectiveness of implementation strategies to improve the normalisation of the guidelines into the practice routine among inpatient stroke management.
Theories, models and frameworks (hereafter referred to as ‘frameworks’) are widely used in implementation research. They provide systematic guidance on determining the implementation process, identifying the determinants (barriers and facilitators) of implementing EBP and evaluating the effect of the implementation strategy. These frameworks offer a structured approach to identifying barriers and facilitators of guideline implementation, understanding the behavioural determinants that influence clinician practices and promoting the normalisation of new practices within clinical routines.6 The reliance on the frameworks has become the hallmark of implementation research. Three popular frameworks—Consolidated Framework for Implementation Research (CFIR), Normalization Process Theory (NPT) and Theoretical Domains Framework (TDF)—have received a staggering 12 845 citations in total7–9 (Web of Knowledge, accessed October 2022), indicating widespread use in research. However, many frameworks assumed the validity of their effectiveness in improving implementation with insufficient validation evidence. Even fewer studies provided empirical evidence for their effectiveness. The debates continue regarding whether the use of the frameworks provided advantages over the use of human instinct.10 There has also been little guidance on which frameworks work better in improving guideline implementation under certain contexts. The expected outcomes of using the CFIR, TDF and NPT frameworks in this study include improved adherence to the integrated clinical practice guidelines for stroke management. Furthermore, frameworks can often be used in conjunction. So it is not only necessary to investigate the use of a single framework but also important to understand the interaction among frameworks. Factorial design can be an efficient trial design to examine both the main effects and the interaction effects. By using a factorial design, the study will also assess how the combination of these frameworks impacts the overall effectiveness of guideline adoption, implementation sustainability and cost-effectiveness.
Moreover, in light of these theoretical frameworks, successful behaviour change efforts in implementing EBPs occur not only in the inner setting where clients and practitioners reside but also in an outer setting, where the organisation and social context reside.11 Contextual factors and organisational constructs or processes are associated with the dissemination and adoption of EBPs in various settings.12 13 In the present project, we propose to employ an innovative method, named qualitative comparative analysis (QCA),14 to identify the configurations (combinations) of the setting features that, when present in the practices, could at most contribute to the implementation outcomes. QCA, a method that originated in political science and sociology, has demonstrated its utility in public health evaluative science.15
In summary, this study will address two key research questions: (1) whether the use of implementation theories, models and frameworks—either individually or in combination—facilitates the effective implementation of a clinical practice guideline for stroke management in Chinese hospitals, and whether the implementation improves clinical outcomes, particularly in the integration of TCM with Western medicine, and (2) which specific combination of contextual factors most significantly contributes to successful guideline implementation. The first question will be investigated using an experimental design, while the second will employ a mixed-method approach, incorporating QCA to identify the key contextual factors influencing implementation outcomes.
Method
Study design
This article adheres to the Standards for Reporting Implementation Studies Statement (online supplemental file 1).16
Supplemental material
This will be a hybrid type III study17 whereby we will primarily test the effect of the implementation strategy in the real-world setting while gathering information on the effectiveness of the clinical intervention (ie, the use of the guideline in this study). Considering the relatively weak evidence base for TCM, it is necessary to collect health and clinical outcomes. However, in this hybrid type III trial, as all groups will be required to implement the clinical practice guideline for stroke management, we will not be able to compare the health outcomes between the groups implementing the guideline to the control groups not implementing it. Instead, the health and clinical outcomes will be evaluated in the factorial design described below.
The overall study will be based on a factorial cluster randomised controlled trial (RCT) design. The design will have three factors, each corresponding to the use of a framework of TDF, CFIR or NPT. Each factor has two levels: in use and not in the use of a given framework. Thus, the complete use of those three factors (frameworks) will form 2×2×2=8 possible configurations or experiment conditions (ECs) (table 1). These experimental conditions will be randomly assigned to the sampled participating organisations using a stratified randomisation technique. Specifically, participating hospitals will be stratified based on key characteristics such as level, size and geographical location to ensure balance across the experimental conditions. Once stratified, hospitals within each stratum will be randomly assigned to one of the eight experimental conditions using a computer-generated randomisation sequence. This method minimises the risk of confounding variables affecting the outcomes, ensuring that differences between conditions are not influenced by variations in hospital characteristics. Each participating organisation will be subject to one EC through this random assignment process. After participants have been exposed to a condition, the assessment measures of outcomes will be used to determine if any changes could be attributed to the experimental conditions. The factorial experiment does not have a fixed and single control group as in a conventional RCT. This complete 2×2×2 factorial design enables estimation of the main effect of each factor, three two-way interactions and a three-way interaction.18 The main effect of a factorial experiment is the effect between the two levels of a factor, collapsing over the levels of all remaining factors. For instance, the main effect of using NPT versus not using it in guiding the implementation of the stroke guideline is the difference in means of EC 1–4 versus EC 5–8, averaging over the levels of the other two factors TDF and CFIR. We select the factorial experiment mainly because of (1) the high efficiency of the design (with the same sample size, it offers much higher statistical power than a conventional RCT)19 and (2) the possibility to examine the interactions between the use of frameworks (eg, we need to understand whether NPT works better when it is accompanied by the use of CFIR). To elaborate on the interaction effects, we are interested in how the frameworks complement each other in practice. For example, the two-way interaction between NPT and CFIR will help us understand whether using NPT in conjunction with CFIR enhances the implementation of the stroke guideline more effectively than using NPT or CFIR alone. Similarly, the interaction between TDF and NPT may reveal whether combining behaviour-change insights from TDF with the process-oriented approach of NPT leads to better implementation outcomes. The three-way interaction—between NPT, CFIR and TDF—will allow us to investigate whether the combined use of all three frameworks yields a synergistic effect, potentially offering a more robust and comprehensive approach to addressing implementation challenges in stroke management. These interaction effects are crucial for determining the optimal combination of frameworks that can improve the adoption of clinical practice guidelines in complex healthcare environments. By examining these interactions, we aim to uncover how different implementation strategies work together and whether certain combinations are more effective under specific conditions.
Experiment conditions (ECs)
Procedures
Selection of the frameworks
We have selected three implementation science frameworks (NPT, CFIR and TDF) for this experiment. The selection of those three frameworks is based on the following criteria: (1) highly popular frameworks in implementation research6 (NPT, CFIR and TDF have received 559, 6760 and 863 citations in peer-reviewed papers as of the year 2021), (2) potential applicability across the entire process of implementation (all three frameworks can, in theory, be used throughout the implementation process from the beginning to the end, despite their relative perceived strengths in a certain phase of the implementation) and (3) potential applicability of combined use with other frameworks.
Experiment conditions and implementation processes
As aforementioned, those three frameworks establish eight ECs ranging from the combined use of all three to the use of none and anything in between. The use of the framework will follow a standard procedure. The procedure will be developed through a modified Delphi process. Based on literature and empirical evidence, the research team will first develop a candidate procedural steps of the implementation procedure. Then a group of 10–15 implementation researchers who are otherwise not associated with the programme will review and rate the procedure through several rounds of internet-based surveys. The Delphi rounds will stop when any of the following conditions are met: consensus reached (defined as Kendall’s W over 0.7), no significant difference between the expert opinions between two successive rounds (defined as not to reject the null hypothesis that there will be an equal number of changes in both directions in the target experts using McNemar χ2 test20) or a maximum of four rounds reached.21 Inspired by the NIATx model,22 the procedure as it stands now may have the following candidate steps:
Assemble an implementation team at each of the participating hospitals.
Identify the barriers and facilitators of implementing the stroke guideline.
Develop and/or identify appropriate implementation techniques to address the barriers and enhance the facilitators.
Assign roles and tasks to the implementation team members to implement the selected techniques.
Monitor the implementation process and make necessary adaptations in implementation techniques.
Evaluate the effect of the implementation.
For each step, the team will be asked to consider the use of the frameworks when appropriate in guiding their investigation. The team has the flexibility in deciding how exactly they will conduct each step. For instance, for step 2, they can use focus groups, interviews, nominal group techniques or Delphi surveys to identify the barriers and facilitators of implementing the guideline,23 24 while using the selected frameworks to guide the content domains of the discussion. As another example, for step 5, they can use rapid cycle techniques such as the Plan-Do-Study-Act25 technique to gather rapid feedback and make necessary changes to the implementation plans.
During the implementation, each participating hospital will receive assistance from a pair of facilitators. Each facilitating pair will consist of a master’s degree student in health management and an undergraduate student in health-related sciences. All facilitators have already completed and passed the study of a 30-hour Introductory Course on Implementation Science, which covers the concept of implementation, implementation frameworks, implementation process and unique research designs and methods (both qualitative and quantitative) in implementation research. The facilitators will be randomised into eight teams, corresponding to the eight ECs (table 1). The purpose of this randomisation is to minimise the influence of the possible differences in implementation competence between the teams. Throughout the implementation, the facilitators will be instructed to use the selected frameworks in that EC only. A PhD student who is knowledgeable about implementation science will serve as the central adviser for all facilitators. The duration of the entire study was 1 year, the implementation strategy will be stopped after 6 months of implementation and the samples will be collected for a full year.
Clinical intervention: TCM guideline
The clinical intervention to be implemented in this study is the Clinical Practice Guideline for the Management of Stroke with Integrated Traditional Chinese Medicine and Western Medicine. The guideline was developed by a multidisciplinary team of experts organised by Guangdong Provincial TCM Hospital (one of the largest academic TCM health centres in China) with methodological support from the GRADE China centre at Lanzhou University. The development of this guideline followed the procedures and principles from the WHO Handbook for Guideline Development, the Manual for the Development of Integrative Medicine Guidelines, as well as the Appraisal of Guidelines for Research and Evaluation, AGREE II.26–29 The guideline30 gave 14 recommendations regarding the use of TCM in managing stroke, covering eight clinical questions: (1) ‘effect of TCM on neurological function in patients with ischaemic stroke’, (2) secondary prevention of ischaemic stroke and transient ischaemic attack by TCM, (3) the effect of blood-activating herbs on patients with acute cerebral infarction thrombolysis, (4) effects of blood-activating herbs on patients transformed by cerebral infarction haemorrhage, (5) effects of blood-activating herbs on patients with hypertensive cerebral haemorrhage, (6) herbal treatment of stroke with impaired consciousness, (7) Chinese medicine treatment for poststroke dysphagia and (8) herbal treatment of poststroke depression. The guideline, however, acknowledges that the overall evidence base for TCM is poor so that almost all recommendations were ‘suggested for consideration’ rather than ‘mandatory’. Yet, this does not diminish the significance of the guideline as it substantially limits the use of numerous Chinese medicines to a few that have some evidence base.
Targeted sites and population
The study will engage four levels of participants related to stroke management in descending order: hospitals, clinical departments, physicians and patients. The study will be conducted among the TCM hospitals in Guangdong province. Despite their name, all those hospitals use both Western medicine and TCM to manage stroke. The practising doctors also received training in Western medicine and TCM. It should be noted that TCM in China is not complementary medicine but is the mainstream. Targeting this group of hospitals is done because, although hospitals practising Western medicine also use TCM in their management of stroke to a large extent, those TCM hospitals more heavily use TCM. Also, the clinicians with TCM specialty traditionally are less interested in basing their practice on evidence-based medicine. Guangdong is among the most economically developed provinces in China, with a per capita GDP of 87 899 RMB (ranked seventh among China’s 34 provinces) in 2020 and a life expectancy of 78.18 (ranked eighth) in 2019. Guangdong is also a stronghold for TCM use.31 The province has developed a network of TCM hospitals for the management of brain conditions. Currently, with a total of 45 member hospitals, the network has included mostly county-level hospitals. China’s hospitals are classified into levels 1, 2 and 3 with the increasing sophistication of services. County hospitals are mainly level 2 hospitals. Specific inclusion and exclusion criteria for the participants at the four levels are detailed in table 2. The criteria have been developed through a Delphi-based expert consensus-building process.
Inclusion and exclusion criteria with rationale for programme participation
Sample size and sampling method
The programme will target to recruit as many as possible from the 45 hospitals in the aforementioned TCM network. One patient admission seems as a sample. The sample size calculation is based on the primary implementation outcome, the physician’s adherence to the guidelines for stroke management concerning TCM, operationalised as a dichotomous variable describing whether to follow the stroke management guideline or not (detailed later in section ‘Outcomes’).
In a complete factorial design, each factor will be allocated to half of the samples and the sample size depends on the smallest and clinically significant difference between the presence and absence of a factor. We power our study on the dichotomous variable ‘Adherence to the guideline-based stroke management’.
According to the formative work with clinicians at the TCM hospitals in Guangdong province, a relative risk of approximately 30% would be considered clinically significant. 50% of physicians (should meet the inclusion and exclusion criteria in the section ‘Targeted sites and population’) followed the guideline-based stroke management (estimates based on our prior work with clinicians) to detect a 30% difference (ie, 65% of the samples followed the guideline). Assuming a two-tailed test, the alpha level of 0.05 and the desired power of 0.8, according to the formula below,32 we can get the sample size of one EC 
:
Therefore, approximately 256 samples are required to detect this effect (32 in each of the eight conditions).
To avoid contamination, this study will consider conducting a between-clusters randomisation approach, in which clusters (clinics or hospitals) are assigned as whole units for implementing experimental conditions. The clusters increase the variance of samples, which requires consideration of the design effect 
. We assume 
. Therefore, the final sample size of one condition (
) can be updated as follows:
Therefore, 512 samples are required (64 in each of the eight conditions).
Data and data collection
Outcomes
The outcomes to be measured have been determined through a modified Delphi expert process. The research team prepared the initial outcome sets, following the RE-AIM (reach, effectiveness, adoption, implementation and maintenance) framework.33 The initial measures domain of the effectiveness in the RE-AIM used the standard outcome set for stroke,34 which was well developed by a group of internationally recognised experts (including one from China) from the International Consortium for Health Outcomes Measurement (ICHOM). The ICHOM standard outcome sets feature acute complications, disease control, as well as a large component of patient-reported outcomes. The initial RE-AIM outcome sets went through two rounds of the anonymous consensus-building process. 10 experts participated in the Delphi process, including 2 stroke-related physicians, 4 health system/service researchers, 3 behavioural scientists and 1 health economist. The process led to the establishment of 24 outcomes within the five RE-AIM domains. All outcomes are deemed to be important to the stakeholders and also can be operationalised. The details of this process will be reported in a separate paper. The final outcome set, including the variable names, definition of the outcome, timeframe for collection and methods of collection, is summarised in online supplemental file 2. The primary outcome will be clinician adherence to guideline-based stroke management (Outcome 4.1.1 in online supplemental file 2). The original guideline research team will develop this checklist. The checklist will include critical process indicators that represent the fidelity of the guideline implementation. The current draft checklist is listed in online supplemental file 3. The checklist will undergo further refinement. The primary outcome will be formulated according to the specific recommendations of the guidelines checklist, either as a continuous variable indicating the proportion (value 0 to 1) of what should be done in the checklist that is done or will be classified as a dichotomous variable (value 0 or 1), through which experts will judge whether the treatment process follows the guidelines or not. The information on the checklist completion will be abstracted from the medical chart.
Supplemental material
Supplemental material
Implementation process data
We will conduct the process evaluation at the end of the programme. Choosing this time point is to minimise the unintended influence on the project as the process evaluation itself may become a form of implementation technique if conducted during the programme. We will use a method that we have developed and termed ‘Silent & Anonymous FEedback (SAFE)’ to obtain the process information which is inspired by the consensus-building techniques of the Delphi method and nominal group techniques,35 but with a focus on anonymity and real-time interaction. SAFE is a method designed to collect process evaluation data in a manner that encourages equal participation, minimises bias and maintains the anonymity of respondents. The SAFE method allows stakeholders, such as clinicians and hospital staff, to provide feedback on the implementation process of the clinical guidelines without fear of judgement or retribution. Here is how the method works:
Participant selection: Key stakeholders are identified, and a random selection is made from different categories (eg, administrators, physicians, nurses) to ensure a broad and representative group.
Anonymity: During a virtual meeting (eg, using an online platform like VooV Meeting), participants join anonymously under pseudonyms. This removes personal identification from the discussion process, allowing for unbiased feedback.
Feedback process: Participants are asked to complete a real-time web-based survey about specific aspects of the guideline implementation. The survey includes questions about the processes, challenges and effectiveness of the frameworks applied.
Discussion: After completing the survey, the results are displayed on the shared screen, and participants are encouraged to share further thoughts and explanations in the chat box, where comments remain anonymous.
Facilitator role: A facilitator guides the session, prompting participants to explain their answers and ensuring that all points are discussed. However, no one is identified, ensuring candid feedback.
Quantitative and qualitative data: The SAFE process collects both quantitative survey data and qualitative discussion inputs (from chat and surveys). These data provide insights into how the frameworks were used and the challenges encountered during the implementation process.
The goal of SAFE is to obtain honest, reflective feedback from all participants in a way that is non-hierarchical, ensuring that all voices are heard equally.
Specifically, SAFE in this study will include the following processes in a 2-hour virtual group session: (1) preparing a list of categories of potential SAFE participants (such as administrators, physicians, nurses, financial personnel, support staff) from an implementation site (ie, the participating hospital); (2) from each category, asking the site to provide several potential participating names; (3) randomly selecting one person from the names under each category; (4) selected participants attending a web-based group meeting (such as using Tencent’s VooV Meeting) anonymously with a pseudonym; (5) at the virtual meeting, a facilitator opening the session to introduce the purpose and agenda; (6) the facilitator then releasing a web-based survey for the participants to fill out immediately, which contains a checklist of processes supposed to be completed during the guideline implementation; (7) survey results being immediately displayed on the shared screen with the participants; (8) the facilitator asking specific questions regarding why and how an item on the survey are performed from each participant’ perspective; (9) all participants being encouraged to write their response on the VooV Meeting chat box that are visible to all participants and to respond to the postings of other participants if they would like to; and finally, (10) the facilitators asking any questions further if needed and then wrapping up. The survey involved in the process of SAFE will be further developed but will include the items about (1) how exactly the frameworks are applied, (2) challenges in applying the frameworks and the approaches adopted to meet those challenges, (3) specific guideline implementation techniques developed as a result of applying the frameworks, (4) modifications made throughout the implementation to those techniques and (5) participants’ rating of the use of the frameworks and the implementation techniques in facilitating the implementation. We should note this process is to solicit not only negative but also positive feedback. The format will give both quantitative (ie, the survey results) and qualitative information (chat box conversation) for later analysis. We believe SAFE will be a more efficient way of engaging busy clinicians and hospital staff. The written response may also improve clarity in the expression of opinions and statements of facts. We will continue to develop more details of the process of SAFE and the survey forms to be used. We intend to publish a detailed protocol for doing SAFE on our project site (https://www.researchgate.net/project/ACACIA-Study) before we implement this in the field. In addition to the formal process of collecting process information through SAFE, we will ask the participating hospitals whether they have used any formative evaluation in their own implementation of the guideline. We would ask them to share the formative evaluation information (which may be qualitative or quantitative data) so that we can retrospectively analyse that data.
Economic variables
When employing an implementation strategy, costs are incurred.36 In the implementation research, there are mainly three costs: (1) development and execution of the implementation strategy; (2) execution of the EBP and (3) downstream costs, including healthcare and non-healthcare resources caused by the intervention.37 Implementation costs are mainly to develop and implement one or more EBPs. In our study, as illustrated in figure 1, we will include planning and preparation for the CFIR, TDF and NPT strategy, training of stakeholders and some changed behaviours related to the strategy. Intervention costs are resource costs directly from the consequence of implementation strategies targeting EBPs. It is recommended to distinguish the implementation costs and intervention costs.38 Apart from preparation and training of EBP, our study will also collect the changed resources needed to implement the intervention. Downstream costs are defined as those resources needed after finishing the intervention, including healthcare utilisation and productivity costs of caregivers. In our study, we take the resources needed for audit and feedback, as well as assessing of maintenance as the downstream cost.
Types of costs to consider measuring and economic consequence (modified from Heather T. Gold).37 This illustrates the relationship between various types of costs and the resulting benefits. The costs are divided into three categories: implementation costs, which include planning, stakeholder training and changes in strategy-related behaviours; intervention costs, covering intervention planning, informal caregiver time and resource utilisation; and downstream costs, such as auditing, feedback and future expenses. These costs, collectively referred to as the ‘Numerator’, contribute to the ultimate benefit, or ‘Denominator’, which is improved adherence to guidelines across different implementation strategies. CFIR, Consolidated Framework for Implementation Research; NTP, Normalization Process Theory; TCM, traditional Chinese medicine; TDF, Theoretical Domains Framework.
To develop the cost estimates, we plan to use time-driven activity-based costing.39 First, we will calculate the capacity cost rate for the stakeholders (including clinicians, nurses, front workers and managers of the healthcare sector) as a function of total annual compensation divided by the annual working time. Then, we will estimate the demand time of study-related behaviours according to box 1. With the capacity cost rate and demand time, the costs for this study can be calculated.
Recording time spent on the study (implementation strategy, intervention and downstream behaviours)
Development, preparation stage (all sites).
Time of site representatives spent with research staff in meetings and establishing procedures inspired by the NIATx model.
Time of research staff and site representatives spent at in-service training.
Time of site implementation science (IS) team spent identifying barriers and facilitators and strategising.
The time needed for project staff (investigators, personnel): for example, packaging strategies.
Time spent with provider contacts, follow-up technical assistance and documentation of each site.
Execution stage (differ among sites).
Time of training, self-education and workshop: time spent on clinicians learning the guideline and strategies (eg, phone, group sessions, online meetings).
Time of site IS team spent delivering strategies on their site.
Time of audit and feedback: any time spent on a summary of the clinical performance of healthcare provider(s) over a specified time (eg, phone follow-ups, documentation).
Time of reminders: any time spent on reminding traditional Chinese medicine (TCM) guidelines (eg, socially based peer reminders, paper-based reminders).
Time of charting: any time spent charting for a TMC guideline patient (eg, registry, progress notes, updating treatment plans).
Time spent with clinicians (phone contacts, email follow-ups).
Analysis
Factorial analysis
Following an intent-to-treat model, a hierarchical logistic regression model will be used to test the hypotheses regarding the main effects of the three frameworks (TDF, CFIR and NPT) and their interaction effects on the study’s primary outcomes. As described in the study design, this study is a cross-section of data with a three-level structure consisting of patients (level 1) nested within physicians (level 2) nested within hospitals (level 3).
Level 1 model
Let
be the outcome of patient
i
seeing physician
j
in hospital
k
. For a between-clusters experiment, one can model level 1 responses as
where
is the intercept for physician
j
in hospital
k
;-
is a level 1 random effect that represents a combination of random patient variability and measurement error. These residual patient effects are assumed normally distributed with a mean of 0 and variance
.
Level 2 model
where
-
is the intercept for physician in modelling the patient effect
;
-
is the level 2 physician random effect that represents the deviation of physician jk’s level 1 patients coefficient,
, from its predicted value based on the physician-level model.
Level 3 model
So that
where
is the intercept term in the hospital level for 
.
NPT, TDF, CFIR are three predictors for the hospital effect, including the main effect and two-way, three-way interaction effect. The predictor will be +1 if the hospital receives the EC and −1 for the control condition.40
~ 
are the corresponding level 3 (cluster-level) coefficients that represent the direction and the strength of association of those three predictors' main effect and interaction effect. 
~ 
expresses half the expected change in
Y
when NPT or TDF or CFIR is taken from −1 to +1.
is a level 3 hospital random effect that represents the deviation of hospital k’s level 2 physician coefficient 
.
The hierarchical logistic regression model not only can simultaneously assess the individual- and cluster-level variance but is also good at estimating the parameter when some data are missing using maximum likelihood if the missingness is Missing at Random or Missing Completely at Random.41 Further, we will examine all the data for missing information and loss to follow-up and will use multiple imputations as an alternative. Lastly, after well defining the hierarchical logistic regression model which best captures the design of the study, the model diagnostics will be performed to test the assumptions of the multilevel models. We will conduct all analyses using R software (R Core Team, 2020), with specific packages such as ‘lme4’ for hierarchical modelling and ‘mice’ for multiple imputations.
Implementation process analysis
The purpose of the process evaluation and analysis is to understand (1) what exact implementation techniques have been taken to implement the stroke guideline and (2) how the frameworks have been used to guide the process of implementation. The quantitative information from the survey data (see section ‘Implementation process data’) will be tabulated descriptively with means, frequencies and statistical tests when appropriate. The qualitative data will be analysed with ATLAS.ti with deductive content analysis (DCA).42 The DCA approach fits with our study as our analysis will be framework-guided. Specifically, two researchers with qualitative expertise will read the transcripts from the SAFE process discussed above and other sources repeatedly to become familiar with them, categorise important ideas and concepts in the material to a coding system informed by the implementation frameworks (specific frameworks to be decided later) and identify corresponding examples of excerpts from the material. The specific ways for each implementation team in applying the three frameworks understudy will be abstracted into a written summary. An expert panel with deep knowledge of those frameworks will evaluate the specific application of the frameworks for their appropriate use, misuse and superficial use. All implementation techniques will be summarised and tabulated.
Economic analysis
We plan to perform cost-effectiveness analysis43 to assess whether the additional costs of implementing each component are justified by the corresponding benefits. The key economic outcome variables include (1) improved adherence rates to the guidelines based on different combinations of components and (2) the cost per percentage increase in adherence achieved. Incremental cost-effectiveness ratios (ICERS)44 will be used to compare the cost and benefit of three implementation framework components.
The ICERS will be defined as ΔC/ΔE, where ΔC is the cost a component adds while ΔE is the effectiveness (ΔC and ΔE are the numerator and denominator, respectively in figure 1). Traditionally, quality-adjusted life-year (QALY) is an effective factor to measure effectiveness. However, we cannot get a quantitative relationship between adherence to the guideline and the QALY in patients with stroke. Therefore, adherence will be directly used for effectiveness. The ICER will provide the cost required to achieve a one-percentage-point increase in adherence to the guidelines. In cases where one intervention is more expensive than another, it may still be deemed cost-effective if it results in a significant improvement in adherence at an acceptable cost. The threshold for what constitutes an acceptable cost will be determined through a SAFE process to ensure consensus among stakeholders. To ensure the robustness of our component selection, we will also calculate 95% CIs around the ICERs using bootstrapping simulation methods,43 providing greater statistical confidence in the cost-effectiveness findings.
Qualitative comparative analysis
To investigate which combinations of outer setting features in conjunction with the use of different frameworks may act as necessary or sufficient conditions for the occurrence of the outcome (successful implementation of stroke clinical guidelines), we propose to use QCA. Based on Boolean logic,45 QCA compares sets (a set that can be defined as a group of elements that share certain characteristics) of conditions and the relationship of conditions to outcomes by examining across-case patterns. QCA’s examination of cross-case patterns acknowledges the diversity of cases and their heterogeneity with regard to their different relevant conditions (‘variables’) and contexts by comparing cases as configurations. Rather than assuming random data distribution, QCA is a non-linear and non-additive method investigating how observations are distributed across rows in a ‘truth table’ (a data metric with 2k rows, where k is the number of conditions; the truth table reflects all logically possible combinations of conditions, with each row referring to a specific combination of conditions (ie, a configuration)). Applying the Quine-McCluskey algorithm (method of prime implicants), the truth table reflecting all logically possible configurations can be reduced to Boolean equations that minimise these combinations which yield prime configurations. QCA uses two measures to assess goodness of fit: consistency (ie, the strength of the link between condition to outcomes, within the range of 0–1) and coverage (ie, the fraction of cases to which relationship applies, within the range of 0–1) to assess goodness of fit. The QCA will be performed using the R software (R Core Team 2020, RStudio text editor 2020) and the R packages ‘CA’46 and ‘SetMethods’.47
In this study, we will use an explanatory sequential design48 to collect qualitative data with key stakeholders from a small number of representative hospitals, which will inform both the qualitative and quantitative data collection for the full study. The QCA method is particularly well suited for this study because it allows us to examine complex causality by identifying the combinations of conditions that are necessary or sufficient to achieve successful implementation outcomes. QCA offers a robust framework for analysing diverse cases while retaining the richness of qualitative data. It is especially advantageous in health systems research where contexts can vary widely across different settings, making it an ideal choice for exploring the complex dynamics of stroke management guideline implementation in Chinese hospitals.
In terms of case selection, we will select three high- and three low-performing hospitals from the full sample of participating hospitals for qualitative data collection using two criteria. First, we evaluate the baseline performance of participating hospitals by using the RE-AIM domains (as discussed in more detail in the earlier section) and rank the hospitals on their baseline performance. Second, we will purposively sample hospitals representing a diversity of geographical locations, sizes and ownership types. This ensures that we capture a wide range of organisational contexts.
We will conduct one-to-one and focus group interviews with key stakeholders who are directly involved in the implementation of stroke management clinical guidelines. This will include hospital senior and middle-level managers, hospital administrators, patient care managers, clinicians, patients, patients’ family members or caregivers. The qualitative data collected through these interviews will be analysed using ATLAS.ti (a qualitative analytical software) to inform the selection of the most relevant contextual features in conjunction with the implementation frameworks under study.
In our approach, we treat each participating hospital as a case (the unit of analysis), which allows us to explore cross-case patterns and identify which configurations of contextual and organisational factors are linked to successful implementation. The integration of qualitative and quantitative data will occur through the construction of truth tables in QCA, where quantitative data will provide the foundational structure, and qualitative insights will be used to deepen our understanding of each case’s conditions. This integration of data will enable us to examine both the causal relationships and the rich contextual factors influencing the implementation process, thereby offering a comprehensive understanding of how the stroke management clinical guidelines are adopted across diverse hospital settings.
Patient and public involvement
We acknowledge the importance of patient and public engagement, however, due to the nature of this study and its specific objectives, we did not incorporate relevant feedback and insights.
Discussion
In this proposed factorial cluster RCT, we will examine the main effect and interaction of the use of implementation frameworks in facilitating the implementation of the stroke management guideline in TCM hospitals in the Guangdong province of China. QCA will be used to analyse the contextual factors that facilitate or hamper the implementation of the guideline. The study will be a type III hybrid design that focuses on testing the effect of the implementation strategies (the use of the implementation frameworks) while collecting and analysing the effect of the health and clinical outcomes.
As of 17 November 2022, we have not found other studies that study the effectiveness of using implementation frameworks in facilitating the implementation of a clinical intervention using an experimental design. As implementation research is rapidly developing, we need to develop empirical evidence on the strengths and weaknesses of using implementation frameworks. Furthermore, the study can provide experiences on how implementation strategies can be tailored to the context of the organisation. In the trial for the effectiveness of a clinical intervention, consistently conducting clinical intervention across organisations is pursued. However, for implementation techniques, many a time they will have to be context-specific. In this trial, although each organisation will be applying the implementation frameworks in a standardised way, they have the flexibility of producing different implementation techniques and packaging those into implementation strategies (bundle of those implementation techniques) that suit the outer and inner contexts of their organisation. In addition to the examination of applying implementation frameworks, the QCA method will enable us to examine the contextual factors. QCA has been used more retrospectively in the past, which has the limitation of having no information on some conditions of interest. In this trial, we will be able to use theories and empirical experiences to help us prospectively plan for the data collection. Finally, the process evaluation of this study will also yield useful information for a comparative understanding of different frameworks and their application in real practice. Recommendations will also be made on how those frameworks might be applied for future studies considering their strengths and weaknesses.
The study has several foreseeable limitations. First, the facilitators and hospital implementation teams are not blinded to the frameworks to be applied to their work. However, unlike a biomedical trial, we expect the placebo effect of this implementation trial to be minimal. Second, it is possible that the hospitals may tap into ideas from other implementation frameworks not assigned to them in their actual implementation work. Our choice of cluster randomisation may help reduce this possible contamination. Third, recruiting enough participants from multiple sites may be difficult due to factors like patient availability and hospital cooperation. To address this, we plan to expand recruitment efforts by involving more hospitals and, if necessary, extend the recruitment period to meet the target. Additionally, we may use statistical methods like multiple imputation to handle missing data, which will help maintain the validity of the analysis even if the full sample size is not reached. While these strategies aim to mitigate recruitment challenges, we acknowledge that sample size limitations could still impact the generalisability of the results.
Ethics and dissemination
This study has been approved by the Institutional Review Board (IRB) of Southern Medical University (approval number: #202261). Informed consent will be obtained from all participants prior to their involvement in the study, including for surveys and qualitative interviews. Participants will be provided with detailed information about the study’s objectives, procedures, potential risks, and benefits, ensuring they fully understand their rights, including the right to withdraw at any time without penalty. To protect participant privacy, all data will be anonymised, and identifying information will be removed during data collection, analysis and reporting. Data will be securely stored and accessible only to authorised research team members, in accordance with institutional and national data protection regulations. Additionally, all research involving human participants, materials or data will be conducted in compliance with the Declaration of Helsinki and other relevant ethical guidelines and regulations. We have taken these measures to ensure the ethical integrity of the study and the protection of our participants.
Ethics statements
Patient consent for publication
Acknowledgments
We would like to express our sincere gratitude to the teachers from the Guangdong Provincial Hospital of Traditional Chinese Medicine for their valuable contributions to this project.
References
Footnotes
X @qingcarolzhao, @romanxu
YeC and D(R)X contributed equally.
Contributors D(R)X, YeC and YiC designed the study. D(R)X and WH wrote the manuscript. CH, ZC and YS provided suggestions and improved the health economics evaluation section. PG provided guidance on causal design. SL, LZ and QZ provided guidance and suggestions on expert consensus. LL provided guidance and supplementation on QCA methodology application. D(R)X is responsible for the overall content as the guarantor. All authors read and approved the final manuscript.
Funding Z Chen acknowledges partial financial support from the General Program of the National Natural Science Foundation of China (Grant No. 72174098)
Competing interests None declared.
Patient and public involvement Patients and the public were not involved in the design, implementation or dissemination of the research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.