Article Text
PDF
Abstract
Objective To compare differences in recruitment and attrition between placebo control randomised trials of surgery, and trials of the same surgical interventions and conditions that used non-operative (non-placebo) controls.
Design Meta-epidemiological study.
Data sources Randomised controlled trials were identified from an electronic search of MEDLINE, EMBASE and Cochrane Central Register of Controlled Trials from their inception date to 21 November 2018.
Study selection Placebo control trials evaluating efficacy of any surgical intervention and non-operative control trials of the same surgical intervention were included in this study. 25 730 records were retrieved from our systemic search, identifying 61 placebo control and 38 non-operative control trials for inclusion in analysis.
Outcome measures Primary outcome measures were recruitment and attrition. These were assessed in terms of recruitment rate (number of participants enrolled, as a proportion of those eligible) and overall attrition rate (composite of dropout, loss to follow-up and cross-overs, expressed as proportion of total sample size). Secondary outcome measures included participant cross-over rate, dropout and loss to follow-up.
Results Unadjusted pooled recruitment and attrition rates were similar between placebo and non-operative control trials. Study characteristics were not significantly different apart from time to primary timepoint which was shorter in studies with placebo controls (365 vs 274 days, p=0.006). After adjusting for covariates (follow-up duration and number of timepoints), the attrition rate of placebo control trials was almost twice as high compared with non-operative controlled-trials (incident rate ratio (IRR) (95% CI) 1.8 (1.1 to 3.0), p=0.032). The incorporation of one additional follow-up timepoint (regardless of follow-up duration) was associated with reduced attrition in placebo control surgical trials (IRR (95% CI) 0.64 (0.52 to 0.79), p<0.001).
Conclusions Placebo control trials of surgery have similar recruitment issues but higher attrition compared with non-operative (non-placebo) control trials. Study design should incorporate strategies such as increased timepoints for given follow-up duration to mitigate losses to follow-up and dropout.
PROSPERO registration number CRD42019117364.
- Randomized Controlled Trial
- SURGERY
- Follow-Up Studies
Data availability statement
Data are available upon reasonable request. Additional analyses included as online supplemental appendixes.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Randomised controlled trials incorporating a placebo control to evaluate effectiveness of a surgical intervention were compared with randomised trials comparing the effectiveness of the same surgical intervention with non-operative controls.
In addition to primary outcomes collected, secondary outcomes including participant cross-over rate, participant dropout and participant loss to follow-up were recorded and evaluated.
To minimise bias, data was extracted independently by pairs of investigators and arbitrated by a third investigator if necessary.
Findings are limited by missing data and non-reporting of recruitment (n=42 studies) or attrition (n=4 studies) data.
The relatively small amount of placebo-controlled surgical trials published in the literature limit the certainty of our evaluations.
Introduction
Placebo control trials are the gold standard for determining the true therapeutic effect of interventions.1 However, placebo trials commonly face difficulties in participant recruitment due to a lack of willingness to participate especially in surgical placebo trials due to its inherently invasive nature and higher risks of anaesthetic adverse events and infection.2–4
Invasive and lengthy procedural processes in surgical trials may also lead to participant attrition.5–7 Attrition refers to losses in participant information either due to dropout or missing data over the duration of a longitudinal study.8 These losses can create imbalances in study groups introducing bias and reduced statistical power secondary to a smaller sample size.8 9
The extent of attrition and recruitment issues in placebo control trials of surgical interventions have not been explored empirically. The aim of this study was therefore to investigate differences in participant recruitment and attrition rates between placebo and non-operative (non-placebo) control surgical trials testing the same surgical intervention to guide future planning of placebo control studies.
Methods
Design
We performed a meta-epidemiological study and registered the protocol in the PROSPERO International Prospective Register of Systematic Reviews (CRD42019117364) (online supplemental files 1 and 2). We followed the reporting guidance of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).10
Supplemental material
Supplemental material
Inclusion criteria and eligible study identification
This study included randomised controlled trials incorporating a placebo control to evaluate the efficacy of any surgical intervention and randomised trials comparing the effectiveness of the same surgical intervention with non-operative controls. The latter may comprise either standard care or no treatment. Trials were excluded if they were not evaluating the same surgical effect as the corresponding placebo control trial, for example, the non-operative control group received co-interventions not provided to the surgical group.
Surgery was defined as any invasive procedure that allows access to internal anatomy for example through a skin incision. The surgical placebo is ill-defined and can vary in fidelity but was defined as any ‘imitation procedure’ differentiated by the patient, which lacks the key surgical element(s).11
This study used the search strategy and eligibility criteria from an associated publication by Karjalainen et al (online supplemental appendix 1).12 Detailed data on the search strategy and eligibility criteria (including the PRISMA diagram of included studies) are available via the supplementary files of Karjalainen et al.12 Based on a full-text assessment, trials were excluded because of two main reasons: they did not meet our definition of a surgical intervention (such as the injection or heating of tissue) or they were duplicate articles. The search identified 62 placebo controlled surgical trials.
Supplemental material
Our search included eligible placebo control trials from a published systematic review by Wartolowska et al1 as well as an extension of its search until 21 November 2018. We also searched the reference lists of included studies for additional eligible studies. To identify relevant effectiveness trials (incorporating non-blinded non-operative controls), relevant Cochrane reviews assessing the index surgical procedure were identified and their literature searches were also extended until 13 March to 15 March, 2019. Where no relevant Cochrane review was identified, a search algorithm was devised and applied to the Cochrane Central Register of Controlled Trials, MEDLINE and Embase from their inception until the same date of search. To determine eligibility, pairs of authors independently completed title/abstract screening (TK, SA) followed by full-text review (PN, SM, LH, MM, SA).
Data extraction
All data were extracted independently by pairs of investigators (PN, SM, LH, MM) and arbitrated by a third investigator (SA) if necessary. Extracted data from included trials included year of publication, participant characteristics (age, sex), sample size, condition, intervention type (open or minimally invasive/percutaneous surgery), planned length of follow-up and number of follow-up timepoints.
Primary and secondary outcome measures
Primary outcomes were participant recruitment and attrition. These outcomes were assessed in terms of recruitment rate (number of participants enrolled, as a proportion of those eligible) and overall attrition rate (composite of dropout, loss to follow-up and cross-overs, expressed as proportion of total sample size).
Secondary measures included the participant cross-over rate, defined as an unplanned protocol violation resulting in participants in the control group receiving the intervention and vice versa, and participant dropout, defined as an inability for the participant to progress further with the study. These were both reported as a proportion of total number recruited. Finally, we also included participant loss to follow-up, defined as the inability of investigators to obtain information at planned timepoints for reasons other than participant dropout. Where available, these components of attrition were characterised at each follow-up timepoint.
Statistical analyses
Descriptive statistics were used to summarise key aspect of the selected studies. The ‘metaprop’ command in Stata V.16 was used to estimate pooled recruitment and attrition rates, stratified by study type (placebo vs non-operative control). Overall recruitment and attrition rates were the primary outcomes used for this analysis. To account for between-study heterogeneity, all analyses were based on the random effect model. Random effect meta-analysis was used to summarise attrition rates (overall, dropout, loss to follow-up and cross-over) in placebo versus non-operative control trials, stratified by trial groups.
Due to the nature of the data (with varying follow-up duration), a generalised linear latent and mixed model13 was employed for random effect Poisson regression to examine incident rate ratio (IRR) for intervention type (placebo or non-operative control). With this model, we also controlled for participant gender, follow-up duration and number of follow-up timepoints.
All trials with attrition and recruitment data were included in analyses. However, reporting biases were suspected in studies with 0% attrition and 100% recruitment and therefore sensitivity analyses excluding these studies were performed.
Funnel plot and Egger’s test were used to assess publication bias, while meta regression was used to examine for the effect of covariates. Risk of bias was assessed according to Cochrane Risk of Bias Tool V.1.0. and detailed in a related publication by Karjalainen et al.12
Patient and public involvement
As this was a meta-epidemiological study and meta-analysis, there was no patient involvement in study design of conduct.
Results
A total of 62 placebo control trials and 38 trials with non-operative controls (100 trials overall) were identified (figure 1). 99 studies were included in the quantitative analysis (1 placebo control trial excluded due to unavailable full text at search date14). Detailed data on these included studies has been included in online supplemental appendix 2. Study cohorts were comparable between placebo and non-operative control trials; however, time to the primary outcome was shorter in studies with placebo controls (365 vs 274 days, p=0.006) (table 1). No significant covariates were identified in meta-regression analyses (online supplemental appendix 3).
Supplemental material
Supplemental material
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart of study selection.
Participant and follow-up characteristics
Participant recruitment
Recruitment rate was available for 57 out of 99 included studies (36 (59.0%) placebo and 21 (55.3%) non-operative controls, respectively) and ranged between 9.3% and 100%.
The random effect pooled rate was similar between placebo and non-operative control trials (rate (95% CI): 76.9% (71.1% to 82.7%) vs 77.6% (66.7% to 88.4%), respectively, p=0.915). This included 10/36 (27.8%) placebo and 3/21 (14.3%) non-operative control studies with 100% recruitment rates. When these studies were excluded, the recruitment rates decreased to 68.7% (59.3% to 78.1%) in the placebo and 74.1% (58.6% to 89.5%) in the non-operative controlled studies, respectively, with no between-group heterogeneity (I2=95%, p=0.562).
Participant attrition
Overall attrition rate was not available for 4 studies (2/61 placebo arms and 2/38 non-operative controls) and ranged from 0% to 80.0% in trials with available data.
Median (IQR) attrition rates were lower in placebo trials (12.4% (6.1%–29.8%)) compared with non-operative control trials (20.7% (9.1%–33.3%)); however, these did not reach statistical significance. These results also comprised 5/59 (8.5%) placebo arm studies and 2/36 (5.6%) of non-operative control studies with no participant attrition. For studies with attrition, the random effect pooled overall attrition (rate (95% CI)) did not differ significantly between placebo (21.2% (17.2% to 25.2%)) and non-operative (23.7% (18.8% to 28.6%)) controlled studies (p=0.811). This was also true for discrete components of attrition including loss to follow-up, dropout and cross-over rates (online supplemental appendices 4–6).
Supplemental material
Supplemental material
Supplemental material
Random effect Poisson regression
The median (IQR) number of follow-up timepoints (4 (3–5.5) and 3.5 (2–6), p=0.748) was similar between non-operative and placebo control trials, respectively. Longest follow-up timepoint (365 (319.5–730) and 365 (183–456) days, p=0.143) was also similar between non-operative and placebo control trials, respectively.
Following correction for covariates especially the varied study durations, Poisson regression analyses showed significant between-group differences in the rates of dropouts, loss to follow-up and attrition (table 2). Poisson regression demonstrated a higher attrition rate in placebo trials compared with non-operative control trials (IRR 1.8 (95% CI 1.1 to 3.0), p=0.032) and was predominantly seen in the medium term (500 days). The higher attrition rate in placebo trials was due to higher loss to follow-up (IRR 2.6 (95% CI 1.04 to 6.3), p=0.042) and higher dropout (IRR 3.5 (95% CI 1.1 to 11.3), p=0.037) as seen in figure 2.
Association between attrition rates and type of control group (placebo or non-operative) in surgical trials
Poisson regression of median attrition rates (dropout, loss to follow-up, death, overall attrition) between placebo and non-operative controls.
The incorporation of just one additional follow-up timepoint (regardless of length of follow-up, that is, increased frequency of visits) is associated with a reduction in attrition (IRR (95% CI) of 0.64 (0.52 to 0.79), p<0.001) in placebo control surgical trials, largely driven by fewer losses to follow-up (IRR (95% CI) of 0.68 (0.52 to 0.89), p=0.004).
Publication bias
Egger test (p<0.001) indicated the presence of publication bias with the majority of included studies having low attrition rates (online supplemental appendix 7). Publication bias was greater in placebo control trials compared with trials of non-operative trials (online supplemental appendices 8 and 9).
Supplemental material
Supplemental material
Supplemental material
Discussion
This review demonstrates key differences in participant recruitment and retainment when comparing placebo control and non-operative (non-placebo) control randomised trials of surgery. After adjustment for the number of follow-up timepoints and study duration, attrition losses were almost twice as high in placebo control compared with non-operative control trials. This was primarily driven by participant follow-up losses and dropouts.
Participant recruitment
Surgical randomised controlled trials can face recruitment rates as low as 8%,15 due to patients frequently failing to meet eligibility criteria for a small and specific target populations.3 16 Addition of a placebo component further exacerbates this problem by undermining willingness to participate.4 17 18 Participant surveys suggest this unwillingness stems from common perceptions that invasive surgical placebos are associated with greater risks (eg, infection).18 19 Data from previous randomised controlled trials as reported by Hare et al,4 indicate participant concerns regarding the possibility of receiving placebo surgery being the most common reason (38%) for non-participation despite eligibility. Contrary to these expectations, our results demonstrated no significant difference in recruitment rate between placebo control and non-operative control trials. Our findings may be biased by sampling from published literature, with the non-representation of placebo control surgery trials that experienced stoppage and/or early termination due to recruitment failure.
Participant attrition
Our findings suggest placebo control surgery trials experience a twofold higher attrition rate (when considering cross-overs, dropouts and follow-up losses) compared with non-operative control surgery trials, after adjusting for the duration and number of follow-up timepoints. One possible cause for higher attrition rates in placebo control trials could be early unblinding. It is well-known that rigorous blinding is required to maintain equipoise (and fidelity) in placebo control surgery trials to ensure participant retention.11 20 21 Meta-analysis by Hróbjartsson et al found that non-blinded control groups suffer from 79% higher risk of dropouts and 55% higher risk of co-intervention use when compared with blinded control groups.22 The difficulties of appropriate blinding (and maintaining fidelity), especially in the context of not receiving treatment with persisting symptoms, likely account for the higher rates of attrition in placebo control surgery trials when compared with other control trials. Included trials in the present meta-analysis were published prior to the development of the Applying Surgical Placebo in Randomised Evaluations (ASPIRE) guidelines for acceptable surgical placebos, and therefore did not report on the fidelity and blinding of their surgical placebos.11
Higher attrition rates in placebo control surgical trials were primarily driven by higher losses to follow-up and participant dropout. With the inherent nature of surgical interventions being a ‘one-time’ irreversible change,23 loss to follow-up and participant withdrawals may be higher when there is a long follow-up period with no concomitant treatments.24 This is typical of placebo surgery trials, while non-operative trials tend to involve comparators that require ongoing intervention (therefore facilitating parallel follow-up).
We also found that differences in attrition rates between placebo and non-operative control trials of surgery arise primarily in the medium term (~500 days), suggestive of a ‘participant demotivation’ phenomenon that develops over moderate-term to longer-term study participation.25–29 Participant demotivation seems to be accelerated in placebo control trials, with the presence of additional uncertainty regarding potential allocation of a ‘surgical placebo’. This demotivation likely peaks following the short-term optimism initially present at enrolment into a placebo control surgery trial. Moreover, the finding of additional follow-up timepoints correlating with a reduction in attrition suggests frequent follow-up timepoints may enable ongoing contact and thus participant retention, as positive relationships between participants and trial staff are fostered.28 30
Publication bias
Trial discontinuation and non-publication is common and occurs more frequently in surgical than medical trials.31–35 Publication bias, or the selective submission or acceptance of a study into literature as such,36 37 is a likely limitation of the present findings. The majority of included studies had low attrition rates overall, indicating less publication of both placebo and non-operative control surgical trials with high attrition rates.8
Strengths and weaknesses
This study has several major strengths including a protocol-driven, preplanned, meta-epidemiological design that included all published surgical placebo trials until November 2018. Given our research question did not assess intervention effectiveness but rather described overall data from a methodological perspective, it is unlikely additional trials will change our conclusion. However, our findings are limited by missing data and non-reporting of recruitment (n=42) or attrition data (n=4) in some trials. Thus, our findings may be an underestimation of the true difference in attrition rates between placebo surgery trials and non-operative trials, as unfavourable attrition/recruitment data is less likely to be published.
Implications and future research
There is a need to investigate reasons why participant attrition occurs at a higher rate when placebo controls are employed in randomised trials of surgery. Future studies build on existing ASPIRE guidelines to explore the relationship between varying levels of placebo fidelity and rates of attrition.11 Patient education and greater transparency may promote confidence and willingness among eligible patients to participate. As such, future studies may also explore patient perceptions and attitudes towards placebo surgical procedures. Strategies to maximise continuous patient engagement may include guaranteeing placebo-exposed patients the surgical intervention if a statistically significant benefit is observed. This study also demonstrated that additional follow-up timepoints are associated with less attrition, thus closer follow-up is recommended in placebo control trials.
Conclusion
Placebo control trials of surgery have higher attrition rates when compared with trials with non-operative (non-placebo) controls. Our findings suggest that the design of surgical placebo trials should incorporate strategies with one key strategy being more frequent follow-up (for a given duration of follow-up) to mitigate losses to follow and dropout.
Transparency and ethical declaration
Authors affirm that this manuscript is an honest, accurate and transparent account of the study being reported; no important aspects of the study have been omitted; and any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Data availability statement
Data are available upon reasonable request. Additional analyses included as online supplemental appendixes.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
PN and SM are joint first authors.
X @TeemuVKarjalain
PN and SM contributed equally.
Contributors SA: Conception and design; administrative support; provision of study material or patients; and final approval of manuscript (guarantor). PN, SM, LH, MM and TK: Collection and assembly of data. All authors: Data analysis and interpretation; manuscript writing.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests RB has received royalties from UpToDate for authoring a chapter unrelated to this paper (Plantar Fasciitis). SA has received grants and/or research contracts from National Health and Medical Research Council, Avant Foundation and ANZ Musculoskeletal Clinical Trial Network as part of Sydney Partnership for Health, Education, Research and Enterprise.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.