Using data for quality improvement

Improving quality is a priority across healthcare.1 Policymakers are increasingly looking towards systematic methods of using data to improve, rather than simply to assure quality.2–4 However, achieving sustainable quality improvement (QI) has proven to be difficult,5 partly due to challenges such as differing interpretations of ‘quality’6; ineffective data systems5; a lack of informatics or improvement skills among health professionals7; or immature organisational QI leadership.8 These challenges are applicable to the perioperative setting; a recent process evaluation of a data-driven QI initiative in emergency laparotomy care concluded that local contextual factors varied the fidelity of the intervention between organisations.9

Audit and feedback interventions

Audit and feedback (A&F) interventions entail the delivery of performance data or other quality indicators to healthcare professionals over specific periods of time to assist them in identifying areas that require improvement.10 11 The expectation is that healthcare professionals will then use this information to make changes in practice, which will ultimately improve healthcare organisation and the quality of care delivered to patients.12 However, despite significant investment in A&F interventions, there is variability in their effectiveness for improving the quality of care.13 14 Simply providing data to hospitals about their performance may not be sufficient to drive QI.11 13–16 This may partially be due to the lack of theory-based explanations in the design and implementation of A&Finterventions, which leads to a lack of understanding of what an intervention consists of, and what it tries to achieve.11 13 17 In a recent study, Gude et al tried to address this gap by proposing the use of control theory to analyse A&F effectiveness in relation to a series of steps: (1) information–intention (feedback needs to convince recipients of the need to change), (2) intention–behaviour (intentions to change need to be translated into actions) and (3) behaviour–impact (actions need to create actual changes in practice).14 Unfortunately, this study did not explore the third dimension, that is, the extent to which the A&F intervention produced changes in practice.

Another gap in A&F studies is the lack of analysis of the social and organisational context where the A&F intervention is implemented or its process of implementation. Looking at aggregated data to ascertain impacts of outcome-monitoring programmes may not provide a sufficiently detailed understanding of the contexts which promote or impede improvement activity based on quality data.18 Indeed, the success of QI initiatives depends on features of both the initiatives themselves and the contexts in which they are implemented.19 A series of reviews have highlighted that neglected areas of research include the study of fidelity to the model or exposure to the intervention, the mechanisms that formed part of the intervention, staff perceptions of the intervention and the impact of the intervention on behaviour change.10 13 20

The Perioperative Quality Improvement Programme

The Perioperative Quality Improvement Programme (PQIP) is a national initiative to measure complications after major elective surgery and seek to improve these through feedback of data to clinicians.21 22 The PQIP team are aiming to overcome the challenges described above to support clinicians and managers in their use of data using an evidence-based approach. PQIP is led by the National Institute of Academic Anaesthesia Health Services Research Centre, working on behalf of the Royal College of Anaesthetists in collaboration with the Royal Colleges of Surgeons (England), Physicians and Nursing and the Faculties of Intensive Care Medicine and of Pain Medicine and has been adopted to the National Institute of Health Research (NIHR) portfolio of research studies.

The intended activities of PQIP are listed below in table 1. PQIP collects risk-adjusted morbidity and mortality data, as well as processes and patient-reported outcomes for patients undergoing selected elective major operations. Hospitals across the UK began enrolling in December 2016; 90 hospitals are currently enrolled and they have collected data on approximately 12 000 patients. Initially, these data were only collected for six surgical specialties: head and neck, thoracics, upper gastrointestinal (GI), lower GI, hepatobiliary and urology. Subsequently, extra surgical specialties have been added: spinal, complex orthopaedics, burns and plastic surgery. Participating hospitals have the option either to try to recruit all eligible patients or to randomly recruit three to five eligible patients per week using an 8-day rolling sampling cycle. To evaluate potential bias within the sample of patients recruited to PQIP, patient characteristics and outcomes will be compared against those not recruited to PQIP by using an extract of administrative data from Hospital Episode Statistics (HES) held by National Health Service (NHS) Digital. Patient-level PQIP data will also be linked with HES and the Office of National Statistics mortality register in order to track readmission and long-term mortality rates. PQIP will also consider patient-level linkage with other relevant registries and National Clinical Audits in order to provide a comprehensive dataset at the lowest local data collection burden. Patients must give informed written consent to participate in the study, in order that identifiable information can be gathered to permit such linkage with external datasets.

PQIP has set up a website with an online database for data entry and, through this, hospitals receive near real-time feedback of their results via dashboards, along with quarterly and yearly reports. Local collaborators will also be able to download anonymised versions of their entire local PQIP dataset, comprising patient characteristics, perioperative processes, clinical outcomes and patient-related outcome measures. These hospital-level data and dashboards, added to the QI tools from the website, are intended to support local and national improvement in the perioperative care of patients. The expectation is that PQIP will provide a significant patient dataset for research, which can be used both locally and nationally, to inform improvements in patient care and satisfaction.

The aims of PQIP are to:

1. Reduce the burden of perioperative complications.

2. Support local QI.

3. Facilitate national and local research.

PQIP hopes to provide local collaborators with sufficiently useful data to support QI of structures or processes related to the incidence or severity of perioperative complications. This process evaluation will further elaborate the programme theories, as understood by either central or local collaborators, as to how this improvement might , does or does not happen.

Aim

The aim of this study is to carry out a formative process evaluation of PQIP by focusing on the programme theories guiding the programme and processes of implementation.

Objectives

1. Document the processes used to design the programme.

2. Examine the views of stakeholders on the programme and how these change through different stages of implementation.

3. Analyse how the programme is implemented.

4. Explore the impact of the programme.

Research questions

1. What are the programme theories (central and at local sites) guiding the design and implementation of PQIP and how do these change through time?

2. What are the factors acting as barriers and enablers in the implementation of PQIP?

3. How did specific contexts shape PQIP and processes of implementation?

4. What was the impact of the programme on professional cultures and how data are used for improvement?

5. What are the limitations of PQIP and how can the programme be improved?

6. In the case of non-PQIP hospitals, how are data normally used (or not)? What are the perceptions of PQIP?

7. How does the use of data for improvement compare between surgical specialties included in PQIP and those which are not part of the programme?

Study design

The study has been designed as a formative process evaluation based on multisited qualitative research. Previous research on QI interventions has indicated that evaluations of complex interventions should not only assess health and organisational outcomes at the end of implementation, they should also seek to understand the intervention design and utilisation plan, implementers’ underlying ideas about its purpose and effectiveness, how specific contexts have shaped the intervention and process of implementation, mechanisms used to integrate or assimilate the intervention into routine practice, and the degree of exposure to the intervention experienced by implementers and target groups.23–25 Furthermore, when evaluations are formative, study findings can be used to guide implementation efforts in the attempt to produce positive outcomes.26

The study will explore aspects of the subjective qualities of data used in improvement.27 According to these approaches, data are never raw, but undergo a process of contextualisation that requires framing and interpretation.28 Interpretations of data are also subject to negotiation by various clinical or non-clinical stakeholders. These approaches will represent the lens through which we explore the creation, packaging, dissemination and ‘use’ of data for QI within the PQIP.

The evaluation will take into consideration three main dimensions of the intervention:

1. Individual and organisational ideas of change (motivations, attitudes, social climate).

2. Practices involved in the design and implementation of the programme (decision-making models, formal and informal behaviour, actions taken to incorporate the programme into practice).

3. Contextualisation (the interaction of the programme with other local and national initiatives, variability in the implementation of the programme across the organisation).

In order to explore these dimensions, the process evaluation multisited qualitative fieldwork will include two components:

1. Analysis of PQIP designers’ and implementers’ (PQIP leads and PQIP central study team) perceptions and daily practices to understand:

   1. PQIP programme theories.

   2. PQIP processes of implementation.

2. Multisited fieldwork in six NHS Trusts (five engaging in PQIP and one not engaging with PQIP) using two waves of data collection to explore:

   1. Local programme theories.

   2. Expectations of PQIP.

   3. Experiences with PQIP.

   4. The extent to which PQIP is embedded in daily practice.

The formative design will entail working with local and central teams to set up regular feedback mechanisms to help shape the programme (and Trusts’ responses to it) across its different stages. This will involve regular feedback to the PQIP project team during their meetings and to participating NHS Trustsafter each wave of data collection; dissemination of findings to wider clinical audiences including oral presentations at national meetings or articles in professional publications.

Two members of the research team are both anaesthetists and members of the PQIP project team (SRM is the project lead). In order to maintain critical distance of the research team, we have involved two other researchers (CV and NF) who are full-time university academics who are not involved in the implementation of PQIP. We will explicitly account for the subjectivity of the research team by engaging in continuous processes of reflexivity, looking for evidence of how researchers/participants acknowledge and respond to the positions of the researchers, and by triangulating between settings when researchers are present and when they are not (for example by using minutes of meetings which researchers have not observed).

Data collection

PQIP project team participants

Interviews will be carried out with three main groups: PQIP lead, project team members and PQIP fellows (see table 2). These participants will be asked to take part in two interviews, the first one at an early stage in the evaluation (2–3 months) and the second one mid-way throughout the evaluation (18 months). The observations will be carried out during regular group and team meetings.

Observations (~20 hours) with the PQIP project team will include project team meetings, Clinical Reference Group meetings and other relevant working group meetings at the Royal College of Anaesthetists.

Surgical specialties

We will focus the multisited fieldwork on a single PQIP surgical specialty, lower GI, as this is widely performed, and therefore offers greater variety when exploring context. Orthopaedic surgery (not involved in PQIP) will be studied in comparison to lower GI. Orthopaedics has been chosen because it is performed in a similarly broad range of hospitals to lower GI surgery, and both specialties have established National Clinical Audit data for both elective and emergency surgical services. In addition, a recent initiative called Getting It Right First Time (‘GIRFT’) has been piloted in orthopaedics. The GIRFT team aim to help hospitals improve quality by using site visits to support targeted self-assessment and peer review of local data, with the emphasis on reducing variation between hospitals. This approach for using data has gained traction within the Department of Health and is therefore a timely comparison to PQIP’s approach, which has the same aim but different methods.

NHS Trusts

The process evaluation aims to build a rich picture of how the context(s) of individual hospitals influence PQIP’s impact. Participating hospitals will be NHS hospitals performing both lower GI and orthopaedic surgery and planning to start, or have recently started, recruiting patients to PQIP. One non-PQIP hospital will also be recruited to enable some comparison to secular trends. The same methods for data collection and analysis will be used at all sites.

Six hospitals have been purposively recruited (table 3) to achieve diversity according to the following criteria:

• Geographical location (eg, variation across the country and balance of urban and rural).

• Academic status (ie, teaching hospitals vs District General Hospital).

• Size (eg, whether they have more or fewer than 500 beds).

• Engagement with PQIP (high to low level of patient recruitment for PQIP and non-PQIP site).

• Length of time participating in PQIP.

NHS staff participants

Interviews

The participant information sheet (PIS) will be provided to all potential interviewees at least 48 hours in advance of any planned interview. Potential participants will then be given the opportunity to discuss the PIS with the researcher and ask questions. The researchers will discuss the consent form with interviewees before requesting written informed consent from them prior to interviews. Staff will be able to withdraw consent at any time before or during interviews. In the event of consent being withdrawn after the completion of an interview, the data provided prior to withdrawal will be retained (anonymised fully) for analysis and publication.

Observations

Before attending any specific meetings, the researcher will contact and seek permission to attend from the meeting chair. The PIS will be circulated to all members with the meeting papers; this will reiterate the study details and the opportunity for members to opt out of the observation. The PIS will also be available at the beginning of the meeting. If a member declines consent, their contributions to the meeting will not be recorded in the field notes, or the researcher will withdraw from the meeting if appropriate.

For the observations, staff interacting with PQIP data will be initially approached by the researcher who will provide them with a copy of the PIS. The researcher will give the staff member a brief description of the evaluation aims, what participation entails, and the possibility of not being included in the observations. Verbal consent of the participants will be recorded. We will give staff the opportunity of not having their activities recorded in our observations and assure them that the researchers have previous experience working with this method and will withdraw from situations where it is not appropriate for observations to be carried out.

Reflexivity

The researchers will engage in a continuous process of reflexivity by documenting their own (and each other’s) assumptions, viewpoints and impacts at research sites. We will also seek any differences in participants’ behaviour between settings when researchers are present and when they are absent (for example by by comparing observations notes with minutes of meetings which researchers have not observed).