You are here

  • Home
  • Archive
  • Volume 14, Issue 12
  • Enhancing representativeness of patient-reported outcomes in routine radiation oncology care: a quality improvement protocol to address non-response

Article Text

Article menu

Download PDFPDF

Health services research
Protocol
Enhancing representativeness of patient-reported outcomes in routine radiation oncology care: a quality improvement protocol to address non-response
  1. Chengbo Zeng1,2,
  2. Neil E Martin2,3,4,
  3. Andrea L Pusic1,2,5,
  4. Maria O Edelen1,2,6,
  5. Jason B Liu1,2,7
  1. 1Patient Reported Outcomes, Value and Experience (PROVE) Center, Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA
  2. 2Harvard Medical School, Boston, Massachusetts, USA
  3. 3Department of Radiation Oncology, Brigham and Women's Hospital, Boston, Massachusetts, USA
  4. 4Department of Radiation Oncology, Dana-Farber Cancer Institute, Boston, Massachusetts, USA
  5. 5Division of Plastic and Reconstructive Surgery, Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA
  6. 6RAND Corporation, Boston, Massachusetts, USA
  7. 7Division of Surgical Oncology, Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA
  1. Correspondence to Dr Chengbo Zeng; czeng8{at}bwh.harvard.edu

Abstract

Introduction Non-response significantly undermines the representativeness of patient-reported outcome (PRO) data, thereby compromising its utility for facilitating high-value, equitable, patient-centred care in cancer clinics. Quality improvement studies are needed to assess the representativeness of PRO data collected in routine care, identify the underlying causes of non-response and develop novel methods to ensure data representativeness. Using a multilevel framework and a mixed-methods approach, we have three aims: (1) characterise the non-response of the Global-10 across clinic, provider and patient levels; (2) identify multilevel causes of non-response and potential strategies to improve representativeness in PRO collection; and (3) develop effective modifications to missing-data methods to enhance the representativeness of pre-existing PRO data.

Methods and analysis Our primary data source is the Patient Reported Outcomes Measurement Information System Global-10, collected as part of routine care at the Radiation Oncology clinics within the Mass General Brigham (MGB) healthcare system. Other sources include (1) Harvard Catalyst for provider-specific data, (2) MGB administrative data, (3) public Centers for Medicare & Medicaid Services data and (4) the National Plan and Provider Enumeration System. We will conduct quantitative analyses to assess variations in Global-10 non-response across multilevel factors. Additionally, we will use qualitative interviews with patients and clinical professionals to understand the causes of non-response and to formulate strategies to expand the reach of PRO collection to underrepresented cancer patients, improve their completions and enhance overall data representativeness. Finally, we will integrate implementation science knowledge and findings from the first two aims into missing-data methods to manage non-response in the pre-existing Global-10 data and to evaluate their performance in preserving representativeness.

Ethics and dissemination The study protocol was reviewed and approved by the Institutional Review Board at the Dana-Farber/Harvard Cancer Center (24-225). Written informed consent will be obtained from participants. Study findings will be disseminated through peer-reviewed publications and presentations at national and international conferences.

  • Patient Reported Outcome Measures
  • Radiation oncology
  • Implementation Science
  • Surveys and Questionnaires
  • STATISTICS & RESEARCH METHODS
  • QUALITATIVE RESEARCH
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi-org.ezproxy.u-pec.fr/10.1136/bmjopen-2024-097127

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    STRENGTHS AND LIMITATIONS OF THIS STUDY

    • This project will dissect non-response using a multilevel framework which has been widely applied in dissemination and implementation science but not yet commonly applied to non-response research.

    • We will examine patient-reported outcome (PRO) data, collected as part of routine care at radiation oncology clinics within Mass General Brigham healthcare system in Massachusetts, USA.

    • Our project introduces methodological innovation by integrating implementation knowledge from PRO data collection experiences into missing-data methods.

    • This is a cross-sectional study, so causal relationships cannot be determined.

    Introduction

    Patient-reported outcome (PRO) measures, or PROMs, are powerful tools in cancer care to enhance clinician-patient communication, identify problematic symptoms and treatment priorities, facilitate shared decision-making and prolong survival.1 2 Aggregated PRO data can also generate patient-centred real-world evidence to inform clinical practice and are increasingly incorporated into national public reporting and value-based healthcare initiatives as measures of care quality.3 4 Regardless of their purpose, PRO data can exist only when patients participate, and non-response emerges as a significant challenge to data representativeness.

    Compared with clinical trials, collecting PROMs in clinical care involves large-scale implementation and is susceptible to significant heterogeneity due to the absence of standardisation, limited resources, variable leadership buy-in and diverse patient populations, resulting in heterogeneity in the collected data.5 6 Furthermore, studies have shown that minoritised groups, such as racial/ethnic minorities and non-English speakers, are less likely to complete PROMs.7–12 Using PRO data that do not represent the patient population to generate evidence or evaluate care delivery threatens the validity of such efforts and can worsen healthcare disparities.13–16 For PROMs to continue to promote high-quality and equitable patient-centred cancer care, it is crucial to identify the underlying causes of non-response and to develop novel methods to ensure data representativeness.

    Research into the implementation of PROMs in clinical care has revealed many patient-, provider- and clinic-level factors that can affect PROM completion rates, such as provider engagement and infrastructure support.17–21 Although identifying the barriers and facilitators to PROMs collection is important, higher collection rates do not necessarily imply better data representativeness. Strategies aimed at increasing collection may differ from those designed to enhance representativeness. Yet, most current research focuses on methods to increase collection instead of methods to enhance PRO data representativeness. Multilevel frameworks can help to not only characterise non-response but to also identify determinants that can improve the representativeness of routinely collected PRO data.22–25 These structured frameworks can simultaneously be used to develop mitigation strategies to target the identified determinants, thereby addressing PRO data representativeness efficiently and effectively.22–25

    The overarching goal of this project is to delineate potential causes of non-response and to devise mitigation strategies to improve the representativeness of PRO data collected as part of routine radiation oncology care. To achieve this, we will examine Patient Reported Outcomes Measurement Information System (PROMIS) Global-10 (Global-10) data, collected as part of routine care at the radiation oncology clinics within the Mass General Brigham (MGB) healthcare system, a large, integrated health system in Massachusetts. Using a multilevel framework and a mixed-methods approach, this project aims to:

    • Aim 1: characterise the non-response of the Global-10, collected as part of routine care, across clinic, provider, and patient levels.

    • Aim 2: identify multilevel causes of non-response and potential strategies to improve representativeness in PROMs collection.

    • Aim 3: develop effective modifications to missing-data methods to enhance the representativeness of pre-existing PRO data.

    Methods and analysis

    Project overview

    The approach consists of a multilevel framework derived from the Consolidated Framework for Implementation Research (CFIR) and a mixed-methods strategy. Figure 1 displays the project overview. In aim 1, we will assess variations in non-response of Global-10 across multilevel factors to identify key areas of concern and organise our findings. In aim 2, we will use qualitative interviews to understand the causes of non-response across the key areas identified from aim 1 and to formulate strategies that can expand the reach of PROMs collection to underrepresented cancer patients, improve their PROMs completions and enhance overall PRO data representativeness. In aim 3, we will incorporate implementation science knowledge and findings from the first two aims into missing-data methods to manage non-response of the pre-existing Global-10 data and to evaluate their performance in preserving representativeness. In the final stages of aim 3, we will compare the results yielded from different missing-data methods. This comparison will not only validate the robustness of our findings but also highlight the most effective modifications to missing-data methods to protect representativeness in PRO data with non-response.

    Figure 1
    Figure 1

    Project overview. PRO, patient-reported outcome; PROMs, PRO measures.

    Conceptual framework

    To ensure thorough investigations into non-response and to facilitate the development of actionable mitigation strategies, this project draws from the CFIR, which has been used to guide the systematic evaluation of determinants in the design and implementation of PROMs collection in clinical care.23 The CFIR encompasses five major domains: intervention characteristics, outer setting (eg, patient needs, external policies), inner setting (eg, institute characteristics, implementation climate and accessible information and technology), characteristics of the individuals involved and implementation process.23–25 Our project focuses on the inner setting and characteristics of the individuals involved (ie, patients and providers) domains. We perceive the inner setting to include factors at the clinic level, such as facility characteristics and implementation climate, which are associated with leadership engagement, available resources and accessible knowledge essential for PROMs collection in diverse populations and problem-solving.20 21 Providers’ clinical workload and annual PROMs collection affect their willingness and ability to collect PROMs, address non-response and reach minority groups.19 21 Patients’ demographic characteristics, socioeconomic status and health status influence their motivation to complete PROMs.7–11 The multilevel framework will guide the design of quantitative analyses (aims 1 and 3) to uncover the landscape of non-response and aid in the development of qualitative interview guides (aim 2) to solicit knowledge on improving representativeness in data collection across clinic, provider and patient levels.

    Study setting

    We will use Global-10 data collected as part of routine care in MGB radiation oncology clinics. MGB is an integrated health system comprised of two academic medical centres as well as eight community hospitals and a large network of community-based physician office practices. In 2012, MGB initiated a standardised PROMs collection programme for routine clinical specialty care, offering participating physician practices and clinics the necessary technical infrastructure, personnel training, continuous education and operational resources for successful implementation.20 26 27 To date, the MGB PROMs programme accrues more than 6 million PROMs per year across more than 475 clinics from more than 80 medical, behavioural health and surgical specialties, representing the largest repository of multispecialty PRO data in the USA.

    The MGB radiation oncology practices uniformly implemented PROMs collection across all clinic sites and remain active today. Because radiation therapy is a fundamental component of cancer care across nearly all cancer types for curative or palliative purposes, radiation oncology serves as a prime specialty from which to conduct our analyses to yield generalisable results applicable across the cancer care continuum. Furthermore, MGB radiation oncology clinics are in different settings (eg, academic vs community) and geographical locations and serve a diverse patient population, allowing for better capture of variations in sex, race/ethnicity and socioeconomic status.

    Data sources

    Multiple data sources will be merged to conduct this study (table 1).28–30 Our primary data source is MGB enterprise data warehouse (EDW), a system-wide data resource that contains harmonised medical, billing, claims and financial data, including PROMs data.

    View this table:
    Table 1

    Proposed multilevel factors for the domains of inner setting and individual characteristics

    Aim 1: characterise the non-response of the Global-10 across clinic, provider and patient levels

    Study design and data analysis

    Preliminary data show Global-10 completions from a total of 34 516 patients with 68 933 submissions representing 70% of the individuals who were assigned the Global-10 in the MGB radiation oncology clinics since the start of data collection in January 2015, with the lowest rate during the COVID-19 pandemic. Given that all patients are assigned the Global-10 as part of their initial visit to radiation oncology clinics, we will conduct a cross-sectional study focusing on the characterisation of Global-10 non-response. Additionally, due to the probable differences in non-response mechanisms between children and adults, this proposal will focus only on adult patients (those aged 18 years or older). In the MGB radiation oncology clinics, patients must complete all assigned items (ie, 10 items on Global-10) to finalise their submissions and obtain the summary scores (ie, global physical and mental health).31 Therefore, we will classify the completion to the assigned Global-10 as response and non-response. Response is defined as patients who completed all 10 items on Global-10. Non-response includes patients who missed at least one of the 10 items on Global-10. Within each clinic, we will calculate the overall frequencies and percentages for respondents and non-respondents, respectively. Furthermore, we will describe the distributions of respondents and non-respondents by factors at patient and provider levels for the overall sample and by each clinic, with a specific focus on underrepresented populations such as racial/ethnic minorities, older age, less-educated individuals and non-English speakers (table 1).

    We will identify the predictors of non-response. Our primary hypothesis is that there are significant relationships between non-response and patients’ demographic characteristics (eg, being a racial/ethnic minority, being of an older age, being a non-English speaker and having a low income), providers’ data collection proficiency and the history of the PROMs programme in the clinics.7–11 18–21 Using bivariate analyses, we will assess the differences in distributions among respondents and non-respondents across multilevel factors. Factors with p values less than 0.20 will enter the multivariable model. For factors outside this threshold, we will refer to existing literature to determine their appropriateness for inclusion in multivariable analyses. We will use multinomial logistic regression and stepwise selection to identify significant predictors of non-response, adjusting for the random effects of clinics and providers.

    Summary

    By analysing Global-10 data from the MGB radiation oncology clinics, we will characterise the heterogeneity in non-response of routinely collected PRO data across clinic-, provider- and patient-level factors. This will not only facilitate our understanding of critical areas of concern but also guide our research in developing strategies to improve representativeness for both PROMs collection and PRO data analysis.

    Aim 2: identify multilevel causes of non-response and potential strategies to improve representativeness in patient-reported outcome measures (PROMs) collection

    Study design and semi-structured interview guides

    We will conduct two separate focus groups with clinical professionals (ie, clinic leaders and providers) and approximately twenty-five 1:1 in-depth interviews with cancer patients in radiation oncology clinics. Clinic leaders and providers will be purposively recruited. Eligible clinic leaders include chief-level executives, administrative leaders, PROMs directors and clinician leaders.18 Provider participants must have been actively involved in PROMs collection in the past year. Regarding patients, we will recruit adults aged 18 years or older who are receiving care. We will employ quota sampling to ensure the inclusion of sex, race, ethnicity and other minority groups. Patients will be recruited with the help of radiation oncology providers or through electronic communication. We will exclude patients with documented cognitive impairment or those unable to provide informed consent.

    Building on the findings from aim 1 and in accordance with our multilevel framework (figure 2 and table 1), we will develop semistructured interview guides tailored for clinical professionals (ie, clinic leadership and providers) and patients. For clinical professionals, we will explore (1) their perceived issues and causes of non-response and (2) potential strategies to enhance representativeness in data collection. For patients, we will solicit their insights regarding (1) PROMs collection overall, (2) the main challenges they face when deciding whether to complete PROMs and (3) potential solutions from their perspectives (eg, technology, rationale, etc). Both interview guides will be pilot tested with three clinical professionals and three patients before their use.

    Figure 2
    Figure 2

    Multilevel framework of PROMs non-response. PROMs, patient-reported outcome measures.

    Data collection and analysis

    For clinical professionals, we will collect demographic characteristics, such as age, sex, job titles, years of experience in their respective fields and history of PROMs collection. We will conduct focus groups separately to foster open discussion and gather a rich variety of perspectives. We plan to recruit seven clinic leaders and 10 providers, specifically from clinics exhibiting the lowest and highest rates of non-response. We believe that two focus group interviews, with at least five clinical professionals in each, will achieve information saturation.32

    For cancer patients, we will collect demographic characteristics and medical information (eg, cancer stage, treatment type and comorbidities) through the MGB EHR and pre-interview surveys as needed. The research team will conduct 1:1 in-depth interviews with cancer patients to understand their experiences and insights. We estimate that each interview will last 45–60 min, and approximately 25 patient interviews will be sufficient to reach thematic saturation.33 A $50 gift card will be provided to each patient for remuneration.

    Quantitative data will be summarised descriptively. All interviews will be professionally transcribed. The research team will analyse qualitative data using the content analysis approach, independently and in conference to facilitate rigour. Discrepancies in coding between team members will be discussed and arbitrated by a third party as necessary to reach consensus. Deductive coding will be used to map the themes to the components of our multilevel framework (figure 2 and table 1) and strategies addressing the causes of non-response across clinic, provider and patient levels. Inductive coding will be used for new themes. We will use NVivo software for data analysis and management and follow consolidated criteria for reporting qualitative research to draft results.

    Summary

    The qualitative study in aim 2 will deepen our understanding of the issues and causes of non-response from both clinical professionals’ and patients’ perspectives to uncover mitigation strategies. Importantly, these strategies can effectively address the issue of ‘missing not at random (MNAR)’.34 The MNAR mechanism is the most difficult to address as it assumes that non-response is related to both observed (eg, patients’ characteristics) and unobserved factors.34 Unlike other types of missing-data mechanisms which will be evaluated in aim 3, non-response operated under MNAR cannot be fixed with post hoc statistical manoeuvres, thus uncovering its potential causes during data collection will provide valuable information.

    Aim 3: develop effective modifications to missing-data methods to enhance the representativeness of pre-existing patient-reported outcome (PRO) data

    Aside from MNAR, the most probable mechanism for non-response of PRO data collected in routine cancer care is ‘missing at random (MAR)’, given the established relationships between non-response and the demographic and clinical characteristics recorded in patients’ medical records. MAR assumes that non-response is related to observed variables.34 Advanced missing-data methods, such as Hot-deck imputation, multiple imputation (MI) and inverse probability weighting (IPW), are recommended for addressing MAR-based non-response.34–37 However, these methods can sometimes result in extreme values or unbalanced weights, making it challenging to fully establish representative data for analysis.38–41 We will seek to improve on these approaches by incorporating multilevel factors based on our multilevel framework. A notable feature of Hot-deck imputation and MI is their ability to incorporate auxiliary variables—those related to outcomes or non-response but not part of the main analyses—to enhance their performance, making them well-suited to modification based on multilevel factors.35–37 42 IPW adjusts for non-response bias using complete cases without imputation.40 Another mechanism for non-response is ‘missing completely at random (MCAR)’, which assumes that non-response is unrelated to any variables.34 MCAR is an unrealistic assumption in routinely collected PRO data as we know non-response bias exists. Therefore, in aim 3, we will explore the solutions to MAR that incorporate the multilevel factors inherently influencing PROMs data collection.

    Study design and data analysis

    We will integrate determinants of non-response, derived from (1) our multilevel framework and (2) findings from the first two aims, into the missing data analysis. The goal is to modify missing-data methods to improve their performance in preserving the representativeness of routinely collected PRO data. We will employ six missing-data methods to address the non-response of pre-existing Global-10 data (table 2). Under the MCAR assumption, listwise deletion will be used as a standard approach for comparison, while Hot-deck imputation, MI and IPW will be used for MAR. Specifically, when applying Hot-deck imputation, MI and IPW, multilevel factors of non-response will be integrated into the analyses. This integration enables us to investigate whether the performance of these methods in achieving representativeness can be enhanced. Notably, although Hot-deck imputation, MI and IPW are primarily recommended for non-response under the MAR assumption, they also effectively address non-response under the MCAR assumption, and thus we will examine both assumptions in this proposal for completeness.34 35

    View this table:
    Table 2

    Missing-data mechanisms and proposed methods

    Performance evaluation

    We aim to rigorously assess the performance of various missing-data methods to enhance the representativeness of Global-10 data, in both scenarios where information from successful PROMs collection is available and where it is absent. Our assessment will comprise a comprehensive comparative analysis focusing on (1) demographic characteristics, (2) summary scores and (3) predictive validity between the complete cases and the overall sample after adjustment for each missing data technique.31 43–45 For demographic characteristics, we will compare the distributions of sex, age, race, ethnicity, language, education, employment status and financial insurance type. Our goal is to determine whether the sample post-adjustment reflects the demographic distributions observed in the overall population who have been assigned the Global-10.43 45 For physical and mental health summary scores, we will describe and compare their distributions (eg, mean, SD, minimum and maximum) between complete cases and the post-adjustment sample. Favourable missing-data methods should produce summary scores with fewer outliers, smaller SD and lower mean values, especially since we account for the non-response in minority groups and cancer patients with advanced diseases and poor health status.44 46 47 Regarding the predictive validity, we will use logistic regression to evaluate the ability of physical and mental health summary scores to predict healthcare utilisation (eg, urgent care visits, hospitalisations, etc) or death.44 Our hypothesis is that after accounting for the non-response, the adjusted summary scores will have stronger correlations with healthcare utilisation or death than the scores from the complete cases.

    Summary

    By developing and examining modifications to statistical methods for handling missing data, aim 3 can identify effective strategies to address non-response and increase representativeness in the analysis of pre-existing, routinely collected PRO data. Given the large volume of such data and significant heterogeneity in its quality, the findings from aim 3 can guide the selection of appropriate statistical approaches and key determinants of non-response to improve representativeness to the extent possible when utilising pre-existing PRO data for quality improvement and patient-centred cancer care.

    Patient and public involvement

    None.

    Ethics and dissemination

    This study has been approved by the Dana-Farber/Harvard Cancer Center (DF/HCC) IRB (24-225). The research team will take all necessary steps possible to protect participants from the few minimal risks potentially associated with the study. All patient identifiers will be removed prior to the analysis. Throughout all project activities, the study team members will adhere to all MGB policies, standards and procedures, as well as any data use agreements related to specific data sources. We will continue to protect confidentiality and prevent inappropriate access, use or disclosure of data.

    In the qualitative studies, potential participants will be informed about the project, and the research staff will confirm their eligibility, provide additional study details and answer any questions they may have. If a potential participant is willing to enrol, the study staff will obtain their written informed consent. Focus group interviews with clinical professionals will be conducted via a secure video-conferencing platform. In-depth interviews with patients will be conducted in a private room or conducted remotely using a secure video-conferencing platform or telephone, according to participant preference. All audio/video recordings, transcripts, surveys and demographic forms for qualitative studies will be stored on secure, encrypted servers on password-protected computers, accessible only by the study research staff. The qualitative data will be collected using password-protected digital recorders. During transcription, all identifying information that could be used to link the data with the participant will be de-identified. Audio files will be destroyed once transcribed, and no personal identifiers will be linked to the transcripts. Study participants will not be identified in any reports, presentations or publications resulting from this study. All quantitative and qualitative data will be stored on encrypted study computers, and all analyses will be conducted at Brigham and Women’s Hospital, where the study database will be hosted.

    Study findings will be disseminated through peer-reviewed publications and presentations at national and international conferences.

    This is a 2-year study funded by the National Cancer Institute of the National Institutes of Health. The study team is currently preparing for data extraction and curation from the MGB EDW and other sources. We plan to dedicate 6 months to aim 1, 9 months to aim 2 and 6 months to aim 3. The final 3 months of the project will focus on manuscript development. The anticipated date of completion is July 2026.

    Discussion

    Our project is innovative in at least four ways. First, we will dissect non-response using a multilevel framework which has been widely applied in dissemination and implementation science but not yet commonly applied to non-response research. Second, the wealth of data available to us through MGB radiation oncology clinics, encompassing clinic-, provider- and patient-level factors related to non-response, is novel in itself. Third, with no established guidelines for reporting and analysing routinely collected PRO data with non-response, our project pioneers the characterisation of PROMs completion by varied degrees of non-response across clinic-, provider- and patient-level factors. This approach can disentangle the intricate landscape of non-response in cancer clinics and facilitate the assessment of relevant causes. Such insights have the potential to significantly impact future non-response mitigation during PROMs collection as well as robustly address its statistical management in the analysis of pre-existing, routinely collected PRO data. Lastly, our project introduces methodological innovation by integrating implementation knowledge from PROMs collection experiences into missing-data methods. This integration aims to scrutinise the effectiveness of these methods in preserving the representativeness of routinely collected PRO data despite non-response, setting a precedent in methodological innovation.

    There are several potential obstacles and limitations that warrant discussion. First, recruiting clinical professionals for qualitative interviews may pose a challenge due to their demanding schedules. To address this, we will liberally conduct focus group interviews via secure teleconferencing platforms outside of normal business hours. Second, recruiting underrepresented cancer patients may also be challenging, with some minority groups potentially reluctant to participate. If this occurs, we will increase the incentives. Third, there might be a lack of direct data sources for some aggregated level factors in our multilevel framework. In such cases, we will collect information through MGB administrative or human resources data or direct contact with clinical professionals. Fourth, our currently developed multilevel framework may not cover all potential factors. We will continue to add additional variables based on available databases and current evidence as the project ensues. Finally, as this is a cross-sectional study, causal relationships cannot be determined.

    The completion of this project will elucidate the characteristics of non-response and its intricate associations with multilevel factors of successful, large-scale PROMs collection in diverse patient populations. Insights from this endeavour will guide the evolution and development of PROMs collection programmes to expand their reach to underrepresented cancer patients and improve data representativeness. This, in turn, enables the utility of PRO data for quality improvement and high-quality, equitable patient–centred cancer care.

    Ethics statements

    Patient consent for publication

    References

    1. ↵
      1. Basch E,
      2. Deal AM,
      3. Kris MG, et al
      . Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol 2016;34:557–65. doi:10.1200/JCO.2015.63.0830
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Basch E,
      2. Schrag D,
      3. Henson S, et al
      . Effect of Electronic Symptom Monitoring on Patient-Reported Outcomes Among Patients With Metastatic Cancer: A Randomized Clinical Trial. JAMA 2022;327:2413–22. doi:10.1001/jama.2022.9265
      OpenUrlCrossRefPubMed
    3. ↵
      1. Klootwijk A,
      2. Bakx P,
      3. Franx A, et al
      . Patient-reported outcome and experience measures for quality improvement in pregnancy and childbirth care: a retrospective cohort study. BMJ Open Qual 2023;12:e001922. doi:10.1136/bmjoq-2022-001922
    4. ↵
      1. Saraswathula A,
      2. Merck SJ,
      3. Bai G, et al
      . The Volume and Cost of Quality Metric Reporting. JAMA 2023;329:1840–7. doi:10.1001/jama.2023.7271
      OpenUrlCrossRefPubMed
    5. ↵
      1. Basch E,
      2. Bennett AV
      . Patient-reported outcomes in clinical trials of rare diseases. J Gen Intern Med 2014;29 Suppl 3:S801–3. doi:10.1007/s11606-014-2892-z
      OpenUrl
    6. ↵
      1. Snyder CF,
      2. Aaronson NK,
      3. Choucair AK, et al
      . Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res 2012;21:1305–14. doi:10.1007/s11136-011-0054-x
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Bernhard J,
      2. Cella DF,
      3. Coates AS, et al
      . Missing quality of life data in cancer clinical trials: serious problems and challenges. Stat Med 1998;17:517–32. doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<517::aid-sim799>3.0.co;2-s
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Yu H,
      2. Yu Q,
      3. Nie Y, et al
      . Data Quality of Longitudinally Collected Patient-Reported Outcomes After Thoracic Surgery: Comparison of Paper- and Web-Based Assessments. J Med Internet Res 2021;23:e28915. doi:10.2196/28915
    9. ↵
      1. Adewuyi TE,
      2. MacLennan G,
      3. Cook JA
      . Non-compliance with randomised allocation and missing outcome data in randomised controlled trials evaluating surgical interventions: a systematic review. BMC Res Notes 2015;8:403. doi:10.1186/s13104-015-1364-9
    10. ↵
      1. Hutchings A,
      2. Neuburger J,
      3. Grosse Frie K, et al
      . Factors associated with non-response in routine use of patient reported outcome measures after elective surgery in England. Health Qual Life Outcomes 2012;10:34. doi:10.1186/1477-7525-10-34
    11. ↵
      1. Srour MK,
      2. Tadros AB,
      3. Sevilimedu V, et al
      . Who Are We Missing: Does Engagement in Patient-Reported Outcome Measures for Breast Cancer Vary by Age, Race, or Disease Stage? Ann Surg Oncol 2022;29:7964–73. doi:10.1245/s10434-022-12477-1
      OpenUrl
    12. ↵
      1. Kim E,
      2. Murray A,
      3. Tolstorukov M, et al
      . Patient Reported Outcome Measure (PROM) Collection Rates When Part of Routine Clinical Workflow in an Academic Radiation Oncology Department. Int J Radiat Oncol Biol Phys 2021;111:e163. doi:10.1016/j.ijrobp.2021.07.637
    13. ↵
      1. Boulware LE,
      2. Purnell TS,
      3. Mohottige D
      . Systemic Kidney Transplant Inequities for Black Individuals: Examining the Contribution of Racialized Kidney Function Estimating Equations. JAMA Netw Open 2021;4:e2034630. doi:10.1001/jamanetworkopen.2020.34630
    14. ↵
      1. Agency for Healthcare Research and Quality
      . Disparities. 2018. Available: https://www.ahrq.gov/topics/disparities.html#accordions
    15. ↵
      1. Amercian Medical Association
      . Reducing disparities in health care. 2019. Available: https://www.ama-assn.org/delivering-care/patient-support-advocacy/reducing-disparities-health-care
    16. ↵
      1. Calvert MJ,
      2. Cruz Rivera S,
      3. Retzer A, et al
      . Patient reported outcome assessment must be inclusive and equitable. Nat Med 2022;28:1120–4. doi:10.1038/s41591-022-01781-8
      OpenUrl
    17. ↵
      1. Mercieca-Bebber R,
      2. Palmer MJ,
      3. Brundage M, et al
      . Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review. BMJ Open 2016;6:e010938. doi:10.1136/bmjopen-2015-010938
    18. ↵
      1. Mou D,
      2. Mjaset C,
      3. Sokas CM, et al
      . Impetus of US hospital leaders to invest in patient-reported outcome measures (PROMs): a qualitative study. BMJ Open 2022;12:e061761. doi:10.1136/bmjopen-2022-061761
    19. ↵
      1. Mou D,
      2. Horn DM,
      3. Heng M, et al
      . Primary care physician’s (PCP) perceived value of patient-reported outcomes (PROs) in clinical practice: a mixed methods study. BMJ Qual Saf 2022;31:221–9. doi:10.1136/bmjqs-2020-012206
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Sisodia RC,
      2. Dankers C,
      3. Orav J, et al
      . Factors Associated With Increased Collection of Patient-Reported Outcomes Within a Large Health Care System. JAMA Netw Open 2020;3:e202764. doi:10.1001/jamanetworkopen.2020.2764
    21. ↵
      1. Palmer MJ,
      2. Mercieca-Bebber R,
      3. King M, et al
      . A systematic review and development of a classification framework for factors associated with missing patient-reported outcome data. Clin Trials 2018;15:95–106. doi:10.1177/1740774517741113
      OpenUrl
    22. ↵
      1. Al Sayah F,
      2. Lahtinen M,
      3. Bonsel GJ, et al
      . A multi-level approach for the use of routinely collected patient-reported outcome measures (PROMs) data in healthcare systems. J Patient Rep Outcomes 2021;5:98. doi:10.1186/s41687-021-00375-1
    23. ↵
      1. Damschroder LJ,
      2. Aron DC,
      3. Keith RE, et al
      . Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Sci 2009;4:50. doi:10.1186/1748-5908-4-50
      OpenUrl
    24. ↵
      1. Ahmed S,
      2. Zidarov D,
      3. Eilayyan O, et al
      . Prospective application of implementation science theories and frameworks to inform use of PROMs in routine clinical care within an integrated pain network. Qual Life Res 2021;30:3035–47. doi:10.1007/s11136-020-02600-8
      OpenUrl
    25. ↵
      1. Stover AM,
      2. Haverman L,
      3. Oers HA, et al
      . ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 2021;30:3015–33. doi:10.1007/s11136-020-02564-9
      OpenUrl
    26. ↵
      1. Sisodia RC,
      2. Rodriguez JA,
      3. Sequist TD
      . Digital disparities: lessons learned from a patient reported outcomes program during the COVID-19 pandemic. J Am Med Inform Assoc 2021;28:2265–8. doi:10.1093/jamia/ocab138
      OpenUrl
    27. ↵
      1. Liu JB,
      2. Kaplan RS,
      3. Bates DW, et al
      . Mass General Brigham’s Patient-Reported Outcomes Measurement System: A Decade of Learnings. NEJM Catalyst 2024;5. doi:10.1056/CAT.23.0397
    28. ↵
      National plan & provider enumeration system. 2023. Available: https://nppes.cms.hhs.gov/#/
    29. ↵
      Harvard catalyst. 2023. Available: https://catalyst.harvard.edu/
    30. ↵
      Centers for medicare & medicaid services. 2023. Available: https://www.cms.gov/
    31. ↵
      1. Hays RD,
      2. Bjorner JB,
      3. Revicki DA, et al
      . Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Qual Life Res 2009;18:873–80. doi:10.1007/s11136-009-9496-9
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Hennink MM,
      2. Kaiser BN,
      3. Weber MB
      . What Influences Saturation? Estimating Sample Sizes in Focus Group Research. Qual Health Res 2019;29:1483–96. doi:10.1177/1049732318821692
      OpenUrlCrossRefPubMed
    33. ↵
      1. Vasileiou K,
      2. Barnett J,
      3. Thorpe S, et al
      . Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period. BMC Med Res Methodol 2018;18:148. doi:10.1186/s12874-018-0594-7
    34. ↵
      1. Enders CK
      . Applied Missing Data Analysis. Second Edition. The Guiford Press, 2022.
    35. ↵
      1. Pedersen AB,
      2. Mikkelsen EM,
      3. Cronin-Fenton D, et al
      . Missing data and multiple imputation in clinical epidemiological research. Clin Epidemiol 2017;9:157–66. doi:10.2147/CLEP.S129785
      OpenUrlCrossRefPubMed
    36. ↵
      1. Hardt J,
      2. Herke M,
      3. Leonhart R
      . Auxiliary variables in multiple imputation in regression with missing X: a warning against including too many in small sample research. BMC Med Res Methodol 2012;12:184. doi:10.1186/1471-2288-12-184
    37. ↵
      1. Myers TA
      . Goodbye, Listwise Deletion: Presenting Hot Deck Imputation as an Easy and Effective Tool for Handling Missing Data. Commun Methods Meas 2011;5:297–310. doi:10.1080/19312458.2011.624490
      OpenUrlCrossRef
    38. ↵
      1. Penny KI,
      2. Jolliffe IT
      . Multivariate outlier detection applied to multiply imputed laboratory data. Stat Med 1999;18:1879–95. doi:10.1002/(sici)1097-0258(19990730)18:14<1879::aid-sim225>3.0.co;2-6
      OpenUrlCrossRefPubMedWeb of Science
    39. ↵
      1. Templ M
      . Enhancing Precision in Large-Scale Data Analysis: An Innovative Robust Imputation Algorithm for Managing Outliers and Missing Values. Math 2023;11:2729. doi:10.3390/math11122729
      OpenUrl
    40. ↵
      1. Seaman SR,
      2. White IR
      . Review of inverse probability weighting for dealing with missing data. Stat Methods Med Res 2013;22:278–95. doi:10.1177/0962280210395740
      OpenUrlCrossRefPubMed
    41. ↵
      1. Raad H,
      2. Cornelius V,
      3. Chan S, et al
      . An evaluation of inverse probability weighting using the propensity score for baseline covariate adjustment in smaller population randomised controlled trials with a continuous outcome. BMC Med Res Methodol 2020;20:70. doi:10.1186/s12874-020-00947-7
    42. ↵
      1. Shrive FM,
      2. Stuart H,
      3. Quan H, et al
      . Dealing with missing data in a multi-question depression scale: a comparison of imputation methods. BMC Med Res Methodol 2006;6:57. doi:10.1186/1471-2288-6-57
    43. ↵
      1. Branham DK,
      2. Finegold K,
      3. Chen L, et al
      . Trends in Missing Race and Ethnicity Information After Imputation in HealthCare.gov Marketplace Enrollment Data, 2015-2021. JAMA Netw Open 2022;5:e2216715. doi:10.1001/jamanetworkopen.2022.16715
    44. ↵
      1. Pak SS,
      2. Miller MJ,
      3. Cheuy VA
      . Use of the PROMIS-10 global health in patients with chronic low back pain in outpatient physical therapy: a retrospective cohort study. J Patient Rep Outcomes 2021;5:81. doi:10.1186/s41687-021-00360-8
    45. ↵
      1. Tsiampalis T,
      2. Panagiotakos DB
      . Missing-data analysis: socio- demographic, clinical and lifestyle determinants of low response rate on self- reported psychological and nutrition related multi- item instruments in the context of the ATTICA epidemiological study. BMC Med Res Methodol 2020;20:148. doi:10.1186/s12874-020-01038-3
    46. ↵
    47. ↵
      1. Seneviratne MG,
      2. Bozkurt S,
      3. Patel MI, et al
      . Distribution of global health measures from routinely collected PROMIS surveys in patients with breast cancer or prostate cancer. Cancer 2019;125:943–51. doi:10.1002/cncr.31895
      OpenUrl
    View Abstract

    Footnotes

    • Contributors CZ is the guarantor. CZ and JBL are the principal investigators of this project and led the study design. CZ, JBL and MOE contributed to the conception and design of the study. CZ led the writing of this protocol manuscript. JBL, MOE, ALP and NEM contributed significantly to the editing of this manuscript. All authors contributed to the editing and final approval of the protocol.

    • Funding CZ, JBL and MOE were supported by the National Cancer Institute of the National Institutes of Health under Award Number R03CA292987. The funders had no role in the conceptualization, analysis or presentation of findings of this study.

    • Competing interests Outside of the submitted work, ALP is the co-developer of the Q-Portfolio measures and receives royalties when used for commercial purposes. ALP, CZ and MOE are supported in part by the National Cancer Institute for unrelated work. All other authors have nothing to disclose.

    • Patient and public involvement Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.

    • Provenance and peer review Not commissioned; peer reviewed for ethical and funding approval prior to submission.