Article Text
PDF
Abstract
Background Benefit–risk assessment (BRA) is used in multiple phases along the health technology’s life-cycle to evaluate the balance between the benefits and risks, as it is fundamental to all stakeholders. BRA and its methodological approaches have been applied primarily in the context of regulatory agencies. However, BRA’s application and extent in the context of health technology assessment (HTA) bodies remain less clear. Our goal is to perform a scoping review to identify and map methodological guidelines and publications on methods of BRA. This will be done considering the different phases of the life-cycle of health technologies to underline both the depth and extent of research concerning BRA, especially in the context of HTA.
Methods and analysis This scoping review protocol was developed following the framework proposed by Arksey and O’Malley, and the updated guidelines by the Joanna Briggs Institute. We will include methodological publications that provide recommendations or guidelines on methods for BRA. We will conduct electronic searches on Medline (PubMed) and EMBASE (Ovid) databases; manual searches on the main websites of HTA bodies and drug regulatory organisations; and contact experts in the field. Systematic extraction forms will be used to screen and assess the identified publications by independent assessors. We will provide a qualitative synthesis using descriptive statistics and visual tools. Results will be summarised in systematic evidence tables and comparative evidence scoping charts.
Ethics and dissemination This review will use data publicly available and does not require ethics approval. The results of this scoping review will contribute to scientific knowledge and act as a basis for methodologists, guideline developers and researchers for the development of BRA to inform regulatory decisions, reimbursement and coverage decision making. The results will be disseminated through peer-reviewed articles, conferences, policy briefs and workshops.
Trial registration number Open Science Framework (https://doi-org.ezproxy.u-pec.fr/10.17605/OSF.IO/69T3V).
- decision making
- public health
- statistics & research methods
- health policy
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
First scoping review aiming at mapping methodological guidelines and publications on methods of benefit–risk assessment, especially in the health technology assessment context.
We will use the framework proposed by Arksey and O’Malley and the refinements made by the Joanna Briggs Institute.
We will perform an electronic search on the main databases as well as manual searches on a vast source of grey literature, including health technology assessment bodies and networks, and regulatory agencies' websites.
Despite our attempt to conduct a comprehensive search, we may miss documents reporting methodological recommendations or guidelines for benefit–risk assessment methods used in practice and not available in the publicly available online literature.
It is expected that the lack of consistent classification and high variability of benefit–risk assessment frameworks and methods used in different contexts and technologies will increase the complexity of extraction and synthesis.
Introduction
Benefit–risk assessment (BRA), also called benefit–harm assessment, is used in multiple phases along a health technology’s life-cycle in a dynamic way to evaluate the balance between its expected benefits and risks (or harms) and it is fundamental to all stakeholders, including patients, healthcare providers, regulators, payers and pharmaceutical companies.1–4 Over the past two decades, the BRA has been used predominantly in regulatory agencies to estimate the benefit–risk or benefit–harm balance of health technology for marketing authorisation and surveillance; to estimate the benefit–risk balance of the new technology as compared with existing ones, as part of the health technology assessment (HTA) process to support decision making on pricing and reimbursement; or to support clinical decisions at patient-level.1 2 Although different stakeholders are interested in determining the benefit–risk balance, they may have different perspectives, values and priorities; subsequently, the assessment methodologies may differ depending on the context.5
Structured BRA processes have been developed to support decision making where multiple outcomes need to be considered. Usually, BRA processes employ framing to clearly define the scope and the decision problem, in a transparent, rational and consistent way.3 6 Such approaches can help establish internal standards for decision making, enhancing the clarity of the decision-making process, encouraging the use of appropriate documentation, and providing consistency of communication and visualisation of benefits and risks to various stakeholders.3
Methodological guidelines for BRA in the context of marketing authorisation have been launched to provide all stakeholders with more accurate and useful information supporting continuous BRA.1 7 8 Several initiatives and methodological publications addressed different frameworks and methodological approaches for BRA, provided recommendations for the conduction of BRA, and incorporated patients’ perspectives into such assessments and tools to present the results of BRA.3–5 7–10 However, most of them were discussed and applied in the context of regulatory decisions and may fail to incorporate the particularities of BRA for the HTA decision-making process.
In a review of key publications and initiatives on frameworks and methodologies for BRA, Mt-Isa et al11 highlighted that by 2013 there were few initiatives within the HTA context. One of these is the initiative of the European Network for Health Technology Assessment (EUnetHTA) and the European Medicines Agency (EMA), which aimed to place the decision-analytic models within the context of BRA to provide more integrated tools.12 The collaboration between EUnetHTA and EMA can foster exchanges to reduce duplication of efforts between regulators and HTA bodies, enhance mutual understanding of evidence needs, facilitate more efficient allocation of resources, improve the efficiency of decision-making processes and enhance access to health technologies.13
However, there are differences in the types of approaches and principles between methods of BRA used for regulatory and HTA decisions,1 14 and the use of BRA and the implementation of such collaboration in the context of HTA is still further to go.1 While BRA done by regulatory agencies tends to address the clinical efficacy and safety perspectives, its application in HTA reports may be broader and is still a work in progress. The systematisation and increased transparency associated with the use of structured BRA frameworks could benefit HTA by not only assessing the balance between benefits and risks (which might include aspects beyond efficacy and safety) but also addressing uncertainties and introducing quantitative parameters for assessment, particularly in cases of data-scarce scenarios. Also, BRA can play a central role in postcommercialisation monitoring studies and adaptive HTA.15 In summary, expanding the scope of BRA and incorporating these elements can improve and fortify the collaboration between regulatory agencies and HTA authorities, resulting in more comprehensive, transparent and informed evaluations of health technologies.16
There is still a gap in the literature regarding the availability of methodological guidelines and publications on methods for BRA and their characteristics in the context of HTA and at population decision-making levels. Therefore, this scoping review aims to map the available methodological guidelines and publications of methods for BRA in the different phases of the life-cycle of health technologies to underline both the depth and extent of research concerning BRA, especially in the context of HTA. Our goal is to identify which definitions have been used, describe BRA approaches (frameworks, metrics, estimation techniques and utility survey techniques) and discuss methodological characteristics of existing quantitative and descriptive approaches for assessing benefits and risks, focusing on quantitative methods. With our findings, we will support researchers, methodologists and health organisations in developing, conducting and reporting BRA.
This review represents the first phase in a larger project to improve BRA use in the context of the Brazilian HTA bodies. Through a partnership with Rede Brasileira de Avaliação de Tecnologias em Saúde—REBRATS, a strategic network built to facilitate the elaboration and dissemination of priority HTA studies for the Brazilian health system,17 this partnership will provide methodological and training support to researchers and technical professionals to increase the use of BRA methods in the reports used during deliberative decision-making processes. Furthermore, findings from this scoping review will inform the development of a methodological guideline on BRA for the Brazilian National HTA body, the Comissão Nacional de Incorporação de Tecnologias no Sistema Único de Saúde (CONITEC), using lessons learnt and best methodological practices from similar organisations.
Methods and analysis
Study design
While scoping reviews share rigorous and transparent methods in their conduct with systematic reviews, scoping reviews are more appropriate than systematic reviews to identify and map evidence relevant to a particular topic in broad sources of evidence.18 19 Specifically, scoping reviews can be useful tools to investigate the design and conduct of research on a particular topic including methodological approaches used, concepts and characteristics.20 This scoping review methodology was based on the methodological framework proposed by Arksey and O’Malley21 and the updated guidelines by the Joanna Briggs Institute.19 22 For this scoping review protocol, we used Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P)23 (online supplemental file 1). The scoping review protocol was registered in the Open Science Framework (https://doi-org.ezproxy.u-pec.fr/10.17605/OSF.IO/69T3V).
Supplemental material
Research question and eligibility criteria
Figure 1 summarises the relationship between our objectives, research questions and eligibility criteria. Our driven research question is ‘What are the methodological guidelines for methods of BRA in the context of health technology’s life-cycle?’
Relationship between the objectives, research question and eligibility criteria for the scoping review. BRA: benefit–risk assessment; HTA: health technology assessment; PCC, population, concept and context.
We developed a research question structured with inclusion and exclusion criteria following the PCC mnemonic (population, concept and context) as described in figure 1 and above in detail.
Population
We will consider methodological documents of methods of BRA conducted for a specific health technology including pharmaceutical drugs, procedures, medical devices, equipment and diagnostic tests. As we will consider methodological documents that may not refer to a specific intervention or focus on a specific population, there will be no restriction regarding any characteristics of populations.
Concept
The overarching concept of interest for this scoping review is to identify and map methodological guidelines and studies of quantitative and descriptive frameworks that address quantitative methods of BRA of health technologies, to find out which definitions have been used to define BRA and to describe methodological characteristics of BRA approaches. This includes the type of framework and the application of formal quantitative methods for measuring/expressing outcomes (metrics), extracting data (estimation techniques) and eliciting preferences (utility survey techniques).4 5 This review defines BRA as the balancing of the expected favourable effects (benefits) and expected unfavourable effects (risks or harms) of health technology. When balancing benefits against risks, the impact of uncertainty on these outcomes and trade-offs between these outcomes must be considered.24
We will exclude studies that reported methods exclusively for assessing harm or benefit outcomes, not the overall BRA balance, that is, for example, a review of only unintended adverse events or other harmful outcomes. Also, we will exclude publications that have focused on methodological approaches that could be used for BRA but did not present their application for BRA. For example, this exclusion criteria would apply to publications discussing the development or use of multicriteria decision analysis for decision making without its direct application in the context of BRA.
Context
This scoping review will consider methodological research studies or documents that provided recommendations or guidance on conducting or reporting a BRA in the context of health technology’s life-cycle such as decision making at the population level including the regulatory decision and HTA decision-making processes, clinical trials and evidence synthesis. These documents can be endorsed or produced by governments or non-governmental organisations or institutions, professional bodies, working groups or expert panels. They may be used to guide methodologists, guideline developers and researchers as the basis for the development of benefit–risk studies to inform regulatory decisions, reimbursement and coverage decision making as part of HTA reports. Also, it can be used as a methodological reference for the development of primary or secondary studies.
Types of publications
We will consider full-text documents that provide recommendations on how to conduct or report BRA. We will include two types of evidence sources: (1) methodological guidelines (ie, guidance, standards, frameworks, models, appraisals, recommendations or consensus) as any related tools or checklists and (2) methodological reviews, methodological studies or research reports addressing specific methods for BRA. We will exclude editorials, comments, opinion papers, conference abstracts, and primary studies of BRA.
Information sources and literature search
We will perform a comprehensive search for both published and unpublished evidence. The identification of evidence will be divided into three information sources: (1) electronic searches on the main life sciences and biomedical databases (electronic databases), (2) manual search for guidelines on the main websites of HTA and drug regulatory organisations, and other related sources (grey literature) and (3) contacting experts in the field. No restrictions regarding language or publication date will be made.
Electronic databases
A three-step search strategy approach was used.19 22 The first step includes an initial limited search on EMBASE (Ovid) and Medline (PubMed). This search was followed by an analysis of the text words contained in the title and abstract of retrieved papers, and of index terms used to describe the articles. A second search using all identified keywords and index terms was undertaken across all included databases. As a result of these two steps, the search strategy was built using free-text terms, indexing terms and validated filters25 26 to balance the sensibility and specificity of the searches to capture the CC’s key concepts: benefit–risk assessment, methods and guidelines or reviews or HTA reports (see online supplemental file 2). The search strategy has been developed and tested by three members of the research team (BDOA, FHdAM and FCG) with experience in systematic and scoping reviews and BRA studies, validated by an experienced research librarian (RC), and peer-reviewed by another reviewer using the Peer Review of Electronic Search Strategies checklist.27 The search strategy was adapted for each database and/or information source (online supplemental files 2 and 3).
With no language limitations, a final search using the developed search strategy will be conducted in the electronic databases EMBASE (Ovid) and Medline (PubMed).
As the third step of search strategy, the reference list of included sources of evidence will also be screened for additional studies. Specifically, the references that will be examined will include those reported in reviews and relevant studies.
Grey literature
An online search on grey literature will be based on an elaborated list of the most relevant HTA bodies, regulatory agencies and health-related sources from the Grey Literature tool of the Canadian Agency for Drugs and Technologies in Health .28 The search for HTA organisations will be supplemented by including other relevant HTA bodies and the leading global HTA networks (online supplemental file 4).
Regarding drug regulatory agencies, we have chosen 12 well-established drug regulatory authorities from four continents, including the USA, Canada, Europe, the UK, Australia, Japan and Brazil28–31 (online supplemental file 5). Additionally, a hand search will be conducted in the references of all included studies and Google Scholar.
Contacting experts in the field
To supplement the systematic online searches, a member of our research team (AFRB) has contacted renowned researchers via email, requesting any available potential guidelines or publications related to methods of BRA. This outreach to experts in the field took place during the period spanning February to March 2023.
Selection process
Electronic databases
A team of four reviewers (BDOA, FCG, FHdAM and AFRB) will perform the selection process according to the predefined inclusion and exclusion criteria. Since a high number of documents are expected to be retrieved from the literature, we will divide the total number of records to be screened and selected by two pairs of reviewers. First, each pair of reviewers will screen independently the title and abstract. The full-text assessment of potentially eligible publications will also be done by two pairs of independent reviewers. Studies that do not meet the inclusion criteria defined previously will be excluded. The reasons for excluding any documents will be recorded. Any disagreements will be solved through a consensus between the team of reviewers, and if there is any disagreement, a third reviewer will be involved (PCS).
Pilot testing will be performed with a random sample of 100 titles/abstracts.22 The entire team will screen the random sample using the prespecified eligibility criteria. A consensus meeting to discuss the agreement will be settled. In case of discrepancies or adjustment needed of the eligibility criteria, another random sample will be screened. The team will start screening when 85% (or greater) agreement is achieved.
Grey literature
The search in grey literature sources will be developed in a stepwise approach. (1) All HTA, drug regulatory, and health-related grey sources will be identified with name, abbreviation, country, language and website link. (2) Two reviewers (BDOA and FCG) will search for eligible studies, technical papers and reports on each webpage of the listed grey sources. (3) An online search will be done on these web pages to identify potential documents with the key terms: ‘benefits and (risks or harms) and methods’. In the case of many search records, we will add other relevant terms, apply search filters or enter specific website sections. This search will be made according to the language of origin of the evidence source, or if more than one language is available, preference will be given to the English, Spanish and Portuguese languages. When documents written in other languages are identified, a certified translator will be hired to translate them. The reviewers (BOA and FCG) will read the title and abstract, and any potentially eligible documents will be saved for eligibility assessment. (4) All full-text documents will be downloaded and saved with the website’s ID, name, language, and URL. The total number of documents found on each website will be recorded. (5) All documents identified in the grey literature will be screened for eligibility based on a full-text examination. The conflicts of eligibility will be solved by consensus, or another reviewer (PCDS) will be available in case of disagreements. Reasons for excluding will be reported.
Contacting experts in the field
In case the contacted experts send potential documents, two reviewers will apply the inclusion and exclusion criteria to select eligible documents.
Software used in the selection process
After running the search strategy in the electronic databases, all identified citations will be uploaded into Rayyan,32 an online platform that facilitates the article screening and selection process and removes duplicates. The selection process of grey literature and documents that will be sent from experts will be performed in Excel. The full text of selected citations will be stored on EndNote with a file in PDF attached.
Charting the data, summarising and reporting the results
A draft data charting form to guide the data extraction from the publications was developed by BDOA and PCDS, and validated by the other authors (FCG, EAS and FHdAM) (table 1). Previous studies focusing on BRA methodologies were used to develop the draft data charting form, we will consider the BRA methodologies classification proposed by Mt-Isa et al5 and Hallgreen et al33 and exemplified by Hughes et al4 and Juhaeri et al.34 This may be further improved at the review stage and the charting table updated accordingly. The vocabulary used for this protocol, including those on the charting data form and that will be used in the scoping review study, is given in online supplemental file 6.
Data charting form for the scoping review on methods of benefit–risk assessment
Data will be extracted independently by three pairs of experienced reviewers (BDOA, FCG, EAS, FHdAM, AFRB and NSF) with an appropriate background in guidelines and BRA, using an online semistructured Google form based on the validated data charting form. Any disagreements will be solved through a consensus between the team of reviewers, and if there is any disagreement, a third reviewer will be involved (PCDS).
We will provide a qualitative synthesis reported in systematic evidence tables along with descriptive statistics to identify common characteristics and map the evidence. Also, we will use appropriate visual sources to provide an overall summary of the results such as charts, figures and maps. Furthermore, we will interpret and discuss the meaning of the findings and implications for future research and practice, and we will report the gaps that should be addressed in future research. The PRISMA Extension for Scoping Reviews will be used to guarantee the quality of study reporting and replicability of the scoping review study.35
Critical appraisal of individual sources of evidence
Since we will include a wide variety of methodological guidelines and publications and there is no standard quality tool to perform the critical appraisal assessment of included studies, we will not perform a critical appraisal of included studies or work.
Patient and public involvement
Patients and/or the public were not involved. This is a scoping review study with no primary data collection. We intend to collaborate with a patient advisory group in the development of future BRA, based on the results of this study.
Ethics and dissemination
With the findings of this scoping review, we will contribute with a better understanding of which BRA methods are available; what were definitions that have been used; and what are the BRA approaches and the main methodological characteristics of existing quantitative approaches for assessing benefits and risks. These contributions are not in the context of regulatory approval or clinical trials, but mainly in the context of HTA bodies. This knowledge will help to provide a scientific basis for researchers, guideline developers, and all stakeholders interested in BRA.
In addition, the insights of this scoping review could be used to enhance the use of BRA in the context of HTA worldwide, enhancing the collaboration of different stakeholders and improving the efficiency of decision-making processes. This protocol and the full publication of this protocol will be disseminated through peer-reviewed publications, conferences, policy briefs and workshops.
Ethics statements
Patient consent for publication
Acknowledgments
We thank Rachel Couban, Health Research Impact Librarian, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada, for her contribution to the development of the electronic search strategy for this review.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @BrunaAscef
Contributors BDOA and PCDS conceived the main idea behind this manuscript. BDOA, PCDS and FCG wrote the manuscript. EAS, FHdAM, AFRB, NSF, BJ and US critically revised the manuscript and made important intellectual contributions to its development. All authors read and approved the final version of the manuscript.
Funding This work was supported by the Rede Brasileira de Avaliação de Tecnologias em Saúde—REBRATS through the CNPq grant number 400224/2022-4, with no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript. Also, this study was supported with grants from the National Institute of Science and Technology for Health Technology Assessment (IATS) – CNPq/Brazil. BDOA, FCG, EAS and FHdAM received CNPq scholarships.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.