Article Text
PDF
Abstract
Background Adequate surveillance of HIV drug resistance prevalence is challenged by heterogenous and inadequate data reporting. To address this issue, we recently published reporting guidance documentation for studies of HIV drug resistance prevalence and incidence.
Objectives In this study, we describe the methods used to develop this reporting guidance.
Design We used a mixed-methods sequential explanatory design involving authors and users of studies of HIV drug resistance prevalence. In the quantitative phase, we conducted a cross-sectional electronic survey (n=51). Survey participants rated various reporting items on whether they are essential to report. Validity ratios were computed to determine the items to discuss in the qualitative phase. In the qualitative phase, two focus group discussions (n=9 in total) discussed this draft item checklist, providing a justification and examples for each item. We conducted a descriptive qualitative analysis of the group discussions to identify emergent themes regarding the qualities of an essential reporting item.
Results We identified 38 potential reporting items that better characterise the study participants, improve the interpretability of study results and clarify the methods used for HIV resistance testing. These items were synthesised to create the reporting item checklist. Qualitative insights formed the basis of the explanation, elaboration, and rationale components of the guidance document.
Conclusions We generated a list of reporting items for studies on the incidence or prevalence of HIV drug resistance along with an explanation of why researchers believe these items are important. Mixed methods allowed for the simultaneous generation and integration of the item list and qualitative insights. The integrated findings were then further developed to become the subsequently published reporting guidance.
- HIV & AIDS
- STATISTICS & RESEARCH METHODS
- Surveys and Questionnaires
- QUALITATIVE RESEARCH
Data availability statement
Data are available upon reasonable request. For request, email cristian.garcia@mail.utoronto.ca.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Mixed methodology allowed for the integration of both quantitative and qualitative methods to support the development of contextually relevant reporting guidelines.
Consensus was obtained through content validity ratios to account for item-specific response rates and chance agreement.
While all WHO regions were represented in the quantitative phase, certain regions were not represented in the qualitative phase.
Target survey response rate was not achieved.
Introduction
HIV drug resistance threatens the efficacy of antiretroviral drugs (ARVs), many of which risk becoming partly or fully inactive due to resistant strains.1 In June 2021, the WHO released an update to its HIV drug resistance strategy, highlighting the importance of monitoring and surveillance efforts and in obtaining high-quality data on HIV drug resistance prevalence estimates.2 However, adequate monitoring of HIV drug resistance worldwide is compromised by heterogenous and inadequate data reporting.3
Inadequate reporting makes it challenging for readers to assess the reliability and interpretability of research findings.4–7 Studies that collect information on HIV drug resistance should be reported comprehensively and consistently to permit pooling, which improves the precision of estimates. Likewise, clear reporting facilitates interpretation and contextualisation of estimates. For example, drug resistance data must be interpreted with due consideration of factors such as participants’ exposure to ARVs, transmission risk group, sampling techniques and the laboratory techniques used to qualify and quantify drug resistance. In previous papers, we have shown the need for reporting guidance for studies reporting the prevalence of HIV drug resistance.3
There is published guidance for researchers seeking to develop health research reporting guidelines.4 In accordance with this framework and to initiate the process of developing reporting guidelines for studies of HIV drug resistance prevalence, we completed formative work including a systematic review of HIV drug resistance prevalence which highlighted considerable heterogeneity in estimates, and a methodological study showing the gaps in reporting.3 8 We recently published a guidance document that contained both a checklist of reporting items that should be included in reports of HIV drug resistance incidence and prevalence, along with item-specific rationale and examples of proper use.9 The objectives of this study were to describe how we identified the checklist items and their accompanying rationale and examples.
Methods
Study design
We conducted a sequential explanatory mixed methods study from 2020 to 2021 which included: a cross-sectional electronic survey (quantitative phase) followed by focus group discussions (qualitative phase). A study diagram is presented in figure 1. The research design was iterative in nature with a system of analysis where the subsequent focus groups were structured to go through each reporting item generated by the electronic survey to produce explanatory data (eg, rationale, themes). Additional methodological details are provided in our published study protocol.10
Outline of sequential explanatory mixed methods study.
Objectives
Our primary research objectives were to:
To identify a list of reporting items considered to be essential for studies on the prevalence of HIV drug resistance.
Identify emergent themes on how and why these reporting items should be interpreted and adopted.
Rationale for design
The purpose of this study was to inform a reporting item checklist supplemented by item-specific elaborations. For this reason, an explanation is required for why each proposed reporting item is important, how it should be reported and illustrate examples of appropriate reporting. Mixed methods suit research objectives that cannot be met by either qualitative or quantitative methodologies alone, and we sought to use the qualitative data to directly explore results of the preceding survey to meet these objectives.11 12 Mixed methods thereby facilitated the efficient development of both the reporting item checklist and the associated item-specific insights in sufficient depth and breadth required to produce the reporting guideline. In a mixed methods study one phase always take priority.13 In this study we place priority on the qualitative because the focus groups could override some of the data items from the quantitative phase if the group agreed. In contrast, the Delphi method is a structured process of obtaining information from a group of experts often used in guideline development.14–16 This method was not selected given its long time scales as the process can become long and drawn out, resulting in fatigue and attrition.17 18 Furthermore, the Delphi approach focuses primarily on consensus and fails to consider disagreements or various perspectives.19 For this study, we focused on the importance of allowing participants to articulate their opinions and disagree with one another during the discussions. Delphi techniques were therefore not suitable to address our research objectives.
Research paradigm
Pragmatism is a useful paradigm for mixed-methods research because it allows for the use of ‘what works’ best in data collection and analysis.13 20 21 Additionally, the pragmatist paradigm incorporates multiple perspectives, linking both subjective and objective knowledge naturally suited for the integration of the quantitative and qualitative data produced in this study. In the lens of pragmatism, we acknowledge that our research occurs within specific sociopolitical and economic contexts.21 These contexts shaped the development of a reporting item checklist that is relevant to authors of HIV drug resistance prevalence research in diverse settings.13
Data integration
There are various approaches to data integration in mixed methods, two of which were present in this study at the design level.22 One form, known as ‘complementarity’ or ‘building’, had the results of one method elaborate and clarify on the findings of the other method.23 24 The published reporting guidance document itself is the product of this integration.9 The document covers a series of recommended reporting items with brief explanations of each item and examples, and illustrates how the two data strands complement each other to form a comprehensive and cohesive guideline document. A second more minor form of integration occurred called ‘development’, where the results of one method inform the data collection of the other method.23 24 This was observed when the focus groups used the item list generated by the preceding survey to guide the discussions, shaping the context to which the qualitative data was collected.
Patient and public involvement
Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Sampling
Quantitative phase
Our purposeful convenience sampling frame for the cross-sectional survey included corresponding authors (n=160) from the 650 studies of HIV drug resistance included in our 2020 systematic review of HIV drug resistance prevalence in key global populations.8 Study invitations were sent to all 160 email addresses in November 2020. The survey link was also disseminated on social media platforms and among HIV journals with authors who have published research on HIV drug resistance prevalence. Considering a population of 160 and assuming an α level of 0.05 and a 10% margin of error, we arrived at a minimum sample of 61 survey respondents to be representative of the population of HIV drug resistance researchers.25 As participant identifying information was used to invite and recruit participants, data collection in this study was non-anonymous.
Qualitative phase
In the qualitative phase, we sought a purposeful sample of survey responders from the quantitative phase who indicated in their survey response their willingness to participate in focus group discussions. When selecting participants for these discussions we sought to achieve at least one male and one female participant from as many of the six WHO regions as possible (Africa, Americas, Europe, South-East Asia, Eastern Mediterranean, Western Pacific). When organising the focus groups, a range of dates and times were proposed to accommodate for various time zones. Based on the preferences of those who responded to their invitation, the first session was set around the EST (GMT-5) time zone and the second was around the Eastern European time zone (GMT+2), on dates that worked for the majority.
Data collection
Quantitative phase
In the quantitative phase, authors of drug resistance prevalence studies were approached to complete a 23-question electronic survey, rating reporting items as ‘essential’, ‘useful but not essential’ or ‘not necessary’. At the end of each section, participants were permitted to suggest any additional items they believed should be considered into an open-text field. To capture participant characteristics, basic sociodemographic data such as age, sex, country of residence, profession, number of years in primary role were also collected as part of the survey. Participants were also asked whether they were interested in participating in the focus group discussions. The electronic survey is available as a online supplemental file 2.
Supplemental material
Qualitative phase
In the qualitative phase, individuals who participated in the survey and expressed interest in participating in focus group discussions were approached to provide consent prior to the discussions. Focus groups were conducted over Zoom in October 2021, with both the session audio, video and chat log recorded and stored. After participant introductions, the facilitator introduced the session and initiated the discussions based on a focus group guide. During the discussions, participants viewed the initial draft list of reported items from the quantitative phase and discussed what made each item essential or not. Participants also discussed all additionally suggested reporting items brought up in the survey. Focus group discussions lasted about 120 min each. Audio files were transcribed by a professional transcription service. Pseudonyms were used to maintain anonymity.
Data analysis
Quantitative phase
We conducted a descriptive analysis of quantitative data using R Studio V.4.0.3, summarising counts (%) for categorical variables, mean (SD) or median (IQR) for continuous or discrete variables. These data were used to compute a content validity ratio (CVR) for each item by dividing the number of those who rated an item as ‘essential’ (Ne) by the total number of participants who rated the item (N), (CVR=[Ne–(N/2)]/(N/2)). The CVR represents the items that at least half of participants consider essential.26 To account for agreement that could be due to chance and the number of respondents, a set of threshold values called critical CVR values (CVRcrit) were calculated.26 These values were specific to the number of respondents who rated an item (eg, CVRcrit(n=50)=0.253, CVRcrit(n=51)=0.250). Each CVRcrit was calculated using the bitesti command in STATA and the critbinom formula in Excel.
Only reporting items with a CVR that exceeds their CVRcrit (ie, those where at least half of respondents agreed were essential, above that of chance) were kept on the draft list of reporting items. An example calculation is provided in online supplemental appendix 1. However, dropped items could be reintroduced if brought up during the focus group discussions.27 All additionally suggested reporting items from the open-text fields of the survey were summarised and discussed in the qualitative phase.
Supplemental material
Qualitative phase
The audio-video recordings were transcribed into text transcripts. We conducted a descriptive qualitative analysis of the data produced from the focus groups, where open codes were generated by identifying repetitions in the text in Taguette, a free and open-source qualitative data analysis tool.28 29 Pre-existing codes or themes were not used to allow for concepts to emerge from the data. Similar codes were grouped, with themes emerging from these groupings in Taguette. Two coders (CG, JMS) worked on the data to verify agreement on the generated themes. Disagreement was resolved by discussion. Descriptive analyses continued cyclically until no new patterns or themes emerged from the data.
Validation checks
In the quantitative phase, we estimated a minimum representative sample size and revised and pilot-tested our survey. In the qualitative phase, we used member-checking, audio-video recordings and duplicate coding to improve the validity of our findings. During the focus group discussions, we minimised facilitator bias by using a discussion guide.
Results
Quantitative results
Participants
51 participants responded to the survey for a response rate of 31.8%, but 84% of the target sample size. The mean age of participants was 48.1 years (SD=10.51) with 17 females (37%), and mean number years of experience in role was 17 (SD=9.45). At least one participant from each WHO region was represented in the survey, with responses from 24 countries. Over a quarter (n=13, 28.3%) of participants were from the African WHO region, with another quarter from the European region (28.3%). Nearly a third of participants were from the Americas region (n=14, 30.4%). The details of sociodemographic characteristics are displayed in table 1.
Sociodemographic characteristics of participants in the quantitative phase of the study (n=51)
Of the 23 proposed reporting items, 15 were retained for further evaluation in the focus group discussions based on the CVR (see table 2). 58 additional reporting items were suggested by survey participants and were evaluated in the focus group discussions.
Initial reporting item checklist, with content validity ratios (CVR) and critical values
Qualitative results
Participants
Two focus group discussions were conducted including a total of nine participants, with four female and five male participants. The mean age was 55.4 years (SD=9.13). Six participants had primary roles in research, two participants clinical primary roles and one participant was from government. The mean years in primary role was 26.6 years (SD=6.71). Both groups were similar with regards to WHO region, with four of five participants in the first group from the Americas region (USA, Canada) and one from the Eastern Mediterranean region (Tunisia). In the second group, three of four participants were from the European region (Italy, Spain, Israel) and one from the Americas (Argentina).
Capturing agreement and disagreement
A total of 76 reporting items were discussed during the focus groups (see online supplemental appendix 2). There were 13 discrepancies between focus groups in their evaluations of the items. 10 of these discrepancies involved one group rating the item as optional rather than essential or not essential. Common reasons reporting items were rated as optional were anticipated challenges capturing the item, or that the item was only relevant for specific study designs. For example, the items ‘place of likely HIV acquisition’ and ‘time on ART regimen’ were discussed by one group to be difficult to accurately capture. There were two items where one group rated the item essential and another non-essential. The first, ‘assay used for HIV diagnosis’ was rated not essential by one group for reasons that it overlapped with another reporting item on the list and was difficult to capture. The second, ‘CD4 count at sampling’ was rated not essential by a group as it depends on the study design and is not available for all types of HIV drug resistance research.
Characteristics of an essential reporting item
Common reasons why reporting items were rated essential were that they improve the interpretability of study results, clarify the generalisability of findings and conclusions, are a standard epidemiological item, or display a combination of these characteristics. For example, essential study-level reporting items like study setting, design, sampling year and sampling strategy were reported to contextualise the study environment to improve comparability and interpretability. In contrast, other items like the total number of participants eligible, screened and consented were considered essential due to being standard epidemiological items.
[It is important to] interpret the study results in the broader context of the population being assessed. And without that information, you don't know whether you can generalize beyond the study at all. (Group 1; participant 2, male)
Essential participant-level items were described to better characterise the study participants, providing further detail on variables like age, sex, gender, transmission risk group, as well as better characterise participants along the HIV care continuum, including items like use of pre-exposure prophylaxis (PrEP) or post-exposure prophylaxis (PEP), date of HIV diagnosis, HIV RNA level, among others. Essential reporting items related to ART were highlighted to differentiate between different types of drug resistance (transmitted, pretreatment, acquired) and provide context to the types of drug resistance observed in the sample. Example reporting items include treatment history, composition of the antiviral regimen (by class and drug), and time on ART regimen.
We need to indicate the therapy in this specific population—the real therapy they receive, not only the ones recommended in guidelines. (Group 2; participant 2, female)
[These items] aid in understanding of why you may or may not see certain drug resistance mutations or mutation patterns for your frequency. (Group 1; participant 4, male)
Essential resistance testing items clarified the methodologies used to conduct HIV resistance testing like the type of resistance test, mutation list used (including list year and version), quality assurance methods, definitions of predicted resistance mutations, among others. Essential reporting items also clarified the levels which HIV drug resistance is reported, at the overall level and by drug family, class and individual drug levels.
I think it’s important to know all these different methods because I want to know why my result is different than their result. What matters is the precise way you analyze your data and how you define resistance. (Group 1; participant 4, male)
Additional findings
Several reporting items lead to discussion over concerns on both the feasibility and ethics of asking authors to report certain data, mainly participant-level items like sexual orientation, migration status, ethnicity, place of residence and method of nucleotide sequence generation. Across various reporting items participants suggested wording revisions to the items, as well as identifying redundancies between items.
There’s a growing concern around the use of molecular epidemiology, particularly in vulnerable populations where certain behaviours are criminalized. We need to keep that in mind and be very cautious when developing this list. (Group 1; participant 2, male)
Sexual orientation and some of this data [is] not easy to collect because [it’s] taboo. (Group 1; participant 3, female)
Discussion
Summary of main results
In this paper, we use mixed methods to produce a list of reporting items to inform reporting guidelines for studies of HIV drug resistance prevalence along with item-specific rationale and examples. Overall, HIV drug resistance experts specified the need for more detailed and transparent reporting on study setting, participant characteristics and study drug-resistance testing methodology. Additional items involved details on laboratory methods, data sources, and the year, version and type of mutation list used. Many items further specified the type of HIV drug resistance observed, providing details within overall (global) resistance to the level of drug families and drug classes. Many of the suggested participant-level items focused on the timing of, duration of, or type of exposure to antiretroviral medications, which influence the risk of HIV drug resistance. Emergent themes elucidated during discussion on the survey response focused on the need for more detailed reporting of various items, concerns over the availability and ethics of reporting sensitive participant data, interpretability and comparability as main reasons to report more detailed data, and the necessity for reporting guidelines to appreciate context-specific prevalence research.
HIV drug resistance experts identified a list of reporting items essential to report in research with prevalence data. To our knowledge this is the first study to use CVRs to quantitatively achieve consensus on a list of reporting items. CVRs are traditionally selected to assess content validity in instrument development research.30 We found that the use of CVRs was a pragmatic and straightforward method to discriminate between essential and non-essential reporting items.
During the focus groups, participants reviewed the results of the quantitative strand and shared their perspectives on what makes a reporting item essential to HIV drug resistance research. Throughout the discussions participants mostly agreed with one another, connecting their opinions to their own research and the context of their country of origin. Given the geographical diversity of the focus groups, participants highlighted the importance of the various country settings that produce HIV drug resistance prevalence data and the appreciation that in some circumstances data is available but cannot be reported to protect patient confidentiality. This finding also reflects the diverse needs of reporting guidelines across various types of HIV drug resistance research and country settings (physical locations where research is conducted, eg, community vs clinical care settings) and contexts (broader complex sociocultural influences like migration patterns and clustering of vulnerable populations). These insights reflect the need for HIV drug resistance prevalence research to stay up-to-date with current global affairs. Participants also expressed concern regarding the ethics of requiring reporting of participant personal information for research conducted in settings where HIV and certain sexual practices are criminalised.
As the focus groups were structured to go through each reporting item from the electronic survey, the discussion produced explanatory data (eg, rationale, themes) for each reporting item in the checklist. As recommended for health research reporting guideline development,4 we will use this information to accompany the forthcoming reporting guidelines in an explanation and elaboration document. Emergent themes between the two focus groups were largely similar.
Influence of context and researchers
Our findings should be interpreted as being relevant to authors or users of HIV drug resistance literature in academic, research, clinical and government settings. We acknowledge that our findings are tied to various socioeconomic, cultural and political factors specific to our team in Canada and the participants’ own countries of origin.
Strengths and limitations
The strengths of this study include the integration of both quantitative and qualitative methodologies to elicit quantitative consensus and qualitative rationale from researchers on the items that should be reported in studies of HIV drug resistance. Additionally, validation checks were made in both phases of the study to improve data quality.
Study limitations include lower than anticipated (~30%) response rates to the survey, and thus the minimum sample size was not reached. Additionally, while we had representation from all WHO regions in the survey sample, we would have liked to have had at least one participant from each WHO region participate in the focus group discussions. These limitations in diversity will be addressed by inviting a diverse group of external reviewers to review the checklist when developing the complete reporting guidelines.
Implications for future research
Overall, support for and willingness to participate in the process to create reporting guidelines for studies of HIV drug resistance is evident among authors of this research. Our participants made several comments on the current lack of guidance for reporting HIV drug resistance prevalence data, reaffirming our previous work demonstrating the need for reporting guidelines in this area of research.
The insights derived from the mixed-methods approach allowed us to clarify, integrate, and elaborate on our findings in the subsequent guidance document, which details how the reporting items should be interpreted and adopted. As our participants were the end-users with interest in this checklist, we welcomed comments on how to best create this guidance document. For example, the document clearly delineates the target users of the guidelines, the types of studies that each reporting item applies to, and whether some reporting items are more applicable for certain study designs. For example, certain participant items like sexual orientation may be unavailable or unethical to report in molecular epidemiology studies with HIV drug resistance prevalence data.31 32 For reporting items that may result in undue harm for participants in contexts where HIV status, gender identity or sexual orientation are stigmatised or criminalised, the guidelines touch on the ethical considerations involved when reporting potentially sensitive data. This guidance documentation also details how to report each item and is now available as a website format including French, Italian, Portuguese, Spanish and Chinese translations.9
Conclusions
We developed a list of reporting items for prevalence studies of HIV drug resistance and item-specific rationale and examples. This data was incorporated into a reporting item checklist that used the insights from the mixed-methods approach to justify and elaborate on the recommended use for these items.
Data availability statement
Data are available upon reasonable request. For request, email cristian.garcia@mail.utoronto.ca.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and this study received ethics approval from the Hamilton Integrated Research Ethics Board (HiREB) project number #11558 on 11 November 2020. Informed consent was obtained before each study phase. Focus groups were not anonymous, however pseudonyms were used to maintain anonymity during data analysis and manuscript writing.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @DDjiadeu
Contributors All authors involved in this work have made substantial contributions to the conception or design of the work, acquisition, analysis and interpretation of data. Authors assisted in drafting the work or critically reviewing it for important intellectual content, as well as providing final approval for publication and agreeing to accountability. CG is the guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.