Article Text
PDF
Abstract
Objective We aimed to develop and validate a new instrument called Questionnaire for the assessment of the knowledge, management and reporting ADR in Paediatrics by Healthcare teams (QUESA-P).
Design This is a cross-sectional study.
Settings and participants Teams of healthcare professionals (HCP) that lead with pharmacological therapy in Paediatrician’s sector (Paediatric-HCP) in seven public hospitals in Brazil.
Outcome An assessment of the knowledge and current management of ADR in Paediatric-HCP.
Methods We developed and validated QUESA-P, using a standardised procedure which included item development and psychometric prevalidation using Cronbach’s Alpha, item-total correlation and test–retest validity for internal consistency and reliability. External criterion was used as criterion validation (the instrument was applied to the focus group expert vs focus group team of Paediatric-HCP in hospitals). The focus group of experts who participated in psychometrics was asked to respond to the QUESA-P twice in order to assess test-retest reliability. The content validity of the initial questionnaire was assessed by the Delphi method and pilot test. Subsequently, we made minor revisions and finalized the QUESA-P
Results Selection of domains and facets were based on literature review made in duplicate by authors. Content validity was done by trial of different examiners (panellists, n=16), conducting analysis through Delphi method (three rounds). The QUESA-P was constructed with three domains. The intraclass correlations (0.80) and the Cronbach’s alpha coefficient (0.82), indicated adequate test–retest reliability and internal consistency for each domain. The application of the QUESA to 61 Paediatric-HCP in hospital resulted in lower mean score of 42.1 ± 3.4 in all domains when compared with expert teams (n= 46) 48.2 ± 3.7 (p <0.001) indicating that the instrument is valid to discriminate QUESA experts and Paediatric-HCP.
Conclusion The selected domains can be used to check weaknesses in the identification, management and reporting of suspected ADR by Paediatric-HCP in Brazil.
- Adverse drug reaction
- Knowledge
- Pharmacovigilance
- Drugs
- Paediatrics
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Although our sample was consistent with this analysis the survey was conducted at seven hospitals, raising the issue of generalisability of the findings.
We had a large number of experts in the scientific committee participating in all rounds.
Considering the test–retest interval is employed in 5 days, it is possible that participants may remember their responses and respond based on recall. However, the 13 questions using a 4-point Likert scale may have prevented participants from responding based on recall to some extent.
To the validation of an instrument used an established method that exams the precision of a determined interference. This instrument was submitted to four distinct validation steps (content validation, internal consistence, reproducibility validation and criteria validation).
QUESA-P should enable healthcare managers to instruct and improve assistant teams and optimising conducts in order to reduce adverse drug reactions.
Introduction
Morbidity and mortality related to the use of drugs are important public health issues, mainly when dealing with Paediatric population.1 2
A significant number of Paediatric patients are affected by adverse drug reactions (ADR). Some systematic reviews found that the prevalence of ADR is 0.3% to 17% in paediatric intensive care units3 and 9.65% (IQR 3.3%–17.35%) in ambulatory care.4 The prevalence of ADRs in pediatrics is generally higher than in adults, especially those related to the administration site reaction or those affecting the nervous system and skin.5
At the Paediatric and Neonatal Intensive Care Unit (NICU), drugs are responsible for the origin of up to 60% of adverse reactions.6 7 Systematic reviews from randomised clinical studies and conducted with Paediatric patients issued until 2007 verified that ADRs occurred in more than half (52%) of these studies.8
The WHO defines ADRs as a response to a medicine which is noxious and unintended, and which occurs at doses normally used in man.9 Studies suggest that lack of knowledge about ADRs, when prescribing a drug, is the main factor involved in its occurrence.10–13
The ascertainment of weaknesses healthcare professionals (HCP) have, regarding knowledge about identification, management and reporting of suspected ADRs, may provide instruments to healthcare managers to instruct and improve assistant teams, optimising conduct in order to reduce ADRs, especially in Paediatrics.12 14
An instrument validated to identify the weaknesses in the knowledge of ADR in Paediatrics for healthcare teams could generate better results in clinical practice.15 This study aimed to develop and validate an instrument to assess the knowledge of ADRs in a hospital from Paediatrics healthcare teams that lead with pharmacological therapy.
Materials and methods
It is a descriptive and analytical cross-sectional study towards the development and validation of an instrument for the assessment of knowledge hospitals’ healthcare teams have about ADR in Paediatrics Questionnaire for the assessment of the knowledge, management and reporting ADR in Paediatrics.
Item development, selection of domains and subdomains (facets)
In order to select the domains used in the creation of the construct, we searched in electronic databases (Pubmed and LILACS) as well as important sources of information: WHO Uppsala center, Pan-American Health Organization and some Surveillance Agencies. Four reviewers (LL, SST, LNC, JC) working in pairs independently evaluate whether the study or bulletins were relevant considering aspects related to concepts, causes and factors, classification, types of ADRs in Paediatrics, drugs that most frequently cause ADRs, steps for identification, reporting and management, besides aspects related to the laws and pharmacovigilance recommended by WHO and surveillance agencies were considered. Concepts and terms established by the WHO9 were the reference base for the development of DOMAIN 1. Several authors discussed the risk factors for the development of ADRs in Paediatrics, among them the exposure to multiple drugs, lack of defined doses for this age group, inadequate frequency, drug administration, period of permanence at the hospital, liver and kidney disease or immaturity and use of off label drugs.6 16–19 Studies show that the drug groups that promote ADRs in Paediatrics most are antibiotics and vaccines, mainly as dermatological reactions.20–23
DOMAIN 2 concerns the most important issue for the professional practice: procedures for the identification of suspected ADRs. This indicator should be well developed in the healthcare team routine. The instrument addresses this issue through two facets: factors that interfere with identification and procedures for the identification of suspected ADRs.
Some factors may prevent the establishment of the relation among the suspected ADRs, the drugs and the symptoms of the disease. It is a necessary practice to use guided techniques to establish causal link between the symptom presented in the child and the use of the drug.
DOMAIN 3 brings the facets which are related to the administrative management regarding ADRs, focussing mainly on the reporting for suspected ADRs. The reference adopted in the development of the statements contained in this domain considers the recommendations from WHO pharmacovigilance systems—Uppsala Center (https://www.who-umc.org/) and the Brazilian National Sanitary Surveillance (ANVISA).24
The 55 preliminary items derived from the existing literature were produced including the three domains and eight subdomains (facets). The instrument was constituted of statements where the individuals interviewed should mark in series of graduated items, the one that best specified their level of agreement with the statement, a modified Likert scale: (1) totally disagree; (2) disagree; (3) agree; (4) totally agree.
The instrument was called QUESA-P (Questionnaire for the assessment of the knowledge, management and reporting ADR in Paediatrics by Healthcare teams).
Content validity
In this step, the instrument was evaluated on the aspects of aim, items from the selected domains, content areas covered and its relevance regarding its purposes. We used Delphi method, a systematic and interactive forecasting method that is based on the independent experience of several experts that answer a questionnaire in one or more cycles.25 The anonymity of the respondents (experts—Scientific Committee), the lack of personal contact among the participants, the statistical representation of the distribution of results and the return of the answers of the group for re-evaluation in subsequent rounds are the main characteristics of Delphi method.26 27 The evaluating Scientifics committee, assessed the content and made suggestions regarding: (i) adequacy of the domains selected for assessing the knowledge on ADR in Paediatrics; (ii) compatibility of the questions to the reality of the professionals working in hospitals and; (iii) clear language and correct terminology for respondents’ better understanding.
Researchers with curriculum in Lattes Platform (www.cnpq.br/lattes) were selected to be part of the scientific team. The candidates were identified through a metasearch engine from CNPq (National Counsel of Technological and Scientific Development) website using keywords: adverse drug reaction, pharmacovigilance and questionnaire. Criteria to constitute a sample of expert committee included: publication of at least three articles on ADRs or pharmacovigilance; projects or articles involving the development of questionnaires about knowledge levels, and issued in the last 4 years. Some experts were also selected based on their experience in risk area management in hospitals or surveillance agencies.
The expert was individually and confidentially invited to be a part of the scientific team through electronic contact, by email and/or telephone. An invitation letter was sent, followed by the Free and Informed Consent Form (FICF). After receiving the signed FICF, QUESA-P version, with instructions and scripts for its evaluation, was sent to them.
QUESA-P was sent three times to each expert in order to reach convergence in the answers, that is, a consensus which represents the consolidation of the assessment of the construct. Each round lasted 15 days for the expert’s answers and another 15 days for adequacy of the researchers, in total 30 days.
Justification for the choice of score was asked when the expert chose the items (1) totally disagree; (2) disagree. After each endorsement round the modified instrument was sent again to the expert committee for their subsequent analysis round. The Scientific committee also contributed suggesting scientific papers, modifying domains and subdomains.
Therefore, the final QUESA-P corresponded to the fourth version of the instrument sent to the panellist, that is, the correction after the third round. This final version was also submitted for analysis regarding: clearness—the language used in the items, in the view of the characteristics of the respondents; pertinence—whether each item was elaborated in a way to evaluate the concept of interest in a determined population; theoretical relevance—the degree of association between the item and the theory; and theoretical dimension—adequacy of each item to the theory studied.
The agreement percentage regarding clearness, pertinence, relevance and dimension among the expert scientific team for each of the statements was calculated in the last round by statistical analysis.
Pretest
With a first draft of QUESA-P, we conducted a pretest with five HCP working in Paediatrics sector that lead with pharmacotherapy (two nurses, two paediatricians and one pharmacist). This pretest with HCP of Paediatric sector served as an opportunity to correct any remaining issues of comprehensibility or ambiguity.
Psychometric validation
Participant survey (n=107) was conducted to collect answer to each question for psychometric validation. The reliability and validity of the QUESA_P were then psychometrically tested using the collected questionnaires (n=107) of two focus group (expert and healthcare team professionals that lead with pharmacotherapy in Paediatric sector).
Participants and survey procedure
For this step, the instrument was submitted to experts’ focus group selected by using the same criteria established for selecting the scientific committee. The expert was individually and confidentially invited to be a part of the expert focus group through electronic contact, by email and/or telephone. An invitation letter was sent, followed by the FICF. After receiving the signed FICF, QUESA-P, version, with instructions and scripts for its evaluation, were sent to them.
For the second focus group (healthcare team) we selected randomly, HCP working in Paediatrician’s sectors in one of seven hospitals in state of São Paulo. The HCP focus team should include physicians, nurses and pharmacists that lead with pharmacotherapy in Paediatric sector. The hospitals were randomly selected using the list of hospitals in the web site of State Department of Health of São Paulo. Each professional was individually and confidentially invited by electronic contact, by email and/or telephone. An invitation letter was sent, followed by the FICF. After receiving the signed FICF we booked a time to meet the professionals in person to give them the QUESA-P in a sealed envelope and asked to complete and return the survey questionnaire to an independent research office in each hospital.
Reliability and reproducibility validation
The internal consistency was evaluated using Cronbach’s a coefficient. With regards to reproducibility this step includes the re-submission of the questionnaire to experts’ focus group, so this group was submitted to the same evaluation in two different days with a 5 day difference (test and retest). The re-submission followed the same procedure used for the first submission. In the second submission, the experts’ focus group answered questions about the time spent and the difficulty in filling in QUESA-P and their perception about the construct. This last item was verified with the use of two questions: (i) the instrument’s evaluation capacity, that is, if the expert deemed the instrument capable of evaluating his/her knowledge and; (ii) if he/she had any suggestions regarding content (domain or fact) that could be included hereof.
Construct validity
Discriminant Criterion Validity assessed the construct validity of the instrument.
Discriminant criterion validity
It consists in verifying if the instrument is able to discriminate the answers of groups of individuals in different stages of knowledge. Criterion validity is statistically estimated and the test is considered valid if the ratio between the test score (X) and the criterion variable score (Y) is high.28
At this stage, we contacted four universities to participate in this study and two of them accepted to collaborate. We randomly selected participants based on the list of students of the last year of the law course. We selected law students considering their curriculum does not include information about pharmacovigilance, and that is why they were considered in this study as laypeople to participate as focus group.
The results obtained from the submission of the instrument in expert focus group were compared with the results from the submission of the instrument to two other focus groups: an HCP who works in Paediatric sectors and a population of laypersons that were used as external criterion.
Patient and public involvement
There was no patient or public involvement in this study.
Statistical methods for psychometric testing
Demographic variables of the participants were summarised using descriptive analyses. The frequency of endorsement was assessed by splitting the response into two categories: ‘disagree’ (original rating=1 and 2) and ‘agree’ (original rating=3, 4), and then calculating the proportion of dichotomous responses. Items with very low (5%) endorsement frequency were discarded.29 For reliability, internal consistency and reproducibility were examined. Statistical analysis for reproducibility of the two sets of answers from the expert focus group in the test–retest were examined using the intraclass correlation coefficient. A coefficient of 0.7 or higher was considered evidence of acceptable test–retest reliability.30 With regard to internal consistency, the homogeneity of the question items in each domain was evaluated using Cronbach’s Alpha coefficient. A coefficient of 0.7 or higher is preferred for a questionnaire to be internally consistent.31
QUESA-P score calculation
In order to calculate the score to assess the knowledge and current management of ADR in Paediatrician’s area (main outcome) the items of QUESA-P were transformed in to a scale of numbers from zero to four according to the Likert scale. For the true statements (TS: 2, 4, 5, 6, 8, 9, 10, 11, 12 and 14), higher values were attributed to the answers ‘agree’ (3) and ‘totally agree’ (4). For false statements we established higher values we attributed to answers ‘disagree’ (3) and ‘totally disagree’ (4). If the respondents answer all statements correctly (theoretical total) his/her score may vary between 42 and 56 points. In Domain 1 (concepts and manifestations about ADR), with six statements, the maximum score interval obtained may vary from 18 to 24; in Domain 2 (identification and management of suspected ADR), from 12 to 16 and; in Domain 3 (reporting of ADR) from 12 to 16. Based on this score, after the submission to the experts’ focus group, a reference was established (or gold standard).
The answers of the three focus groups were compared using Student’s t-test for independent samples, using two-tailed analysis and considering 5% of significance level. This analysis allows for verifying the discriminatory capacity of the instrument between all focus groups. Statistical analysis was conducted with the use of SPSS software V.20.0.
Results
The main steps and results involved in the construction and validation of QUESA-P instrument are summarised in the flowchart, (figure 1 ).
Validation steps from Questionnaire for the assessment of the knowledge, management and reporting ADR in Paediatrics by Healthcare teams (QUESA-P). ADR, adverse drug reactions; D, Domain; F, Facet; ST, Statement; R, Round.
Content validation
Of the 25 scientific experts selected at Lattes Platform, 16 agreed to participate in the study. The second round of instrument evaluation had the participation of 14 experts and the third had 13 participants. Among the 13, 10 (62.5%) were women, 7 were pharmacists, 6 physicians and 3 were nurses. They selected the relevant items for knowledge of ADR and determined the extent to which the items represent the construct of interest (content validity). The initial instrument had 46 statements, 9 facets (subdomains) and 3 domains. In the last round, the instrument was finalised with 03 domains, 07 facets and 13 statements. The percentage of agreement among the scientific expert’s committee after the third round was >75% regarding clearness, pertinence, theoretical relevance and theoretical dimension.
The pretest with five HCP from Paediatric sector did not reveal any additional issues (figure 1).
Psychometric validation
A total 107 questionnaires accompanied by the background characteristics of the participants were subjected to reliability and validity testing.
Answered questionnaires from 71 expert focus groups were collected, and 46 of the fully answered questionnaires were subjected to item analysis. Of these 13% (n=6) doctors; 4.3% (n=2) nurses and 82.7% (n=38) pharmacists, mostly women 65.2% (n=30); 46.8% (n=22) with more than 21 years of training; between 28.2% (n=13) masters, 34% (n=16) doctors and 14.9% (n=7) post doctors. The weekly workload, in Paediatrics, was 85.1% (n=40) up to 10 hours; 59.6% (n=28) present time in the hospital area up to 4 years, particularly in Paediatrics 80.4% (n=37) in a public hospital 36.9% (n=17) (table 1).
Characteristic of expert focus group (n=46) and health professionals’ teams (n=61)
From 121 HCP that lead with pharmacotherapy in Paediatric sector invited to participate, 61 accepted it. From them 39.3% (n=24) were paediatricians; 41% (n=25) nurses and 19.7% (n=12) pharmacists, mostly 90% (n=54) women, 51.7% (n=31) with less than 10 years of training; exception for physicians where 45.8% (n=11) have between 11 and 20 years of training; 83% (n=51) are specialists; with a weekly workload of up to 10 hours in Paediatrics, with the exception of the majority (58.3%) (n=14) with weekly workload in Paediatrics between 11 and 30 hours. (n=52), particularly in Paediatrics 35% (n=21) up to 4 years and 65% (n=40) over 5 years, working in units (n=28), NICU and Surgical Centre, 62.2% (n=38) and other 44.2% (n=27). 47.5% (n=29) work in a private hospital (table 1).
Table 2 shows Cronbach’s Alpha coefficient for each QUESA-P scale items. The value obtained for the entire instrument was 0.82 indicating acceptable internal consistency.
Mean, SD, corrected item-total correlation and coefficient Alpha of Cronbach for each QUESA-P scale items (n=107)
Reproducibility validation
Regarding reproducibility, the intraclass correlation coefficient was 0.80. Their reproducibility was considered sufficient. The average score obtained by the experts’ group in the first submission was 3.49+0.30 and in the second (retest) 3.42±0.30. These results show that there is no statistically significant difference between them (p=0.690) (table 3).
Average values obtained from the submission (test) and re-submission (re-test) of QUESA-P in the experts’ focus group (n=46)
Of the 148 law students invited to be part of the focus group 55 accepted and fully answered questionnaires. We used them to calculate the discrimination validity. The majority of law students were female (n=39; 70.9%) and were in the last semester of the course.
The mean score by each domain and the total items of QUESA-P were different between focus groups. It means the QUESA-P has specificity (table 4).
Discriminant validity. Total mean score of QUESA-P items for each focus group
Thirty-three experts answered the questions related to the time needed to fill in QUESA-P. Among them, 22 (66.6%) said it took them 5 min to fill it in. Twenty-three experts (69.7%) informed they had no difficulties in understanding the statements from QUESA-P.
The instrument QUESA-P
The instrument’s validation regarding content, internal consistency, reproducibility and criteria is presented in online supplementary appendix.
Supplemental material
Discussion
We developed and tested a Questionnaire for the assessment of the knowledge, management and reporting ADR in Paediatrics by Healthcare teams (QUESA-P). The questionnaire includes 13 specific statements that assess the knowledge, management and reporting about suspected ADRs in three important domains: (i) Characteristics and manifestations of ADRs; (ii) Identification and management of suspected ADRs and; (iii) Reporting of suspected ADRs.
QUESA-P is structured in three parts, the first one is a script for filling in the instrument; the second is the characterisation of the professional interviewed and; the third is constituted of the 13 statements with four possibilities to answer definitely agree, agree, disagree and definitely disagree.
On the basis of psychometric testing, the QUESA-P was judged to be reliable and valid as a questionnaire for use in healthcare teams to identify the weaknesses about the knowledge on ADR in Paediatrics. Regarding reliability, good to excellent internal consistency and reproducibility were observed in all domains. The QUESA-P also demonstrated external validity and discriminant validity. The three focus groups differ significantly in each domain and in total items.
The inclusion of laypersons ensured the correction of phrases and terms that were not very clear and showed consistent discriminant validity of QUESA-P.
The focus group health professionals team that lead with pharmacotherapy in Paediatric sector did not demonstrate good knowledge about Domain 1 (concepts and manifestations about ADR) and 3 (Reporting of ADR in Paediatrics) obtained a better score in Domain 2 (Identification and Management of Drug Adverse Reaction) but still lower score than expert group.
Some studies show that under graduation courses in healthcare areas or even internships periods present insufficient content regarding pharmacovigilance issues.32 33 QUESA-P submission points out such weaknesses concerning abilities in management and reporting of suspected ADRs.
According to the WHO, ADR spontaneous reporting constitutes the main source of information in pharmacovigilance.9 The improvement of suspected ADR rates in Paediatrics is one of the most important challenges in public healthcare.5 ANVISA supports spontaneous reporting as effective in decreasing under-reporting, but this practice is seldom used by healthcare team professionals in Brazil, a situation confirmed by several national authors.10 34–37
Spontaneous reporting of suspected ADR is one of the main methods followed by HCP. However, under-reporting and selective descriptions of suspected ADRs are limitations that prevent the safety evaluation and the efficacy of the products in the market.15 Spontaneous reporting involves training HCP with techniques for the identification and management of suspected ADRs.11 Under-reporting may be related to the low level of participation of the professionals, preventing the establishment of ADR prevalence in the population studied.13 15
The effectiveness of a pharmacovigilance programme depends directly on the active participation of HCP.38 Sometimes, the professionals fear that the identification of suspected drug adverse reactions may negatively reflect his/her competence or place them under risk of being sued. Some are reluctant in reporting them because they feel unprepared to establish a causality between the use of the drug and the ADR and others do not have the knowledge to do so.36
It is important to know what health staff know about ADR so that the care process and rational drug use are not compromised. Punctual identification for overcoming these difficulties will only be possible through specific training aiming to improve the quality of the provided care. QUESA-P may assess the knowledge of the healthcare team and point out such weaknesses.
Thus, the elaboration of instruments that are able to assess the knowledge and necessary abilities a professional should have to deal with suspected ADRs is essential for a country where the pharmacovigilance system does not present total efficiency.
While knowledge of ADR is necessary, it is not enough for the pharmacovigilance system to be efficient. Meanwhile, having a clear picture of how healthcare teams are prepared to identify, manage and report suspected ADRs may be the first step in the establishment of measures that improve safety in the use of drugs in loco and in a country.
Conclusion
Based on the finding of this study, QUESA-P is a potentially useful tool for assessing knowledge of Drug Adverse Reaction in Paediatrics and verifying weaknesses in the identification, management and notification of ADR by hospital healthcare teams in Brazil and consequently improving the pharmacovigilance system by preventing adverse reaction involved in drug therapy in Paediatrics.
Acknowledgments
The authors would like to thank all experts—Scientific Committee and health care professionals who participated in the study.
References
Footnotes
Twitter @@lulopesbr
Contributors LL contributed to the study design. SST conducted measurements. LNC did the statistical analyses. LL, JC, SSL analysed the results and wrote the initial draft of the manuscript. All authors read and approved the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval This study was approved by Research Ethics Committee from UNISO, Protocol no. 30/2010.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information.