Article Text
PDF
Abstract
Purpose To translate and adapt the Chelsea Critical Care Physical Assessment Tool (CPAx) into Chinese version (‘CPAx-Chi’), test the reliability and validity of CPAx-Chi, and verify the cut-off point for the diagnosis of intensive care unit-acquired weakness (ICU-AW).
Study design Cross-sectional observational study.
Methods Forward and back translation, cross-cultural adaptation and pretesting of CPAx into CPAx-Chi were based on the Brislin model. Participants were recruited from the general ICU of five third-grade class-A hospitals in western China. Two hundred critically ill adult patients (median age: 53 years; 64% men) with duration of ICU stay ≥48 hours and Glasgow Coma Scale ≥11 were included in this study. Two researchers simultaneously and independently assessed eligible patients using the Medical Research Council Muscle Score (MRC-Score) and CPAx-Chi.
Results The content validity index of items was 0.889. The content validity index of scale was 0.955. Taking the MRC-Score scale as standard, the criterion validity of CPAx-Chi was r=0.758 (p<0.001) for researcher A, and r=0.65 (p<0.001) for researcher B. Cronbach’s α was 0.939. The inter-rater reliability was 0.902 (p<0.001). The area under the receiver operating characteristic curves of CPAx-Chi for diagnosing ICU-AW based on MRC-Score ≤48 were 0.899 (95% CI 0.862 to 1.025) and 0.874 (95% CI 0.824 to 0.925) for researcher B. The best cut-off point for CPAx-Chi for the diagnosis of ICU-AW was 31.5. The sensitivity was 87% and specificity was 77% for researcher A, whereas it was 0.621, 31.5, 75% and 87% for researcher B, respectively. The consistency was high when taking CPAx-Chi ≤31 and MRC-Score ≤48 as the cut-off points for the diagnosis of ICU-AW. Cohen’s kappa=0.845 (p=0.02) in researcher A and 0.839 (p=0.04) for researcher B.
Conclusions CPAx-Chi demonstrated content validity, criterion-related validity and reliability. CPAx-Chi showed the best accuracy in assessment of patients at risk of ICU-AW with good sensitivity and specificity at a recommended cut-off of 31.
- adult intensive & critical care
- clinical trials
Data availability statement
Data may be obtained from a third party and are not publicly available. Data can be requested from the Ethics Committee (contact via the First Hospital of Lanzhou University, Lanzhou, Gansu, China; email: ldyylwh@126.com) for researchers who meet the criteria for access to confidential data.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Two researchers assessed and collected data independently, which improved the reference value of the validation data.
We took Medical Research Council Muscle Score ≤48 as the criterion to demonstrate the best cut-off point for the diagnosis of intensive care unit-acquired weakness using the Chinese version of Chelsea Critical Care Physical Assessment Tool (CPAx-Chi).
This is a non-randomised pool of participants chosen primarily by their availability during the study period.
There were specific exclusion criteria that may have stopped the potential ‘ceiling and floor’ effects of CPAx-Chi to be tested.
Introduction
Intensive care unit-acquired weakness (ICU-AW) is a severe and debilitating complication in critically ill patients. The prevalence of ICU-AW in patients receiving mechanical ventilation for more than 4–7 days has been reported to be 38%–86%.1–5 The prevalence of ICU-AW in patients with sepsis is 86%.1 2 5 Early identification, assessment and active prevention are crucial to reduce ICU-AW risk because the pathophysiological mechanism of ICU-AW is not clear, and efficacious pharmacotherapy is lacking.1 6
A gold standard for the diagnosis of ICU-AW is not available, and the Medical Research Council Muscle Score (MRC-Score) is the most widely used diagnostic tool for ICU-AW.7 Other tests are also frequently used to test for ICU-AW but there is no uniform cut-off point. The MRC-Score evaluates the strength subjectively in three muscle groups of all four limbs according to the Oxford Muscle Strength Grading Scale. The latter is not only affected by several factors, it also cannot evaluate respiratory function. Several studies have shown that diaphragmatic dysfunction is correlated significantly with ICU-AW,8–10 and that the function of respiratory muscles may be related to the occurrence and development of ICU-AW.
The Chelsea Critical Care Physical Assessment Tool (CPAx) could be an optimal tool for predicting and evaluating ICU-AW. CPAx can be used to measure physical function, mobility, grip strength, respiratory function and cough ability.11–13 CPAx has been translated into several languages for use in the UK, Sweden, Denmark and other countries.14 15 However, a Chinese version of CPAx (CPAx-Chi), or the cut-off point of CPAx for the diagnosis of ICU-AW is lacking. Therefore, the aim of this study is to translate and adapt the CPAx into ‘CPAx-Chi’, test the reliability and validity of CPAx-Chi, and verify the cut-off point for the diagnosis of ICU-AW.
Materials and methods
Translation, cross-cultural adaptation and pretesting
The translation of the original CPAx tool into Chinese was completed with the consent and assistance of the primary original author (EJ Corner).11–13 Translation, cross-cultural adaptation and pretesting were done based on the model described by Brislin.16–18
Translation
Three bilingual authors with Chinese as their native language undertook the forward translation of CPAx from English to Chinese. One was a physician experienced within the specialty of critical illness; one was a nurse experienced within the specialty of critical illness; one was a graduate student in nursing with College English Test 6 certification unfamiliar with clinical medicine. A seminar was conducted to discuss and synthesise the results of the three translators. Different opinions were resolved through group consultation, and then integrated into CPAx-Chi, which was named ‘CPAx-Chi-Forward’.
Back translation
Three bilingual translators with English as their native language translated CPAx-Chi-Forward back into English. One was a doctoral student in nursing based in the UK; one was a doctoral student in physiotherapy based in Canada; one was a certified English linguist. They were unfamiliar with and blinded to the original CPAx version. A seminar was conducted to discuss and compare CPAx-Chi-Forward with the original CPAx.11 12 Discrepancies between the three translations were discussed until consensus was reached, and then the final synthesised back-translated English version was named ‘CPAx-Eng-Back’. The researchers provided a final report that included the annotations from translators about their rationale for translation, choices and linguistic considerations to the author of the original CPAx.
Cross-cultural adaptation
Nine experts revised the items of CPAx-Chi-Forward based on their theoretical knowledge, practical experience, subjective feelings and expression in the Chinese language. Two were specialists in critical care medicine, five were nursing specialists in critical care, one was a respiratory therapist and one was a physiotherapist. During the process, some words were rephrased or adjusted due to linguistic, grammatical, terminological or cultural differences between English and Chinese. Changes from the original CPAx version to the synthesised back-translated English version were discussed and accepted by the original author.11
Pretesting and verifying cultural adaptation
Forty ICU nurses from the First Hospital of Lanzhou University applied CPAx-Chi-Forward and manual dynamometer (WCS-100) to assess ICU patients. Meanwhile, a 5-step Likert scale method was used to assess if the written expression in CPAx-Chi-Forward was readily comprehensible, well described and conform to Chinese grammar, and suggestions could be noted. The result showed that there were no significant differences regarding the assessments of ‘readily comprehensive’, ‘well described’, ‘conform to Chinese grammar’ in nurses with varied sex, nationality, professional title or time working in the ICU (p>0.05) (table 1) from culture adaptation. Adjustments were not deemed necessary and CPAx-Chi-Forward had good cross-cultural adaptation. Therefore, the final CPAx-Chi was accepted.
Verifying cultural adaptation differences (n=40)
Verification of CPAx-Chi
Study design
This was a cross-sectional observational study, and the flow chart is shown in figure 1. We took the MRC-Score which is the most widely used diagnostic tool for ICU-AW as the comparator to test CPAx-Chi, and a manual dynamometer was used to assess grip strength (WCS-100). Meanwhile, researcher A and researcher B simultaneously and independently assessed eligible patients using the MRC-Score and CPAx-Chi.
The flow diagram of the research. CPAx, Chelsea Critical Care Physical Assessment Tool; CPAx-Chi, Chinese version of Chelsea Critical Care Physical Assessment Tool; ICU-AW, intensive care unit-acquired weakness.
Participants
Adult critically ill patients were recruited from the general ICU of five third-grade class-A hospitals in western China from September 2019 to June 2020. The recruiter explained the purpose and significance of the study to participants who meet the eligibility criteria, and then serial numbered.
The inclusion criteria were the following: (1) critically ill patients eligible for ICU admission; (2) age ≥18 years; (3) duration of ICU stay ≥48 hours; (4) Glasgow Coma Scale (GCS) score ≥11; (5) volunteered to participate in our study.
Patients were excluded if they may be misdiagnosed as ICU-AW just like: (1) unstable fracture, limb deformity or limb dysfunction; (2) myasthenia gravis or Guillain-Barre syndrome.
Sample size
The sample size was calculated by the principles of scale development. In general, the sample size was 10–15 times the number of scale items and add taking into 20% account loss to follow-up and participant attrition.19 The sample size of this study was 120–180. However, there are some studies that introduced that a sample size of 200 is reasonably good for ordinary factor-analytical work with 40 or fewer variables.19–21 Therefore, we finally took 200 cases as the sample size of this study.
Ethical approval and consent to participate
In the pretesting study, we found that there were significant differences between patients with informed consent and the patients with no informed consent, especially in the items respiratory function, grip strength and transferring from bed to a chair. In order to ensure data validity and quality, the Ethics Committee provided a waiver of informed consent which was uploaded as supplemental information (online supplemental material). In addition, it was routine work that researchers assessed eligible patients using the MRC-Score and CPAx-Chi.
Supplemental material
Patient and public involvement
The study was designed to test the CPAx-Chi, and verified the cut-off point of CPAx-Chi to diagnose ICU-AW. However, patients were not involved in the design of the survey instrument, recruitment or conduct of the study. Patients who participated did so anonymously, and therefore the study team will be unable to disseminate the results to study participants.
Statistical analyses
SPSS V.22.0 (IBM) was employed for statistical analyses. Frequency and percentages were used for dichotomous variables. The mean±SD was used for continuous variables. Content validity (CV) and criterion-related validity were employed to test the validity of CPAx-Chi. CV index (CVI) included the scale-level CVI (S-CVI) and item-level CVI (I-CVI) which is the most widely used index in scale evaluation. The expert authority coefficient and Kendall synergy coefficient were used to calculate expert evaluation results, and the more Kendall synergy coefficient, the more consistent the results are. Cronbach’s α coefficient and inter-rater reliability were used to test the reliability of CPAx-Chi. The MRC-Score was taken as the standard to calculate the receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) of CPAx-Chi. The cut-off point of CPAx-Chi was determined by the maximum value of the Youden Index (YI). The kappa test was used to test the consistency of the MRC-Score and CPAx-Chi. P<0.05 was considered significant.
Results
Characteristics of participants
Nine experts adjusted the cultural adaptability of CPAx-Chi, and evaluated the importance and relevance of each item in the scale. Two (22.22%) were specialists in critical care medicine, five (55.56%) were nursing specialists in critical care, one (11.11%) was a respiratory therapist and one (11.11%) was a physiotherapist. The median age of specialists was 38 (IQR 33–50) years. The median time the specialists had been working in the ICU was 13 (IQR 6–23) years. There were nine specialists that included one (11.11%) undergraduate, four (44.44%) masters and four (44.44%) doctors; four (44.44%) intermediate titles and five (55.55%) senior titles.
Two-hundred critically ill patients participated in this study (128 (64%) men and 72 (36%) women; mean age: 53.24±15.06 years). The Acute Physiology and Chronic Health Evaluation score was 15.04±6.70. The mean duration of ICU stay was 9.04±6.15 days. The mean duration of hospital stay was 20.79±11.84 days. The duration of mechanical ventilation was 3.55±5.19 days. The principal diagnoses of participants were: craniocerebral injury (16, 8%), respiratory failure (22, 11%), surgical complications (68, 34%), hepatobiliary disease (42, 21%), cardiovascular disease (20, 10%), shock (14, 7%) and other (18, 9%). Also, 190 (95%) patients were transferred to other departments, 2 (1%) patients were discharged from ICU, 2 (1%) patients were transferred to other hospitals and 6 (3%) patients died.
Validity
Content validity
The I-CVI was from 0.889 to 1. The S-CVI, which is the average of I-CVI, was 0.955. The median expert authority coefficient was 0.85 (IQR 0.75–0.95). The Kendall synergy coefficient was 0.61 (p=0.842), and a significant difference was not detected in the degree of expert coordination.
Criterion validity
The correlation coefficient for ICU-AW assessment by researcher A between the MRC-Score and CPAx-Chi was 0.60 (p<0.001). The correlation coefficient for ICU-AW assessment by researcher B between the MRC-Score and CPAx-Chi was 0.65 (p<0.001) (table 2).
Criterion validity (n=200)
Reliability
The internal consistency of CPAx-Chi was acceptable (Cronbach’s α=0.939). The correlation coefficient between researcher A and researcher B in the items of CPAx-Chi was between 0.668 and 0.992 (p<0.001). The correlation coefficient between researcher A and researcher B in CPAx-Chi total score was 0.902 (p<0.001) (table 3).
Inter-rater reliability (n=200)
Best cut-off point for the diagnosis of ICU-AW using CPAx-Chi
The ROC curve for ICU-AW diagnosis with CPAx-Chi was drawn taking MRC-Score ≤48 as the standard for the diagnosis of ICU-AW. An MRC-Score ranging from 0 to 48 was termed ‘1’ (ICU-AW group). An MRC-Score >48 was termed ‘0’ (non-ICU-AW group).
The AUC for researcher A was 0.899 (95% CI 0.862 to 1.025) (figure 2). The AUC for researcher B was 0.874 (95% CI 0.824 to 0.925) (figure 3). The best cut-off point was determined by the maximum value of the YI. The maximum YI for researcher A was 0.643, the cut-off point was 31.5, the sensitivity was 87% and specificity was 77%. The maximum YI for researcher B was 0.621, the cut-off point was 31.5, the sensitivity was 75% and specificity was 87%.
The ROC curve and area under the ROC curve of researcher A. ROC, receiver operating characteristic curve.
The ROC curve and area under the ROC curve of researcher B. ROC, receiver operating characteristic curve.
MRC-Score and CPAx-Chi were consistent for the diagnosis of ICU-AW
We calculated 31 as the best cut-off point to diagnose ICU-AW using CPAx-Chi. Hence, if the total score of CPAx-Chi ranged from 0 to 31, it was marked as 1 (ICU-AW group), and if the total score of CPAx-Chi ranged from 32 to 50, it was marked as 0 (non-ICU-AW group). We found no significant difference in the total score of the ICU-AW group and non-ICU-AW group for researcher A (F=4.53, p=0.035) or researcher B (F=6.51, p=0.011). The test for consistency suggested that accepting CPAx-Chi ≤31.5 and MRC-Score ≤48 as the best cut-off points for the diagnosis of ICU-AW, then kappa was 0.845 (p=0.02) for researcher A, and kappa=0.839 (p=0.04) for researcher B (table 4).
MRC-Score and CPAx-Chi were consistent for the diagnosis of ICU-AW (n=200)
Discussion
Translation
The present study is the first to translate CPAx from English to Chinese using the Brislin model to guarantee sufficient equivalency.16–18 Our study was strengthened by including a multidisciplinary team to remedy content variance, and included two Chinese nurses with English certifications studying, respectively, in the UK and Canada. We undertook tests for criterion validity and reliability for the completed translation.
Validity of CPAx-Chi
Validity is the degree that a measured result reflects the measured content. The more consistent the measured result is with the measured content, the higher is the validity.21 22 According to the handbook of scale development, when the number of experts is more than five, the good standard of I-CVI is more than 0.78, and the experts must be authoritative and coordinated.22 23
The present study involved nine ICU multidisciplinary experts with deep theoretical knowledge and clinical experience. The expert authority coefficient ranged from 0.75 to 0.95. The Kendall synergy coefficient was 0.61 (p=0.842) and I-CVI ranged from 0.889 to 1. Therefore, CPAx-Chi had good content validity.24 25
Corner et al demonstrated that the CVI of CPAx was 1 (p<0.05).11 12 They also showed that CPAx has good predictive validity, and that the CPAx score could be used as an alternative indicator of functional prognosis in critically ill patients by analysing the relationship between the CPAx score and patient outcomes.13 Other colleagues demonstrated the criterion validity of CPAx taking the scores for the MRC, Short Form (SF)-36, Sequential Organ Failure Assessment (SOFA) and GCS as a standard.26 They found that the correlation coefficient between the CPAx score and MRC-Score was 0.65 (p<0.001). The correlation coefficients for the right upper limb, left upper limb, right lower limb and left lower limb with the CPAx score were, respectively, 0.69, 0.64, 0.69 and 0.67. The correlation coefficient between the CPAx score and SOFA score was 0.68 (p<0.001). The correlation coefficient between the CPAx score and GCS was 0.74 (p<0.001). The correlation coefficient between the physical-function item of SF-36 and the CPAx score was 0.72 (p=0.013). The correlation coefficient between the mental-function component of SF-36 and the CPAx score was 0.024 (p=0.95). In the present study, the correlation coefficient between the CPAx-Chi score and the items of the MRC-Score ranged from 0.60 to 0.65 (p<0.001). Therefore, CPAx-Chi had good validity.
Reliability of CPAx-Chi
Cronbach’s α mainly reflects the internal consistency of a scale.21 22 In general, Cronbach’s α should be >0.7; a value <0.6 indicates that the items of scale must be revised. From the perspective of psychometrics, the ‘ideal’ Cronbach’s α should be >0.8.27–29 The inter-rater reliability mainly demonstrates the consistency of evaluation results among different evaluators, and the stability of scales used among different evaluators.30 31 An inter-rater correlation coefficient >0.7 indicates that the inter-rater reliability is good. The inter-rater correlation coefficient ranging from 0.8 to 0.9 indicates that the inter-rater reliability is high.14 21 22 31 In the present study, Cronbach’s α for CPAx-Chi was 0.939, and the inter-rater reliability of the CPAx-Chi score was 0.902 (p<0.001). The inter-rater correlation coefficient was >0.8 for the items of respiratory function, transfer from bed to chair and grip strength. The inter-rater correlation coefficient of other items of CPAx-Chi was all >0.7. Therefore, CPAx-Chi had good reliability.
Best cut-off point, sensitivity and specificity of CPAx-Chi
Typically, evaluation of diagnostic performance is based on the ROC curve and AUC. If the AUC of a certain scale is 1, then it is considered to be a ‘perfect’ diagnostic tool, but the perfect tool does not exist in the real world. Hence, if the AUC of one scale ranges from 0.85 to 0.95, then the measurement effect of the scale is very good. If the AUC of one scale ranges from 0.5 to 0.7, then the measurement effect of the scale is considered to be undesirable. If the AUC of one scale is 0.5, then the measurement effect of the scale is barely functional.32–34 Our experts regarded an MRC-Score ≤48 as the standard to diagnose ICU-AW. First, some studies have demonstrated the value of diagnostic ICU-AW using the Barthel Index,35 grip strength,36 ICU Mobility Scale,37 de Morton Mobility Index38 and the Physical Function Intensive Care Test39 using MRC-Score ≤48 as the standard. Second, the best cut-off point, sensitivity and specificity of neuromuscular ultrasound, electrophysiological recordings, electromyography and other objective diagnostic methods used to diagnose ICU-AW have been verified using MRC-Score ≤48 as the criterion.26 40–42 Third, scholars have constructed several models of early prediction of ICU-AW by taking MRC-Score ≤48 as a diagnostic criterion.43–45 In the present study, the best cut-off point for the diagnosis of ICU-AW with CPAx-Chi was 31 points. This was verified by taking MRC-Score ≤48 as the criterion, and the sensitivity and specificity were good.
The kappa statistic quantifies inter-rater reliability for ordinal and nominal measures. In general, a kappa value between 0.40 and 0.60 indicates ‘moderate’ agreement, between 0.61 and 0.80 denotes ‘substantial’ agreement, and >0.81 reflects ‘excellent’ agreement; a negative value for kappa represents disagreement.46 47 The concordance of the kappa value was high when taking the MRC-Score ≤48 and CPAx-Chi ≤31 as the best cut-off points to diagnose ICU-AW for researcher A and researcher B.
Strengths and limitations
This is the first study about Chinesisation, adaptation and validation of the CPAx in critically ill patients. Second, there were two researchers assessed and collected data independently, which improved the reference value of the validation data. Third, this is the first study that demonstrated the best cut-off point for the diagnosis of ICU-AW using CPAx-Chi. However, there are some limitations in the study. First, this is a non-randomised pool of participants chosen primarily by their availability during the study period. Second, there were specific exclusion criteria that may have stopped the potential ‘ceiling and floor’ effects of CPAx-Chi to be tested. Third, there is no ‘gold standard’ for diagnosis of ICU-AW. but we took MRC-Score ≤48 which was the most widely used ‘standard’ for ICU-AW as the criterion to demonstrate the best cut-off point for the diagnosis of ICU-AW using CPAx-Chi. In the future, we need to take other tools just like the maximum inspiratory pressure, MRC-Score, electromyography and neuromuscular ultrasound as the criterion to further demonstrate the best cut-off point for the diagnosis of ICU-AW using CPAx-Chi.
Conclusions
We have demonstrated that CPAx-Chi had high criterion validity and reliability for assessing ICU-AW in adult patients in the ICU. CPAx-Chi showed high sensitivity and specificity in assessment of patients at risk of ICU-AW at a recommended cut-off of 31 points. To further confirm the clinical value of CPAx in assessing and diagnosing ICU-AW, it must be applied together with the MRC-Score, ultrasound, electrophysiology and electromyography. Also, multicentre, large-sample and randomised trials are needed to verify the best cut-off point for CPAx.
Data availability statement
Data may be obtained from a third party and are not publicly available. Data can be requested from the Ethics Committee (contact via the First Hospital of Lanzhou University, Lanzhou, Gansu, China; email: ldyylwh@126.com) for researchers who meet the criteria for access to confidential data.
Ethics statements
Ethics approval
The study was reviewed by the Ethics Committee of the First Hospital of Lanzhou University (LDYYLL2019-232) in Lanzhou, China.
Acknowledgments
The authors gratefully acknowledge all the participants. The authors gratefully acknowledge Charlesworth Author Services which had helped us to revise the language in the manuscript.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
ZZ, GW, JG and HW are joint first authors.
Contributors ZZ and YW conceived of the study, participated in its design and coordination, and helped to draft the manuscript. GW, HW and BJ performed the experiment and investigation. ND and JT participated in the design of the study and performed the statistical analysis. BL and JG participated in the project administration. WY and JT participated in the manuscript editing and review. All authors read and approved the final manuscript.
Funding This study was based on projects of: (1) the Health Commission of Gan Su Province (project no. GSWSHL2020-11) and (2) the First Hospital of Lanzhou University (no. ldyyyn2019-104).
Disclaimer The funding source had no involvement in data collection/analyses or manuscript preparation.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.