Article Text
PDF
Abstract
Objectives We pilot-tested the VISually Independent test battery Of NeuroCOGnition (VISION-Cog) to determine its feasibility, comprehensibility and acceptability in evaluating cognitive impairment (CI) in visually impaired older Asian adults.
Design The VISION-Cog was iteratively fine-tuned through pilot studies and expert-panel discussion. In the first pilot study (Stage 1), we recruited 15 visually impaired and cognitively normal participants aged ≥60 years to examine the pilot VISION-Cog’s feasibility (length of time to administer), comprehensibility (clarity of instructions) and acceptability (participant burden). We then presented the pilot results to the expert panel (Stage 2) who decided via agreement on a revised version of the VISION-Cog. Subsequently, we conducted a second pilot study (Stage 3) on another four participants to ascertain improvement in feasibility, comprehensibility and acceptability of the revised version.
Setting Singapore Eye Research Institute.
Participants Nineteen Asian adults aged ≥60 years with visual impairment (defined as near visual acuity worse than N8) were recruited.
Outcome measure Revised VISION-Cog.
Result The VISION-Cog was deemed feasible, taking approximately 60 min to complete on average. All participants agreed that the test instructions were clear, and the battery did not cause undue discomfort or frustration. The data collector rated all tests as very user-friendly (score of 5/5). Minor modifications to the pilot VISION-Cog were suggested by the panel to improve its safety, clarity of instructions and content validity, which were incorporated and iteratively tested in the second pilot study until no further issues emerged.
Conclusions Using an iterative mixed-methods process, we have developed a feasible, comprehensible and acceptable 5-domain and 9-item visually independent VISION-Cog test battery suitable to assist CI diagnosis in older adults with visual impairment. We will assess its diagnostic potential against clinician-based assessment of CI in subsequent phases.
- dementia
- neurology
- dementia
- neuro-ophthalmology
- delirium & cognitive disorders
- old age psychiatry
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This study follows a mixed-methods multistage methodology to ensure a scientifically robust development process.
The revised VISually Independent test battery Of NeuroCOGnition (VISION-Cog) was approved by a panel of expert members from clinically diverse backgrounds to ensure content validity and clinical relevance.
Our pilot studies incorporated feedback from a diverse range of participants across a spectrum of educational levels and visual status to ascertain the appropriateness of our tests for visually impaired older adults.
The results would be more generalisable if Malay-spealking and Tamil-speaking participants were included.
A real-world implementation study may be needed to comprehensively assess the feasibility and acceptability of the VISION-Cog in clinical settings.
Introduction
With the global upsurge in ageing populations,1 the number of individuals with visual and cognitive impairments (VI and CI) is estimated to double and triple by 2050, respectively.2 3 Clinicians will therefore frequently encounter older adults with both VI and CI as one in every three older adults with CI is visually impaired.4 5 Currently, cognitive assessment of visually impaired individuals is challenging because screening questionnaires (eg, Mini-Mental Status Examination (MMSE) and Montreal Cognitive Assessment) and diagnostic neuropsychological batteries (eg, Vascular Dementia Battery (VDB) or Repeated Battery for the Assessment of Neuropsychological Status) depend heavily on visual input.6–8 Importantly, patients with VI have been reported to perform significantly better on visually independent cognitive tests.9 As such, there is an urgent unmet need for more accurate diagnostic instruments to support the clinical diagnosis of CI in visually impaired older adults.
Although several visually independent neuropsychological tests to evaluate CI in this special population have been developed previously, they have inherent limitations. For example, the Haptic Intelligence Scale and Tactual Progressive Matrices, are no longer commercially available,10 while commercially available batteries, such as the Cognitive Test for the Blind and the Neuropsychological Assessment of Adults with Visual Impairment,11 12 were constructed primarily for young adults, and therefore comprise cognitively demanding tests inappropriate for older adults due to complicated instructions, high levels of difficulty and long durations of assessment.11–13 To date, valid and reliable neuropsychological test batteries suitable for evaluating CI among older adults with VI are unavailable.
To address this important gap, our group has developed, using a multiphase, mixed-methods approach, a pilot version of the VISually Independent test battery Of NeuroCOGnition (VISION-Cog), a new diagnostic instrument to assess CI in older adults with VI (Phase 1).13 Here, we report subsequent pilot studies to revise and fine-tune the VISION-Cog (Phase 2). This phase consisted of three stages: (1) A study in which the pilot VISION-Cog was first pilot-tested to examine its feasibility, comprehensibility and acceptability; (2) expert-panel discussion to seek agreement on a revised VISION-Cog version; and (3) a second study to ascertain improvement in feasibility, comprehensibility and acceptability of the revised VISION-Cog compared with the pilot version. We hypothesised that at the end of pilot-testing, the revised VISION-Cog would be feasible, comprehensible and acceptable and represent the final version of the VISION-Cog ready for subsequent validation phases.
Stage 1: first pilot study
Methods
Participants
A total of 15 participants were recruited from the ongoing Population Health and Eye Disease Profile in Elderly Singaporeans (PIONEER) study, which is comprised of older adults aged ≥60 years from three major ethnicities—Chinese, Malay and Indian—in Singapore. Details of the study design and methodology have been described previously.14 Participants who were aged ≥60 years, spoke either English or Mandarin fluently and presented with near VI were included. Participants were excluded if they presented with CI, any conditions that would compromise neurocognitive ability, depression, hearing impairment, tactile impairment or motor impairment. Details of the inclusion and exclusion criteria are outlined in online supplemental table S1. The main causes of VI among included participants are shown in online supplemental table S2.
Supplemental material
Visually impaired participants who had previously participated in the PIONEER study were identified. As education and VI were deemed the most likely factors to potentially affect performance of the VISION-Cog tests by our study team, we purposively recruited three to five participants into each of the four groups stratified by educational status (primary and below, secondary and higher) and VI level (mild VI and moderate–severe VI).15 We also aimed to balance other demographic characteristics within each group, including age, sex and language.16 Participant recruitment, test administration and test revision were conducted iteratively until the VISION-Cog met the criteria of feasibility, comprehensibility and acceptability.15 16
The study was performed at the Singapore Eye Research Institute research clinic located at the Singapore National Eye Centre. Written informed consent was obtained from all participants before participation in the study.
Patient and public involvement
Patients or the public were not involved in the design, conduct, reporting or dissemination plans of our research.
Measures
The pilot VISION-Cog is a visually independent neuropsychological battery measuring five cognitive domains, namely memory and learning, language, executive function, complex attention and perceptual-motor. It includes nine visually independent tests, namely the Modified Spatial Memory Test, List Learning, List Recall and List Recognition, Adapted Token Test, Semantic Fluency, Modified Spatial Analysis, Verbal Subtests of the Frontal Battery Assessment, Digit Symbol, Digit Span Forwards and Digit Span Backwards. The test protocols have been described in detail previously,13 and were formulated into a standardised test manual for the current study. The manual was then professionally translated into Mandarin by a certified translation centre using a forward-backward translation procedure to ensure the accuracy of instructions. Moreover, certain phrases were reworded into culturally meaningful alternatives which were semantically and phonemically as similar as possible to the original English phrases (eg, ‘hold your hands’ in English were rephrased as ‘take your hands’ in Chinese).
Procedure
A standard test session included: (1) study explanation and consent taking, (2) eligibility assessment, (3) VISION-Cog administration and (4) cognitive interview. As part of the PIONEER study protocol, sociodemographic, medical and psychosocial data of participants were previously collected. However, participants in the VISION-Cog pilot study were further required to complete the near visual acuity (VA), tactile and motor tests, the Patient Health Questionnaire-9 for depressive symptomatology, and the Modified MMSE Blind as part of eligibility screening.
Binocular presenting near VA was performed under photopic conditions at 40 cm with the participant’s habitual correction using the vision screener (Total EyeCare, Singapore). Near VI was selected over distance VI because the VISION-Cog is performed at the near range of vision. Mild VI was defined as near VA worse than N8 but better than or equivalent to N12 while moderate–severe VI was defined as near VA worse than N12. As educational level and severity of VI were deemed as critical factors affecting test performance, participants were stratified into four groups based on education (primary or lower, second or higher) and severity of VI (mild, moderate–severe).
Pilot VISION-Cog administration
The VISION-Cog was administered individually to eligible and consented participants in the participant’s preferred language by a data collector (author TVA) fluent in both English and Mandarin, with participants using their presenting VA with habitual correction. Time spent completing the study explanation, consent taking and test battery was recorded. The average time to complete each step and the overall procedure was calculated to evaluate the feasibility of the pilot VISION-Cog. The test battery was considered feasible if the total duration did not exceed 90 min as determined by our expert panel in the previous phase.13
Comprehensibility and acceptability evaluation of the pilot VISION-Cog
Cognitive interviews were conducted by the data collector to assess participants’ comprehension and acceptability of the pilot VISION-Cog tests. A semi-structured interview guide with verbal probing questions was developed to elicit information from the participants after the completion of each neuropsychological test (online supplemental table S3). Since this cognitive interviewing added another layer of cognitive burden and additional time required by the participants, we performed the cognitive interviews for the tactile tests only. As all auditory tests have been used clinically in Singapore, the study team deemed that they should have good comprehensibility and acceptability for older adults.17 18
The interviews were audio-recorded, and field notes were written by the data collector. The data collector also observed the participants while they performed the tests and took note of instances of hesitation or difficulty performing the tests. Data obtained from participants were summarised by the data collector and discussed among the study team (TAV, EF, KD, PG, SYQ and EL) after every testing round of one or two participants. Revisions were subsequently made to the VISION-Cog and iteratively evaluated on new participants. The study team determined if the modified version functioned as intended without introducing further difficulties in comprehension and acceptability. The process was repeated until participants did not report any new issues. At the end of each testing round, the data collector rated the user-friendliness of each test using a Likert-type scale from 1 to 5, with 1 being ‘Not user-friendly at all’ and 5 being ‘Very user-friendly’. User-friendliness was assessed using verbal and non-verbal cues from the participants together with their perception on the VISION-Cog’s tactile tests. Additional written comments to support the rating were also provided by the data collector (eg, ‘no frustration or discomfort observed’ for a rating of 5). A comprehensible and acceptable battery should meet the following criteria: (1) the majority of participants can understand and complete the tests without feeling physically uncomfortable or frustrated (≥75%) and (2) the data collector finds the tests user-friendly (score≥3).19
Revision of the pilot VISION-Cog
The study progress was presented in biweekly consensus meetings attended by neuropsychologists and clinical research fellows. After discussion among the members, the tests were changed in line with suggestions from the data collector and participants and retested in a new participant sample. This process was conducted iteratively until no new issues emerged.
Results of stage 1
A total of 15 participants (mean age (SD): 77.3 (6.5), 40% men, 46.7% with primary and lower education and 53.3% English-speaking) were included in the study. An approximately equal number of participants (n=3 or 4) were recruited into each of the four groups stratified by educational status and VI levels (table 1).
Demographic characteristics of included participants in the first pilot study
After administering the pilot VISION-Cog to the first two participants, the data collector noticed that in the Modified Spatial Analysis test, participants relied on their residual vision to look at marker-drawn lines on the tactile cards, instead of using their hands to touch the reference diagrams as intended, to place the wooden shapes into the correct positions. The cognitive interview further supported this observation (Online supplemental file 1, Appendix 1). Thus, the research panel agreed with full consensus to erase the marker-drawn lines on the tactile cards so that participants would rely less on vision and depend more on tactile sense to follow the instructions. After implementing this modification, both cognitive interviews and observations from the data collector showed participants performing the test with the intended approach of using the reference diagrams to construct the required patterns with the wooden shapes (Online supplemental file 1, Appendix 1). Subsequently, the Modified Spatial Analysis test was subsequently performed following this updated procedure.
Table 2 illustrates the average time spent for each test and the overall time taken for the full battery. The mean duration to complete the test-taking formalities and VISION-Cog tests was approximately 6 and 60 min, respectively, resulting in an average of 66 min to complete the test battery. This is within the 90-min time frame of successful implementation determined by our expert panel. As such, our pilot VISION-Cog was deemed feasible.
Average time spent on the pilot VISION-Cog together with data collector’s evaluations from the first pilot study
Moreover, in the cognitive interviews, participants commented positively on all VISION-Cog tests. The instructions were clear and easy to understand (100% of participants). All participants were comfortable with the VISION-Cog set-up and performance and were not frustrated with or confused by the instructions. While some participants mentioned minor frustration with the Modified Spatial Analysis (Form Matching and Form Matching/Size Transformation) subtests due to their high difficulty level, they were still enthusiastic about these subtests because of their cognitively stimulating nature (Online supplemental file 1, Appendix 1). Due to this issue, the user-friendliness rating of this test was 3/5. In contrast, other tactile tests were rated as very user-friendly (5/5) (table 2). The data collector did not observe any difficulties with participants manoeuvring the tactile material and following the instructions. Overall, the battery met the predetermined comprehensible and acceptable criteria. Results from this pilot study were presented to the expert panel in the subsequent consensus meeting.
Stage 2: expert panel discussion
Methods
The consensus meeting with a multidisciplinary panel was conducted face-to-face on 16 August 2022 for a 3-hour period. The purpose of this meeting was to seek experts’ opinions on the results obtained from the pilot study and revise the pilot VISION-Cog accordingly. Similar to the consensus meeting reported in a prior publication,13 the expert panel consisted of one neuropsychologist, one neurologist, one neuro-ophthalmologist, one geriatrician and one psychiatrist/public health expert. The Informal Consensus Method (ICM), a qualitative technique of judgement aggregation based on majority voting, was used to conduct the meeting.20 Only tactile tests in the pilot VISION-Cog were presented and discussed in this meeting because the study team considered the auditory tests, which have been used clinically in Singapore, to be comprehensible and acceptable for older adults.17 18
During the meeting, each tactile test in the pilot VISION-Cog was demonstrated and the results from the pilot study were presented in turn. The panel subsequently discussed the feasibility, comprehensibility and acceptability of each test. After this discussion, each panel member voted on (1) the inclusion of the test and (2) any modifications needed. If there was disagreement, further discussion was required to resolve the discrepancies, followed by another vote. Similar to the previous consensus meeting, we used the content validity ratio (CVR) to evaluate consensus.13 CVR has a range of −1 to 1 with a higher score demonstrating better agreement. To achieve consensus with five expert members, a minimum CVR of 0.99 was needed.21 This process was iterative for every tactile test until the expert panel fully agreed on a revised version of the VISION-Cog.
Results of stage 2
At the end of the second consensus meeting, the expert panel unanimously agreed on a revised version of the 5-domain VISION-Cog (table 3). In this, all nine tests in the pilot VISION-Cog were included. These comprised: (1) the Modified Spatial Memory Test and List Learning, List Recall and List Recognition for the memory domain; (2) Adapted Token Test and Semantic Fluency for the language domain; (3) Modified Spatial Analysis (Form Matching and Form Matching/Size Transformation) and Verbal Subtests of the Frontal Battery Assessment for the executive function domain; (4) Digit Symbol, Modified Spatial Analysis (With Time Bonus), Digit Span Forwards and Digit Span Backwards for the complex attention domain; and (5) Modified Spatial Analysis (Shape Matching and Puzzle Construction) for the Perceptual-Motor domain. The voting results are shown in online supplemental table S4. The panel also suggested minor modifications for these tests (eg, blunt the pencil and screw tips) to improve safety, clarity of instructions and content validity (online supplemental table S5). To improve the user-friendliness score of the Modified Spatial Analysis (Form Matching and Form Matching/Size Transformation) subtests, the panel recommended a practice trial for these subtests to reduce frustration and mitigate task challenges (online supplemental table S5). Based on the panel’s suggestions, a revised procedure of the VISION-Cog was formulated (online supplemental table S6), which was evaluated in a second pilot study.
Included neuropsychological tests in the revised VISION-Cog
Stage 3: second pilot study
Methods
In the second pilot study, the revised VISION-Cog was administered to another four eligible participants. Similar sampling strategy as the first pilot study was employed. The criteria of feasibility, comprehensibility and acceptability were the same as those in the first pilot study.
Results of stage 3
Of the four participants (mean age (SD): 71.3 (4.0)) who completed the second pilot study, 75.0% were men, 25.0% had primary and lower education and 50.0% were English-speaking (table 4).
Demographic characteristics of included participants in the second pilot study
It took approximately 61 min, on average, to complete the test battery, which is considered feasible (table 5). All participants commented that the revised VISION-Cog had clear and easy-to-understand instructions. Moreover, they were comfortable with performing the VISION-Cog and were not frustrated with or confused by its instructions. Furthermore, all tactile tests were rated as very user-friendly by the data collector (table 5). Overall, the revised VISION-Cog was deemed comprehensible and acceptable.
Average time spent on the revised VISION-Cog together with data collector’s evaluations from the second pilot study
Discussion
Following an iterative procedure of pilot testing and expert panel discussion, we have formulated a final version of the 5-domain and 9-test VISION-Cog: a feasible, comprehensible and acceptable visually independent neuropsychological battery to assess CI in older adults with VI. The diagnostic performance and psychometric properties, such as concurrent validity, inter-rater reliability and test–retest reliability, of the VISION-Cog will be evaluated in subsequent phases to assess its potential to replace visually dependent neuropsychological batteries and support the diagnosis of CI in visually impaired older adults.
Our final VISION-Cog consisted of nine tests covering five cognitive domains of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, including memory, language, executive function, complex attention and perceptual-motor. The 1-hour average test battery duration is comparable to other visually dependent batteries currently in research and clinical use such as the VDB and 10/66 Dementia Research Group Test Battery,18 22 indicating that older adults will be able to comfortably complete the VISION-Cog in clinical settings. Moreover, 100% of participants reported that our test battery was comprehensible (good clarity of instructions) and acceptable (reasonable level of cognitive burden), further supporting the suitability of our battery for older adults.
The Form Matching and Form Matching/Size Transformation subtests were reported to cause some frustration in our older adult pilot participants; however, this was due to their high difficulty levels rather than the clarity of instructions. Frustration is an unpleasant feeling that occurs when one fails to achieve the goal of his action.23 While frustration may impede thinking and lead to premature task discontinuation, many participants in our study did not give up on the subtests and even reported enthusiastic engagement with difficult trials. This may be explained by the concept of desirable difficulties, which may promote comprehension, thinking and learning.24 Some degree of difficulty is beneficial for individuals to develop a more sophisticated understanding of the tasks and perform them better subsequently.25 As such, we did not change any components of these subtests. Moreover, participants’ inability to achieve full scores due to the difficult trials also prevented a ceiling effect from occurring in these subtests. At the same time, the first trial of each subtest was designed to be easy so that everyone could perform accurately, thereby limiting the floor effect.
The consensus meeting was organised using the ICM,20 a qualitative approach that has been commonly employed to achieve consensus on clinical guidelines and medical research discussions,20 and which has proven efficiency in reaching consensus for a small expert panel (n=5). However, a possible disadvantage of the ICM is that a dominant authority figure among the panel members might exert greater influence on group interaction, causing the results to favour the dominant voice while others might not openly share their opinions. To circumvent this issue, we chose our distinguished panel members with recognisable expertise in different psychological and medical fields so that their voices were treated with equal weight. Moreover, the group dynamics were already well-established because panel members had worked together during a previous consensus meeting and 6-month post-consensus-meeting discussion to formulate the pilot VISION-Cog with our research team.13 Furthermore, to improve the robustness of the ICM’s results, we evaluated consensus using the CVR, a well-known statistical method to achieve agreement,21 instead of using majority voting to imply consensus.
The strengths of our study include a multistage and multiphase process guided by both inductive and deductive approaches that contributed to a scientifically robust development of the VISION-Cog. Moreover, the content of our test battery was unanimously approved by a panel of expert members from clinically diverse backgrounds, further assuring its clinical relevance and content validity. In addition, our pilot studies incorporated feedback from a diverse range of participants across a spectrum of educational levels and visual status to ascertain the appropriateness of our tests for visually impaired older adults. Nonetheless, potential limitations of our study include a lack of Malay-speaking and Tamil-speaking individuals which may compromise the generalisability of our test battery. Thus, we will culturally adapt the VISION-Cog for Malay, Tamil and non-Asian populations in future studies. Second, we chose a narrow definition of feasibility (duration of test completion) and acceptability (participant burden) which was suitable to evaluate the development process of our test battery. Assessment of feasibility and acceptability may need to be broadened in a real-world implementation study where our battery is incorporated into a clinical diagnostic workflow of CI evaluation. For example, feasibility may also be defined by how often participants require breaks or the number of times the test battery was completed when offered.26 Acceptability can also be evaluated more extensively using a theoretical framework of acceptability.27 As such, we will also re-examine the feasibility and acceptability of the VISION-Cog using an implementation science approach as a secondary aim in the subsequent phase of determining the VISION-Cog’s diagnostic performance in a clinical setting.
In conclusion, after an iterative process of pilot testing and consensus meeting, our revised 5-domain 9-test VISION-Cog is feasible, comprehensible and acceptable. The VISION-Cog is a promising battery capable of replacing vision-dependent neuropsychological batteries and supporting the clinician-based diagnosis of CI in visually impaired older adults. In future work, we will recruit a large sample of visually impaired older adults to determine its diagnostic performance. The VISION-Cog will be compared against (1) a visually dependent battery to establish any diagnostic accuracy improvement and (2) the current clinical standard to assess concurrent validity.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study received ethical approval from the SingHealth Institutional Review Board (CIRB, Reference #2016/3089). All protocols were performed in accordance with the Declaration of Helsinki principles.
Acknowledgments
We thank the PIONEER team for their contribution to this project.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors TAV, EF, KD, PG and EL designed and conducted the research. SYQ, CC, ST, ASLN, PY, DY, DM and EL provided support and critical appraisal throughout. EL provided the overall supervision of the research and is the guarantor of the content. All authors contributed to data interpretation and the writing of the manuscript.
Funding This work was supported by the Singapore National Medical Research Council (NMRC-CSA-SI #JRNMRR140601). EL is supported by the NMRC Senior-Clinician Scientist Awards (#NMRC/CSASI/0009/2016). The grant funding bodies had no roles in the design, conduct or data analysis of the study, nor in manuscript preparation and approval.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.