Article Text
PDF
Abstract
Objectives Research has shown that having adequate quantity and quality of physical activity can contribute to the health and well-being of children. Nonetheless, existing tools to measure these constructs in children have limitations in terms of their objectivity and scalability. In this study, we provide criterion validity evidence of two systems built on commercially available sensors (ie, gyroscopes and infrared cameras), designed to measure children’s moderate-to-vigorous physical activity and fundamental movement skill proficiencies.
Design Cross-sectional.
Setting Primary schools in Hong Kong.
Participants Data from 30 (age=8.55±1.25 years) and 1174 (age=9.15±1.63 years) children were included for the validation of physical activity and fundamental movement skills measures, respectively. Children’s outcomes were simultaneously measured using the developed systems and existing, well-established measures (accelerometers and expert ratings).
Results We found a strong correlation between physical activity outcomes measured using our developed system and accelerometers (Pearson r=0.795). Motor skill proficiency scored using our real-time rating system had strong agreement with expert ratings (percentage agreement=84%–94%, kappa=0.661 to 0.859).
Discussion Results of the current study supported the application of the respective systems in physical education and large-scale research studies. Collection of such data at mass levels could help researchers depict the complex relation between children’s quantity and quality of physical activity.
- community child health
- preventive medicine
- public health
Data availability statement
Data are available upon reasonable request. Data used for analyses in the paper will be available upon reasonable request made to the lead author.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Used well-established tools (ie, accelerometry for physical activity, expert ratings of recorded fundamental movement skills) to generate comparative measure to examine criterion validity of developed tools.
Measurements taken in real-life school settings.
Application of accessible, consumer grade devices.
Modest validation sample size for physical activity measures.
Introduction
Regular engagement in physical activity (PA) is related to a myriad of beneficial health outcomes in children.1 One construct previously found to be related to PA engagement in children and adolescents is fundamental movement skills (FMS).2 3 FMS comprise of basic movements such as balancing, running, jumping, throwing and catching. They are considered as the ‘building blocks’ for more complete movement skills or patterns that are required for common sport and exercise activities. As such, children who are competent in FMS may be able to take up complex sport skills more easily and rapidly. This will potentially enhance their current and future participation in sport and PA.4 5
Typically, FMS can be broadly categorised into three types, namely stability, locomotor and ball skills. Stability skills involve one’s ability to maintain balance under various static postures (eg, stand on one leg) or in motion (eg, twisting). Locomotor skills refer to movement skills that involve the transfer of body position from one point to another (eg, running and jumping), while ball skills (also referred to as object control skills) refer to one’s ability to effectively manipulate balls or other related objects (eg, rackets and bats). Research has shown that children and adolescents’ PA might be related to their competence in locomotor skills, or ball skills or both.6–11 Some researchers have also found that competence in these skills may contribute to other health and well-being outcomes.2 12 However, in a meta-analysis, Barnett et al3 suggested that there was insufficient evidence to support PA and FMS competence of children being associated. The mixed findings in the literature suggest that the relation between these constructs is complex and warrants further investigation. Nonetheless, standardised, objective and low-cost measurement tools for these outcomes had been lacking previously. Such measurement issues limit generalisability and comparability of research findings, and in turn limit the scale and design of potential studies aiming at depicting the association between PA, FMS and other constructs.
Measurement of physical activity and fundamental movement skills
With a goal of improving children’s health and well-being, it is imperative for researchers and practitioners (ie, physical education teachers) to monitor children’s PA and developments in FMS effectively and efficiently. In the extant literature, there are many established methods for measuring these key constructs. For example, PA of children could be measured using self-report questionnaires or objective measures, such as pedometers or accelerometers.13 These measures, nonetheless, have their shortcomings. For example, self-reported questionnaires are susceptible to recall difficulties or reporting biases. Accelerometers, by contrast, provide data that are valid and objective, yet they are typically administered for shorter periods (eg, 5–14 days) and involve heavy data processing, and hence are not appropriate for mass application over extended periods. As such, some researchers have turned to Internet of Things (IoT) for the acquisition of PA data.14 15 For example, smartphones or wrist-worn activity trackers paired with mobile devices could be used for data collection and transfer over the cloud for research purposes. Such approaches could be adopted for large-scale studies involving continuous activity tracking. Nonetheless, to the authors’ knowledge, most of these systems require each user to possess a mobile phone for data transfer. Despite the accessibility of mobile devices to adults nowadays, most primary school-aged children do not own such devices. Hence, continuous tracking of children’s PA in a large scale remains to be a challenge for researchers.
Measurement of children’s FMS also has its challenges. Currently in the literature, there are two approaches of measuring FMS. Product-based assessments typically involve measuring the time or score generated from completing a series of movement tasks or an obstacle course.16–18 Whereas process-based measures encompass those that are scored based on whether children met preset criteria during their performance of movement skills.19 Generally speaking, product-based assessments have the advantage of being less time-consuming and requiring less expertise, in terms of movement education, of assessors. By contrast, process-based assessments put more emphasis on the quality of movement, and many of them require trained experts to conduct the scoring, hence the scores derived would be susceptible to inter-rater differences. Furthermore, most protocols also require movement behaviours to be video-recorded and rated retrospectively. Accordingly, the time required to complete ratings would also be longer.
To address the challenges and shortcomings of PA and FMS measurements, in particular the barriers to rapid and mass collection of reliable and valid data related to the quantity and quality of children’s PA, two systems were developed to collect data for the two constructs, respectively, in a large sample of primary school-aged children. Both data collection systems were developed under the Fun to Move@JC project.20 Fun to Move@JC is a project designed to enhance PA, or more broadly physical literacy,21 of primary school-aged children and their parents in Hong Kong. Apart from providing teacher professional development training and provision of resources to support PA of children in schools and at home, the introduction of information technology to support physical education and children’s PA engagement was also a key objective of the project. In particular, a system was developed to measure PA of children and parents using the Fun to Move@JC Sport Band (figure 1), a wrist-worn activity tracker developed under the project. In parallel, a Fundamental Movement Skill Rater (FMS Rater; figure 2) was developed to provide rapid, objective assessments for children’s FMS. In this paper, we will provide the underpinnings of the two developed systems and evidence to support the criterion validity of data collected correspondingly.
The Fun to Move@JC Sport Band.
The user interface of the fundamental movement skill rater.
Fun to Move@JC Sport Band
The Fun to Move@JC Sport Band is an IoT system developed to collect PA. The developed device is a wrist-worn activity tracker which measures time-based step counts. Specifically, step counts recorded by the device are stored in 15 s epochs. Each epoch of time-based step counts, when transferred to the cloud-based data platform, will be converted to a corresponding PA intensity (eg, light, <3 metabolic equivalent; moderate-to-vigorous, >3 metabolic equivalent) using a big data engine developed for the project. The cut-off values for PA intensities were derived through a series of calibration tests with primary school-aged children, conducted in both laboratory and field settings, using results from accelerometry and cardiopulmonary testing as criterion measures. Data stored in the devices can be transferred to the servers using a companion mobile application. Alternatively, to reduce the reliance on mobile devices for children, Sport Band data are also synchronised to our cloud servers using gateways placed in classrooms of children. Specifically, Raspberry Pi (Raspberry Pi Foundation, Cambridge, UK) devices were programmed to serve as data transfer gateways. Data stored in Sport Bands in proximity (ie, inside the classroom) will be downloaded to the gateways, and then transferred to our cloud servers. This data transfer protocol eliminates the need for mobile devices, which most children do not possess, to gather data from the deployed wearables.
Fundamental movement skills rater
The rating system developed scores children’s performances in various FMS using a process-based approach. Movement skills performed are captured using the Kinect (Microsoft, Washington, USA) infrared video camera, and scored using a set of rule-based criteria adopted from the Test of Gross Motor Development V.3 (TGMD-3).22 Specifically, children are asked to perform either a locomotor or ball skill facing the camera. Markers were placed on the floor to guide students to their starting position for the assessments. Except for the lead-in run for the skill Kick, all movement skills were performed approximately 3–5 m from the camera to ensure body joints can be accurately tracked by the sensor. The 3D coordinates of joints are captured using Kinect, and the movement sequences will be scored against a set of preset criteria (see table 1). Another Kinect-based system had been developed to measure FMS in the past23 and has shown promising results. However, the previous system23 only measured locomotor skills, whereas the current FMS Rater has the capacity to measure both locomotor and ball skills. The current system provides real-time scoring of skills, and therefore increases the efficiency of FMS assessments, and can also be used as a tool to support teachers’ instruction during physical education. The scores and videos captured from all assessments are uploaded to the same data platform that stores Sport Band data. All captured data can be reviewed by physical education teachers and summarised through a web-portal. As such, physical educators could track the performances of children, either individually or collectively, over time.
Demographic information of participants
In the current study, we provide criterion validity evidence of PA and FMS outcomes measured using the Fun to Move@JC Sport Band and FMS Rater, respectively. In terms of the validity evidence for the FMS Rater, the current paper focuses on that in relation to ball skills, as results from a similar system for locomotor skills had been reported previously.23 We also examined the associations between PA and FMS outcomes in a sample of primary school-aged children. We hypothesised that moderate-to-vigorous physical activity (MVPA) measured using the Sport Band will be positively associated with FMS scores.
Method
Patient and public involvement
The public was not involved in the design, or conduct, or reporting, or dissemination plans of our research.
Participant and procedures
All participants in this cross-sectional study were attending primary schools in Hong Kong. Parents of all participants provided written informed consent to take part in the Fun to Move@JC project and the studies described in this paper. All research protocols of the study were reviewed and approved by The Joint Chinese University of Hong Kong—New Territories East Cluster Clinical Research Ethics Committee (Ref.: 2017.515). The validation sample for the Sport Band included 30 children (53% boys; mean age=8.55±1.25 years; see table 1 for details). Participants were invited to wear the Sport Band on the wrist, and a wGT3X-BT (ActiGraph, Florida, USA) at their right hip simultaneously over a 7-day period. Data for MVPA measured using the research-grade accelerometers were extracted using standardised criteria for children.24 Daily accelerometer-measured PA was considered valid if participants wore the devices for at least 480 min (ie, 8 hours) within the day. By contract, Sport Band data for a day were considered valid if (i) there were at least 240 non-zero 15 s epochs and (ii) the first and final non-zero epochs of the day were at least 8 hours apart. The unit of output and analyses was children’s time spent in MVPA per calendar day. Therefore, larger values for this continuous variable represent engagement in more MVPA. Data collected using both sets of devices over a 7-day period were used for comparison purposes.
The validation sample for the FMS Rater included 1174 children (52% male; age=9.15±1.63 years; see table 1 for details) from grades 1–6. Measurements were administered during school physical education classes by trained research assistants. Due to time constraints, each child only completed assessments for one to three ball skills during one physical education lesson. For each assessment, children were first shown a video containing a ‘correct’ demonstrating of the skill, and were then asked to replicate the skill. Children performed each skill once, and their performances were rated using the FMS Rater. In addition, the same performances were also video-recorded and rated using traditional observational methods. These ratings were conducted by two expert raters independently, where each child’s performance was rated by one of the raters only. The raters were graduates of sports science and physical education and were trained by the authors to assess FMS. Specifically, both raters received a 1 hour training by an author of the paper. During the training session, they were introduced to general methods of conducting process-based assessment of motor skills and were guided through the assessment criteria for all skills. At the end of the training, the raters were provided with, for each skill, 8–10 video clips of children’s FMS performance. The trained raters were then asked to rate all skill performances based on the set criteria individually. Their scores were then compared with a set of results that were agreed between multiple authors of the study. Accordance with previous practice,7 both raters have established over 90% coding reliability with the authors when applying the TGMD-3.22 Prior to this study, the two raters had 2 and 5 years of experiences, respectively, in FMS assessments on several hundreds of children using the TGMD-3. Since some assessment criteria were modified to align with those used in the FMS Rater in this study, a pilot test was conducted specifically for this study. The two raters each rated 20 children performances per skill independently, and their scores reached an agreement above 90% for each skill. They also met and discussed discrepancies in their ratings, and agreed to standardised approaches for scoring in this study. To examine the criterion validity, scores derived from the FMS Rater were compared with ratings conducted by expert raters. Scores by criteria within each skill, and the overall skill score, were used for statistical analyses.
Data analyses
Pearson correlation was calculated to determine the strength of the linear association between MVPA scores captured using accelerometers and Fun to Move@JC Sport Bands. Data for a day were included for analyses only if the corresponding accelerometer-measured and Sport Band-measured MVPA were considered valid. In terms of FMS data, the percentage agreement and kappa coefficients for criteria of each skill (parameterised as 0=criterion not met; 1=criterion met), scored by the FMS and expert raters, respectively, were calculated. The kappa coefficient accounts for expected agreement by chance and hence is a more robust measure for examining agreements.25 In this study, kappa values between 0.60 and 0.79 were treated as moderate, while those above 0.80 were considered strong.25
As an exploratory attempt, we also examined the relation between children’s Sport Band-measured MVPA and FMS scores derived from the developed rating system. Data for this analysis were drawn from the cloud storage, where data collected from both systems were synchronised to. To ensure the representativeness of the collected data, we only included data from children who had valid Sport Band data (using the criteria mentioned above) on at least 5 weekdays and 2 weekend days. To account for differences in terms of children’s attributes and skill difficulty, FMS scores were standardised by skill, and by children’s grade and sex prior to analyses. Specifically, the score of each FMS assessment was converted to a z-score based on the child’s grade level, sex and the skill. An aggregated score for FMS was then calculated by taking the mean of z-scores from all assessments. Pearson correlation between MVPA and FMS scores was then computed to examine the linear relation between these two outcomes.
Results
Validation of sport band scores
Under free-living conditions, participants provided 55% and 78% of valid daily accelerometer and Sport Band data, respectively. In combination, valid information on both devices were obtained in 93 (44%) participant-days. All participants had at least 1 day with both valid accelerometer and Sport Band data, which were used for comparison purposes. Within this sample, five children reported playing piano while wearing the Sport Band. Similar to many wrist-worn devices, the forearm-only movements involved in instrument playing would also be categorised as PA behaviours, leading to overestimates of overall MVPA. Therefore, for the purpose of the current paper, data from these children were removed, resulting an effective sample of 25 children. In this final sample, the mean MVPA measured using accelerometers and Sport Band were 43.8±25.0 min and 49.2±20.4 min, respectively. That is, compared with accelerometers, the MVPA measured using Sport Bands overestimated by 5.41 min per day.
The Pearson correlation between accelerometer-measured and Sport Band-measured MVPA was r=0.795 (p<0.001). However, results of the Shapiro-Wilk test suggested that both sets of data were non-normal (ps<0.05). As such, the Spearman correlation between the two measures was calculated. A significant correlation between was also found with Spearman’s ρ=0.460 (p=0.021).
Validation of FMS rater scores
The overall percentage agreement between ball skill scores measured using the FMS Rater and those scored by expert raters were between 84% and 94%, while the corresponding kappa coefficients were between 0.661 and 0.859 (detailed results are presented in table 2). Within the developed assessment schemes for the seven skills, five included exactly one criterion that was scored by the test administrator, as these criteria either involved fine bodily movements or ball movements that cannot be detected by the infrared sensor. When the ratings for these assessment criteria were excluded, the percentage agreements ranged between 80% and 91%, while the kappa coefficients were from 0.508 to .821.
Scoring criteria of the fundamental movement skill rater and the respective rating accuracies based on performances of 1174 children
Additional analyses: association between sport band and FMS rater scores
We examined the association between Sport Band-measured MVPA and FMS scores measured using the FMS Rater between May and July 2021. During this period, 75 children had both valid Sport Band and FMS data collected. The Pearson correlation between Sport Band-measured MVPA and standardised FMS scores (by students’ sex and grade level) was r=0.217 (p=0.067).
Discussion
While the application of information technology to assist teaching in contemporary education has risen, less research and development had been placed on physical education specifically. As such, we developed a series of tools that could be applied to physical education classrooms and beyond to measure the quantity (ie, MVPA) and quality (ie, FMS) of children’s PA. In this study, we provided criterion validity evidence of the Fun to Move@JC Sport Band and FMS Rater to measure children’s MVPA and FMS proficiency, respectively. We demonstrated that MVPA measured using our Sport Band was strongly associated with accelerometer-measured outcomes. Also, children’s FMS measured using the FMS Rater also showed moderate to strong consistency with expert-rated scores.
In the extant literature, accelerometers are one of the most common objective measure of PA with strong validity evidence. Despite its accuracy, administrating accelerometers to a large population, especially for extended periods, is challenging due to the complexity in terms of data extraction and analyses. Such difficulties can sometimes be overcome by using wrist-worn activity trackers or smart watches,26 yet data collection would rely on participants having compatible mobile devices, and being compliant to synchronising data regularly. For primary school-aged children, these remain to be important barriers. Nonetheless, the Fun to Move@JC Sport Band has potential of overcoming these challenges due to the simplified data collection approach employed. In this study, we found that data collected from Sport Bands are strongly related to accelerometer-measured MVPA, while also being able to capture valid data on a higher proportion of days compared with its counterpart (78% vs 55%). We acknowledge that the compliance rates of accelerometers have been low when compared with other studies conducted locally.27 28 Nonetheless, the valid rate of Sport Band in the current study is comparable to, or better than, accelerometer compliance in previous studies. In fact, previous research has shown that wrist-devices, compared with hip-worn devices, may lead to better compliance.29 Therefore, our results suggest that the developed device can be used for long-term monitoring of children’s PA.
With regard to results related to the FMS Rater, our results suggested that assessment results of ball skills from the developed system have a high level of agreement with expert ratings. Using simple biomechanical principles, this application allows teachers to provide real-time assessment to children’s performances in FMS. In fact, physical educators cited the lack of FMS knowledge, insufficient support and time requirements as major barriers to assessments of these skills at schools.30 The developed FMS Rater has potential to overcome these barriers since the tool does not require teachers to have an in-depth knowledge in FMS. The time required to complete the assessments, and that to receive feedback and results, can also be reduced greatly. Also, the display on the FMS Rater clearly indicates the criteria which children completed successfully (or not). This reminds teachers what the key criteria to a successful performance include, which can also be used as instructional cues to assist their teaching. In combination, physical educators’ assessment literacy can be improved through using the rating system. In turn, this may improve teaching practices and children’s skill acquisition.31
Through continuous monitoring of children’s MVPA and regular FMS assessments, researchers will have an appropriate infrastructure to examine the association between the interaction of these outcomes. In fact, research examining the association between these outcomes had resulted in inconsistent results. While some studies found positive associations between these outcomes, some others did not.3 Holfelder and Schott9 suggested that other factors such as perceived skill competence and socioeconomic status may also be salient factors that are associated with FMS proficiency. By greatly improving the efficiency and accessibility of FMS assessments, large-scale studies will become less logistically challenging. Moreover, the continuous, long-term stream of related data may allow researchers to examine the causal or reciprocal nature of these variables.
Despite evidence supporting the validity of MVPA and FMS measures, this study has several limitations. First, we found that on days when participants reported playing piano, their Sport Band-measured MVPA may be greatly overestimated. In our study, such data were excluded from the analyses, and hence such impact could be minimised. The absence of such information would negatively impact the validity of Sport Band-measured MVPA. In fact, if these cases were not excluded, the correlation between Sport Band-measured and accelerometer-measured MVPA would be reduced to r=0.461. Therefore, when used to examine cross-sectional associations, participants should be asked to report whether they might have participated in such activities, so their data could be appropriately adjusted for. Further investigation is also needed to examine how estimates for changes in MVPA over time might be affected by activities involving upper arm movements predominately.
Second, the sample sizes for some analyses in our study were relatively small. Despite observing strong correlations between Sport Band-measured MVPA and the criterion measure, there is a need to expand the pool of participants in terms of age and PA levels. Repeating the procedures in a wider of participants will also expand the external validity of our findings. Also, when examining the relation between Sport Band-measured and FMS Rater-measured outcomes, there was only a modest number of children (n=75) with data for both indicators. Results from a post-hoc power analysis suggested that the observed power for this analysis was 0.455, falling short to the common benchmark of 0.80. As such, the results of the current study should be interpreted with caution. Unfortunately, this was partially due to the restrictions and reductions to school physical education as a result of the COVID-19 pandemic. In fact, most children with FMS were only assessed on one to two skills. The overall scores generated for comparison purposes may not be sufficiently representative. To further examine the relation between FMS and MVPA, more skills per child should be assessed. If sufficient data become available, the relation between distinct FMS and activity behaviours could also be examined.
Information technology have greatly impacted our education and general lives, but few developments on this front have been made to support physical educators. In this paper, we described and provided validity evidence of two systems to measure primary school-aged children’s PA quantity and quality, respectively. These systems allow teachers and researchers to better understand children’s physical developments and behaviours, which are related to their health. The systems could benefit teachers by improving the quality of their instruction and provision of feedback to children. Researchers could lend on the developed systems to examine interactions between children’s activity behaviours, their motor developments, physical fitness and other psychosocial variables (eg, perceived competence and motivation). Finally, the devices could promote self-monitoring of PA in children and make PA and physical education more interesting and fun. Consequently, children may become more active and healthier, which is probably the most important and desirable outcome.
Data availability statement
Data are available upon reasonable request. Data used for analyses in the paper will be available upon reasonable request made to the lead author.
Ethics statements
Patient consent for publication
Ethics approval
The protocols of the study were reviewed and approved by The Joint Chinese University of Hong Kong—New Territories East Cluster Clinical Research Ethics Committee (Ref.: 2017.515). Parents of all participants provided written informed consent to take part in the study.
References
Footnotes
Twitter @ballmokey88
Contributors Conceptualisation: ASH and JYYN; methodology: ASH, CHSC and JYYN; software: JC and GJ; formal analysis: JYYN; investigation: ASH and JYYN; resources: ASH and JC; data curation: GJ and JYYN; writing—original draft preparation: ASH, JYYN and YY; writing—review and editing: ASH, JC, CHSC, GJ, YY and JYYN; supervision: ASH; project administration: CHSC and JN; funding acquisition: ASH; guarantor: ASH.
Funding This work was supported by The Hong Kong Jockey Club Charities Trust.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.