Article Text
PDF
Abstract
Objectives To determine the accuracy of the caries risk assessment system and performance of clinicians in their attempts to predict caries for children during routine practice.
Design Longitudinal study.
Setting and participants Data on caries risk assessment conducted by clinicians during routine practice while providing care for children in the South Australian School Dental Service (SA SDS) were collected from electronic patient records. Baseline data on caries experience, clinicians’ ratings of caries risk status and child demographics were obtained for all SA SDS patients aged 5–15 years examined during 2002–2005.
Outcome measure Children’s caries incidence rate, calculated using examination data after a follow-up period of 6–48 months from baseline, was used as the gold standard to compute the sensitivity (Se) and specificity (Sp) of clinicians’ baseline ratings of caries risk. Multivariate binomial regression models were used to evaluate effects of children's baseline characteristics on Se and Sp.
Results A total of 133 clinicians rated caries risk status of 71 430 children during 2002–2005. The observed Se and Sp were 0.48 and 0.86, respectively (Se+Sp=1.34). Caries experience at baseline was the strongest factor influencing accuracy in multivariable regression model. Among children with no caries experience at baseline, overall accuracy (Se+Sp) was only 1.05, whereas it was 1.28 among children with at least one tooth surfaces with caries experience at baseline.
Conclusions Clinicians’ accuracy in predicting caries risk during routine practice was similar to levels reported in research settings that simulated patient care. Accuracy was acceptable in children who had prior caries experience at the baseline examination, while it was poor among children with no caries experience.
- Epidemiology
- caries risk asseessment
- sensitivity
- specificity
- school dental service
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/
Statistics from Altmetric.com
Strengths and limitations of study
-
This is one among few studies measured accuracy in caries risk assessment in a real life clinical situation.
-
The study used incidence density to adjust the differences in number of teeth presence and the difference in time between baselines and follow up examination among children enrolled in the study.
-
The concern about this study was children were treated according to their caries risk assessment (CRA) levels. In a hypothetically perfect world, this would delete the potential of predicting the dependent variable because caries would have been prevented based on the CRA. This is unavoidable in any real life clinical situation. However, this issue had been checked against the use of fissure sealant and the results showed that of the prevention provided to high risk children might be effect very little on the accuracy of CRA. Finding of this study, therefore, provide evidence of practice of CRA in the South Australian School Dental Service.
Introduction
Despite effective population oral health preventive programmes in many western countries, dental caries among children remains a major public health issue.1 Certain groups of children develop high levels of the disease that compromise their quality of life and place a substantial burden on the healthcare system.2–4 High-risk strategy is another potential approach to address the skewed distribution of dental caries in the population. It is prudent that, in order for the high-risk strategy to work, risk status of children would need to be identified as correctly as possible and appropriate preventive care would then be applied to them.5
Several studies have reported the clinicians’ ability to accurately identify risk for chronic dental diseases such as caries and periodontal disease.6 ,7 These studies reported a reasonable level of accuracy, which was measured by combined sensitivity (Se) and specificity (Sp). They also reported a wide variation in clinician accuracy. These studies were conducted under special circumstances where examiners were dentists specially trained in using risk assessment criteria. Those circumstances may provide a ‘proof of principle’ for the validity of caries risk assessment (CRA). However, there is little information available about accuracy in routine practice by clinicians who are not specially trained in CRA. This information is important, as the accuracy of CRA in routine practice will have implications for oral health outcomes and cost of dental care.
Studies have shown that clinicians’ subjective estimate of a child's risk of developing caries was the single best predictor of decayed, missing or filled tooth surfaces (DMFS/dmfs) increment in a multivariate model adjusting for other factors.8 This finding implied that caries risk could be reasonably predicted with information routinely available to clinicians at the time of examination, without the need for expensive or time-consuming biological methods that have been promoted for CRA. This conclusion has been supported by studies from Finland where dentists achieved high Sp (Sp=0.90), although with low Se (Se=0.44) in predicting caries risk using their subjective judgement alone.6 In that study, the combined Se and Sp was 1.34 which was reasonable. Importantly, some individual dentists predicted caries with a high combined Se and Sp that approached a score of 1.60.6
Other studies have investigated factors that might contribute to clinicians’ judgements about caries risk by studying child-related factors associated with clinicians’ assessment of caries risk. A study of South Australian children reported that clinicians’ assessment of caries risk was strongly associated with the caries experience of a child's teeth present at the time of assessment. For example, among 6-year-olds, mean dmfs of high-risk children was almost 50 times higher than that of low-risk children (9.91 and 0.20, respectively). Among 12-year-olds, mean DMFS of the high-risk children was five times greater than that of the low-risk children.9 However, that cross-sectional study did not investigate the accuracy of clinicians’ risk assessment, as judged against children's subsequent rate of caries development.
South Australia currently has a highly developed public dental programme for the provision of dental care for school children. The service is provided mainly by dental therapists. Every child in South Australia is eligible for care in the School Dental Service (SDS). Children are invited to enrol in the South Australian SDS (SA SDS) when they start school at the age of 5 years. Enrolment can occur at any time throughout their schooling. Children are able to access any clinic in the SA SDS system. Coverage of the SDS system was over 65% of the state's primary school child population10 between 2002 and 2005, the period used for the current study. Dental care was fully subsidised for children in primary school (aged 5–12 years, approximately).
In the early 1990s, the SA SDS adopted the risk assessment strategy as the approach to individual patient management in the SDS clinic and implemented it as a personalised dental care programme.11 At each dental examination, the children would be classified as having either high, medium or low risk of developing caries. Clinical guidelines for classification of risk status were developed within each South Australian Dental Services health region. The broad guideline basically asked clinician to consider patient as high, medium or low risk if patient satisfied one or more factors on the list such as having past or current caries experience, active decalcified lesion and severe hypoplasia. Clinicians were also instructed to look for dietary habits, oral hygiene habits, fluoride exposure and social history factors while making decision on child risk level. The decision regarding risk level was left to be made based on the subjective estimate of the dentist or dental therapist who assessed and provided care for the child. Once the risk status of a child had been assigned, appropriate services (treatments or preventions) and recall interval would be determined for the individual. Preventive services such as fissure sealants and professionally applied fluoride are more routinely provided to high-risk children as compared with low-risk children. Oral health education was provided to all children. Recall interval could range from 10 to 15 months for high-risk children and 18 to 24+ months for low-risk children. However, the accuracy of the CRA system in general and the performance of individual clinicians in their attempts to predict caries have not been examined. In addition, knowing an individual clinician's accuracy would help to further explore clinician factors that might influence the accuracy of CRA. Such an understanding would help to assess the effectiveness of the risk-based prevention strategy at a programme level and help to deliver better dental care to children in South Australia. Therefore, this study aimed to quantify the accuracy of CRA in routine practice and initially explore the factors that are associated with the accuracy of CRA in the SA SDS.
Methods
The sampling frame for this study was children aged 5–15 years examined in the SA SDS between 2002 and 2005. In SA, since 2001, patient data have been captured in electronic patient records (TITANIUM) as part of routine clinic practice. Clinician had been trained in a small group. Deidentified records were extracted with only identification number by SA SDS and transferred to the Australian Research Centre for Population Oral Health (ARCPOH) at the University of Adelaide. Informed parental consent was obtained for the clinical data extraction. Analysis was conducted with SAS V.9.1.
Data collection
The baseline and follow-up examination
The first task during data management was to select from the dataset each child’s first examination in the 4-year period (‘baseline’ examination). Dates from that baseline examination were used to select a subsequent follow-up examination, if one existed. The first available re-examination of a child that was made 6 or more months after his/her baseline examination was chosen as the follow-up examination for that child. Data from the baseline and the selected follow-up examination were used for the analysis. Children with only one examination during the study period were excluded from the analysis.
Data items
Information recorded during each examination and exported from TITANIUM included: caries experience of each tooth surface; child’s risk status at baseline examination as classified by the examining clinician and sociodemographic parameters such as child sex, residency location, country of birth and indigenous status. Clinician code was also provided in the dataset.
Individual tooth surfaces were classified as decayed, filled or missing because of caries. An additional code designated surfaces that contained fissure sealants and that were otherwise sound and not restored. Five surfaces were coded for all teeth including molars, incisors and canines. For all dentitions, additional guidelines were used to distinguish between teeth missing due to caries and teeth that might have been exfoliated or were extracted for orthodontic reasons.12 The SA SDS clinical staff (where the vast majority are dental therapists) were trained in assessment and recording of dental caries following the guidelines developed by the ARCPOH researchers based on the WHO guidelines13 and the National Institute of Dental Research14 protocol. However, there were no additional procedures for calibrating examiners.
Computation of dmfs and DMFS indices
Tooth surface-level data from baseline examinations were used to compute dmfs/DMFS. Deciduous dmfs was calculated for children aged 5–10 years and permanent DMFS was calculated for children aged 6–15 years. The dmfs and DMFS scores were calculated as sum of DMFS due to caries of the deciduous or permanent dentition. For each child, total number of dmfs+DMFS was calculated.
Computation of caries incidence density (ID)
Net caries increment (NCI) was computed using a De Paola grid.15 The De Paola grid documented the status of all surfaces at the two examinations, and mapped the changes in status that were caries increments. It also identified reversals in caries status or false decrements. Such false negatives were used to estimate the error associated with apparent positive increments. This method assumed that the number of errors due to false increments was equivalent to the number of errors due to false decrements and the resulting NCI represented the corrected estimate of true caries activity. In order to calculate surface-years at risk, the De Paola grid was also used to measure tooth surface time at risk for each individual.16
Caries ID for each individual at the tooth-surface level was computed according to the above equation. In this study, children were aged between 5 and 15 years. Most children aged between 6 and 11 have a mixed dentition. A large proportion of clinician's risk assessment and predictions of caries development was for children with mixed dentitions in this study. Therefore, ID was calculated as combined ID for deciduous and permanent dentitions. ID was calculated following the above formula, total number of new carious lesions including deciduous and permanent dentition divided by total number of surface-year at risk in deciduous and permanent dentition. For children who were 11 years or older at baseline, increment in the deciduous teeth was considered as zero.
Calculation of Se and Sp
In order to achieve the aim of study in measuring the accuracy of CRA in routine practice and to enable comparison to that reported in other studies, Se and Sp were used as the measures of CRA accuracy.
ID (observed rate of caries development) was dichotomised, classifying children as having a high caries rate if they developed at least 1.2 new carious events per 100 surface-years at risk. Children with a lower rate of caries were classified as having a low caries rate. The dichotomised rate was then used as the gold standard for calculating Se and Sp. The cut-off of 1.2 new carious events per 100 surface-years at risk was selected because the resulting proportion of children with a high caries rate was equivalent to or concordant with the proportion of children judged by clinicians to be at high risk at baseline.
Se and Sp were calculated using contingency tables that cross-classified children according to clinician’s baseline risk assessment and observed rate of caries development (ID). Se and Sp were calculated for each clinician who had examined at least 20 children with baseline and follow-up data during study period. In order to calculate Se and Sp, low and medium categories of risk assigned by clinicians at baseline were aggregated into one group.
Variables used in the study
Risk status at the baseline examination was used as an independent variable. Sociodemographic characteristics of children recorded routinely during the examination were used as additional explanatory variables. Sex, age in years, indigenous status and country of birth (Australia or elsewhere) were collected for every child. Fluoridation status of the area of residence was assigned based on level of fluoride in public water supplies. Children were also classified into two groups with/without caries based on their caries experience (dmfs+DMFS) at baseline examination.
Analysis
Before developing a regression model, a description of this child population was carried out. A number of multivariate regression models were generated using SAS PROC GENMOD. The purpose of these models was to estimate associations between child-level and clinician-level characteristics associated with each measure of accuracy: Se and Sp. Separate models estimated probabilities (either Se or Sp) using the binomial distribution and an identity link. Level of significance, direction and magnitude of the effect of each factor were examined.
In the models for Se, the probability of being predicted as high risk of developing caries at baseline among children who had developed a high caries rate was estimated. The intercept of this model was the estimated Se in the population when all child-level factors were fixed at the selected reference category for that factor. Reference categories were selected to represent either the largest number of children from among two or more categories of a child-level characteristic or a conventional demographic group. The estimates of individual factors indicated direction and magnitude of effect of those factors. If the estimate of a factor has a negative value, this indicated that Se or Sp was lower in the non-reference category than in the reference category.
In the models of Sp, probability of being predicted as low risk at baseline among children who had a low rate of the disease during the follow-up was estimated. Similarly, direction and magnitude of the effect of individual factors were presented as the estimates of the models.
Results
A total of 133 clinicians examined 71 430 children with two or more examinations within the time interval at least 6 months apart. The majority of clinicians practised as dental therapists (83%) while the remainder 17% were dentists. About 88.6% of clinicians were women and 79.4% were Australian-born. The modal age group was 41–50 years (41.3% of clinicians) and only 12.7% were aged 23–30 years. Nearly one-half of the respondents had 21–30 years of experience working in dentistry. More than half of the respondents reported working part-time, while only 2.4% reported they were employed on a casual basis (data not shown). The distribution of children by sex and residential location was similar to distribution of children for the whole state of South Australia. Two-thirds of the children were from metropolitan areas. Just under half of the children had experience of deciduous or permanent caries at baseline (table 1).
Study sample characteristics
Time interval between two examinations ranges from 6 months to 4 years. Around 12% of children had follow-up after less than 12 months and 90% of children of this group were classified as medium or high risk (table 1). Only 2% of children had a follow-up examination later than 4 years. These children were excluded from the analysis.
Children who were classified at baseline as high risk of developing caries had significantly higher mean baseline dmfs and DMFS compared with those who were classified as medium or low risk of developing caries (7.52 vs 1.36 and 0.20 for deciduous dmfs; 1.30 vs 0.57 and 0.29 for permanent DMFS; table 2). High-risk children developed nine and two times higher caries ID for deciduous dentition compared with low-risk and medium-risk children, respectively (table 2). Similarly, for permanent dentition, the mean of ID was 0.99 among high-risk children, which is five times higher than that among low-risk children (ID=0.17). High-risk children developed almost nine times higher caries ID for combined deciduous and permanent dentition compared with low-risk children (table 2).
Caries experience at baseline and caries increment during the follow-up by baseline risk
Se of individual clinicians ranged from 0 to 0.92 with the mean of 0.46 (figure 1). Over half of the clinicians had Se within the range of 0.40–0.60, while only 14% of clinicians achieved high Se of 0.7 or more (figure 1). Individual clinicians had a mean Sp of 0.86 with a minimum of 0.61 and a maximum of 1.00. About 82% of clinicians had Sp within the range of 0.80–1.00 (figure 2). The level of overall accuracy for each clinician was assessed by combining Se and Sp (Se+Sp). The majority of clinicians had a combined Se+Sp ranging from 1.30 to 1.50. There were 5% of clinicians who had a low combined Se+Sp. There were also 5% of clinicians who achieved a combined Se+Sp over 1.50 (figure 3).
Individual clinician's sensitivity.
Individual clinician's specificity.
Individual clinician combined sensitivity+specificity (Se+Sp).
Se and Sp of CRA among SA SDS children was 0.47 and 0.86, respectively (table 3). The combined Se and Sp was 1.33. More than half of the children who developed a high rate of caries within one recall cycle were not classified as being high risk at baseline.
Overall sensitivity and specificity among children without/with caries experience at baseline
Se and Sp were influenced by children's caries experience at baseline. Among children without caries experience at baseline, Se was low (se=0.07) while Sp was almost perfect (sp=0.98). Among children with caries experience at baseline, Se improved significantly compared with that among the no caries experience group (0.57 vs 0.07, respectively), while Sp decreased from 0.98 among group without caries experience to 0.72 among the group with caries experience (table 3).
A number of other child-level factors were found associated with Se and Sp scores of CRA in multivariate models (table 4). The directions of association between Se and Sp with every variable were reversed except for the level of fluoride concentration in the water. Examining children with no caries experience reduced the Se by 0.48 and increased the Sp by 0.26. Children's age was also another factor that influenced the accuracy of CRA. Examining younger children increased the Se by 0.11 while decreasing the Sp by 0.03 (table 4).
Multivariate binomial regression model for sensitivity and specificity by child-level factors
Discussion
In this study of caries, clinicians were moderately accurate in correctly identifying children who later developed a low rate of caries (Sp=0.86). However, they had lower accuracy in correctly identifying children who later developed a high rate of caries (Se=0.47). Among children with caries experience at their baseline examination, there was greater accuracy in correctly predicting a high rate of caries (Se=0.57), although it came at the expense of reduced accuracy in correctly predicting a low rate of caries (Sp=0.71). However, accuracy was scarcely better than chance (Se+Sp=1.05) among children with no caries experience at their baseline examination.
Several aspects of the study setting and the study methods need to be considered when interpreting the results. As the study was based on routine practice, children received dental care, including caries prevention such as fissure sealants, fluoride varnish and dental health education, according to their level of assessed risk. A certain proportion of the predicted disease could have been prevented by the interventions provided. This would result in an underestimation of CRA accuracy (Se and Sp). In this population, high-risk children were significantly more likely to receive fissure sealants compared with the children who had lower risk. Therefore, the more intensive use of fissure sealants in high-risk children was expected to reduce the rate of new caries during the follow-up period. High-risk children who received fissure sealants had a lower rate of caries development than high-risk children who did not receive sealants. Similarly, low-risk children who received sealants had a lower rate of caries development than those low-risk children who did not receive sealants. This was expected to result in lower accuracy among children who received sealants than among children who did not receive new sealants. However, the observed overall accuracy was virtually identical between the two groups (data not shown).
A possible reason for this finding might be that the effect of fissure sealant on caries increment in this child population was small. Also, the difference in the underlying rate of caries increment between the high-risk and the low-risk groups was substantial. The preventive effect of fissure sealants was not enough to offset the difference in caries rate between the two risk groups. It was possible to conclude that the accuracy of CRA in this study population was not significantly biased by the preventive treatment provided to the high-risk children.
Furthermore, when assigning a child’s risk status, clinicians might form their judgements after considering the child’s likely clinical status over a longer period than that observed in this study. This again might contribute to underestimation of accuracy with the methods used here.
A useful risk assessment programme is one with high Se and Sp. However, with the inherent trade-off between Se and Sp, it might be impractical to achieve high Se and Sp simultaneously. The overall CRA accuracy over one recall interval of SA SDS was 1.33. This level of accuracy was similar to the level reported in a Finnish study,6 which was conducted in a similar primary care environment (children aged 5–17 years assessed by dentists and dental hygienists working in a healthcare centre). The current study, therefore, further validated a realistic level in ‘real-life’ accuracy. The components of clinicians’ accuracy were similar in both studies, with Se=0.44 and Sp=0.90 in Finish study and Se=0.47 and Sp=0.86 in this South Australian study.
In the North Carolina study,8 ,17 the highest CRA's accuracy level reached 1.45 (Se=0.60 and Sp=0.85) among children in grades 1 and 5. Clinicians in the current study had lower accuracy, especially lower Se. However, conditions in the current study were different to that of the North Carolina research study, where a more rigid research protocol was applied. Clinicians in the North Carolina study were specifically trained for the purpose of CRA. In the current study, clinicians practiced CRA as a routine part of their daily clinical practice that involved ongoing clinical patient care, including examination visits, preventive and restorative services and recalls. Therefore, it was expected that the CRA process may have lower accuracy level compared with that observed in the North Carolina study.
Clinicians in this study varied considerably in their accuracy of caries risk prediction. Some clinicians predicted caries with a high combined Se and Sp that approached a score over 1.50, while some clinicians had a much lower accuracy. This finding is comparable with results reported in Finland by Alanen.6 The variation among clinician's accuracy of CRA indicates that there are some clinician-related factors that might account for the difference in the level of accuracy.
A study evaluating predictive accuracy of CRA performed during routine clinical practice among adult patients also reported a level of overall accuracy that was similar to the level observed in this study.18 The observed Se score in this current study was slightly lower compared with that in the other study (0.47 vs 0.57, respectively). However, CRA in this South Australian study was performed in a population including children with a mixture of deciduous dentition, mixed dentition phase and permanent dentition in contrast to the above study where adults with permanent dentition were examined.
The difficulty in individual assessment of future caries risk is widely accepted. This was evident in this current study. However, at the group level, risk assessment is much stronger. The fact that children in the low-risk group developed significantly fewer new carious lesions than children in high-risk group (table 2) revealed that risk assessment at the group level was far more accurate.
The stratified analysis (table 3) revealed some level of interaction between clinicians’ ability to predict caries and the true risk of developing caries. Clinicians were more likely to assign high-risk status to children with caries experience at baseline, who in turn, might truly be at higher risk of developing new caries. Hence, the non-stratified analysis was a mixed effect of the aforementioned issues. The stratified analysis isolated the clinicians’ true capacity to predict caries, and in both strata, the accuracy level was lower than the non-stratified estimates of accuracy.
The stratified analysis, confirmed by the multivariate analysis, indicated the strong effect of previous caries experience on accuracy of CRA. This finding was reported in the previous assessment of CRA. It means that CRA in the current form may have little value among children who have no previous caries experience.
There may be some criticism that the data used to measure the outcome variable, dental caries rate, were collected by uncalibrated clinicians. However, those clinicians were similarly trained and used uniform clinical manuals to perform the examinations. In addition, the protocol was developed by experienced oral epidemiologists from the University of Adelaide in collaboration with South Australian Dental Service clinical leaders. This approach was also consistent with a recent statement by Hausen et al19 that “In large enough settings, data obtained from patient records could possibly be used as a replacement for separate surveys.” Also, analyses were based on the presence/absence of cavitated caries lesion (either filled or not), which is reliable.20
The time interval between examinations in purposively designed CRA studies was often set to be uniform. Caries increment is time dependent. Therefore, NCI was often the ‘gold standard’ of choice in those studies. This was not possible in this study, where the time interval between examinations varied considerably among children. The time factor was important in computation of caries increment. Children in the general population may have different time intervals between their dental visits that may affect the amount of disease development during the recalls. The children in this study were of different age groups who would naturally have different numbers of teeth, deciduous and permanent, present in their mouth, and hence being at risk for having caries. One advanced technique that was used in this study was the calculation of ID. The ID calculated in this study can adjust for different time intervals and the number of teeth present in the mouth. The time and the number of tooth surfaces present indicate the level of risk exposure for a child during the study period. Variation in risk exposure level was appropriately handled. For that reason, ID was used as the ‘gold standard’ in this study which helped to overcome that problem.
This study provided evidence that routine CRA performance among children in the SA SDS was relatively accurate at the group level. However, at the individual level, accuracy was lower, especially among those children with no previous caries experience. The study findings are in accordance with the existing studies that we cannot predict caries very well and that there is a large difference between caries-free and caries-active populations. It also suggests that this might be the time for researchers to minimise the search for more information on how to predict caries, as caries risk prediction is only for clinical management, and to pay more attention to research and providing an evidence-based approach to population prevention strategies according to caries-free and caries-active status. An explicit decision about CRA should be made in the future: CRA is a population oral health prevention strategy or CRA is a clinical monitoring strategy.21
To conclude, this study confirms that CRA within SDS is only a method to justify when to place children into different recall interval.
References
Footnotes
-
Contributors All the authors have contributed to the manuscript and have approved the final version.
-
Funding This study was supported by the Adelaide Research Scholarship, the South Australian Dental Service and an Australian Dental Research Foundation grant.
-
Competing interests None.
-
Patient consent Obtained.
-
Ethics approval Ethical approval was received from the University of Adelaide's Human Research Ethics Committee and the South Australian Dental Service.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement Data are stored at the Australian Research Centre for Population Oral Health. Access to the data can be requested.