Article Text
PDF
Abstract
Objective Missed hospital appointments are common among outpatients and have significant clinical and economic consequences. The purpose of this study is to develop a predictive model of missed hospital appointments and to evaluate different overbooking strategies.
Study design Retrospective cross-sectional analysis.
Setting Outpatient clinics of the Royal Hospital in Muscat, Oman.
Participants All outpatient clinic appointments scheduled between January 2014 and February 2021 (n=947 364).
Primary and secondary outcome measures Predictive models were created using logistic regression for the entire cohort and individual practices to predict missed hospital appointments. The performance of the models was evaluated using a holdout set. Simulations were performed to compare the effectiveness of predictive model-based overbooking and organisational overbooking in optimising appointment utilisation.
Results Of the 947 364 outpatient appointments booked, 201 877 (21.3%) were missed. The proportion of missed appointments varied by clinic, ranging from 13.8% in oncology to 28.3% in urology. The area under the receiver operating characteristic curve (AUC) for the overall predictive model was 0.771 (95% CI: 0.768 to 0.775), while the AUC for the clinic-specific predictive model was 0.845 (95% CI: 0.836 to 0.855) for oncology and 0.738 (95% CI: 0.732 to 0.744) for paediatrics. The overbooking strategy based on the predictive model outperformed systematic overbooking, with shortages of available appointments at 10.4% in oncology and 25.0% in gastroenterology.
Conclusions Predictive models can effectively estimate the probability of missing a hospital appointment with high accuracy. Using these models to guide overbooking strategies can enable better appointment scheduling without burdening clinics and reduce the impact of missed appointments.
- Artificial Intelligence
- Electronic Health Records
- HEALTH SERVICES ADMINISTRATION & MANAGEMENT
Data availability statement
Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. The data that support the findings of this study are available from Ministry of Health, Sultanate of Oman, but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This study used a large hospital data set, providing robust data for the development of a model to predict missed out-patient hospital appointments.
The methodology integrated a diverse set of variables to improve prediction accuracy.
The results were based on data from a single hospital, which may limit the generalisability of the findings.
The overbooking strategy evaluated in this study reflects real-world scenarios but lacks experimental validation.
Background
One of the global challenges in any healthcare system is hospital appointment non-attendance. The rate of missed appointments varies around the world, ranging between 14.9% (Europe) and 27.1% (North America)1 2 and across healthcare settings. Missed hospital appointments affect the ability of the healthcare facility to provide a good service, leading to patient dissatisfaction; increased waiting times and therefore increased morbidity and mortality.3 In the UK, £216 million is the estimated annual cost as a result of one million missed GP appointments every month.4 With rising costs and increasing demands of healthcare systems, there is a need to use available resources to provide quality care to all patients.5 6
Clinical prediction models (CPMs) can be used to predict people at risk of developing certain diseases, predicting disease prognosis and adverse outcomes.7 They have shown a positive impact in reducing costs, assisting in better decision making for patient health, allocation of resources and effective utilisation of medical services.8 Prediction models have been used widely to identify patients with a higher risk of missing their hospital appointments. A systematic review including 50 articles showed an increase in the use of such models in the last 10 years by 82% across a range of healthcare settings.9
Prediction models are used in UK hospitals to guide appointment strategies, and it has been reported that the NHS could save millions using such models.10 11 Several prediction models for missed appointments have been developed with area under the receiver operating characteristic curve (AUC) ranging from 0.60 to 0.86.12 13 These studies use data from a single hospital clinic and were conducted in developed countries.14 15
Missed hospital appointments are also a major concern for the Royal Hospital, Sultanate of Oman, which has an extremely high percentage of missed appointments (22.3% overall and up to 30.3% in Urology clinic). Hence, there is a need to implement interventions to reduce the impact of the problem.16 To the best of our knowledge, no study has developed a prediction model for missed hospital appointments in Oman, but there is an opportunity to do so as electronic health record data are available. In this study, we aimed to: (1) develop and validate prediction models for missed hospital appointments using the routinely collected data within the patient’s electronic medical records (EMR); and illustrate, through a simulation, the use of the developed prediction models in managing overbooking and compare to the systematic overbooking approach being used within the hospital currently.
Methods
Data
Appointment data were extracted from the hospital health information management (ALSHIFA) system, a patient EMR system.17 All scheduled outpatients appointments were extracted between January 2014 and February 2021 from The Royal Hospital, the largest tertiary referral hospital in the capital city of Muscat, Sultanate of Oman. The data did not include cancelled appointments or rescheduled appointments and walk-in appointments made within the emergency department. From the complete data set, we split the data by clinics as follows: one overall data set including all clinics except the Paediatric and Obstetrics clinics due to distinct populations; one data set for the Paediatric clinic; one data set for the Obstetrics and Gynaecology clinic and a data set for each of the remaining five clinics in the overall data set (Surgery, Urology, Oncology, Gastroenterology and Diabetic and Endocrine clinic). We applied the data cleaning process as previously described by Alawadhi et al.16 18
Statistical analysis
Risk prediction model
Logistic regression models were developed to predict the risk of missed hospital appointments in each data set separately. For each clinic-specific data set, patients were randomly divided into a development and validation cohort (80% and 20%, respectively). The development and validation cohorts for Diabetic and Endocrine, Surgery, Urology, Oncology and Gastroenterology clinics were combined to generate the development and validation cohorts for the overall model, respectively. This was to ensure that all models were developed and validated on the same data, such that the development data from each clinic were also used as development data for the individual clinics. Development data were used to fit the model, and each developed model was validated in its associated validation data.
Based on our previous work, models were adjusted for the most influential factors for missed appointments, including information on gender, appointment day and month, marital status, governorate (place of residence), appointment waiting time, nationality and service cost (patient contribution to medical service based on age, nationality and monthly income). For example, in our previous work, the adjusted OR for missed appointment for Male patient was 1.08 (95% CI: 1.06 to 1.10), for appointment day Thursday (adjusted OR 0.84 (95% CI: 0.83 to 0.86), for appointment month June was 1.24 (95% CI: 1.20 to 1.29), and for waiting time more than 120 days, the adjusted OR was 1.87 (95% CI: 1.84 to 1.91). Since this study builds on our previous findings, our primary focus here is on developing and internally validating each prediction model and then comparing their use for overbooking with systematic overbooking.16 Appointments were categorised as attended if the patient’s visit was created and logged in the system and missing otherwise. All variables were considered linear except age, where fractional polynomials were used.19
Performance of the models was evaluated by computing the AUC, mean squared error, percentage of correct prediction (PCP), calibration slope and calibration intercept (calibration-in-the-large).20–24 Calibration curves were also produced.
Simulating different overbooking approaches for appointment scheduling
After the development and validation of the models, a simulation study was performed to evaluate a range of overbooking approaches that could be used in clinical practice and the possible added value of using prediction models. This simulation study used the following steps. First, the average number of daily appointments was calculated for each clinic-specific data set (online supplemental table 2) in order to define the number of available daily appointments. Then, a random sample of data for each clinic was extracted based on this average, and a systematic overbooking simulation was performed, overbooking by 5%, 10%, 15%, 20% and 25%, 30%, 35% and 40%. For example, if the average number of daily appointments were 100 and the overbooking approach was 5%, 100 plus 5 patients would be randomly sampled and the true rate of attendance examined. The systematic overbooking approach was compared with an overbooking approach that used the prediction model where patient-specific probabilities were estimated, and the number of missed appointments predicted in each sample was used to determine the overbooking percentage, sampling an additional number of patients from the clinic-specific data set before examining the true attendance rate.
Supplemental material
The simulation was performed 1000 times for each clinic-specific data set. Within each iteration, the difference between the number of available and the number of patients who attended, after applying overbooking, was calculated. A positive value indicated that the number of available appointments exceeded attended appointments (clinic underutilised); a negative value indicated that the number of attended appointments exceeded the number of available appointments (clinic overburdened) and zero indicated that the attended appointments were equal to available appointments. The difference between attended appointments and available appointments was converted to a percentage of appointments available, allowing comparison of approaches across clinics. For each of the 1000 iterations, we calculated the mean, median and the 2.5th and 97.5th percentiles of the difference between the number of attended appointments and the number of available appointments.
Results
Baseline characteristics of the final data set used in the study
There were 947 364 appointments in the final data set, of which 201 877 (21.3%) were missed. The data set included 576 127 (60.8%) female patients and the mean age was 31 years old (table 1). The rate of missed appointments was high for patients with waiting times less than 30 days and more than 120 days (17.2%, 26.6%, respectively). Patients with social affair coverage missed 16.8% of their hospital appointments (4229), whereas patients who had to pay their visit and registration fees missed 21.3% of their hospital appointments (175 026). The rate of missed hospital appointments varied across clinics, ranging from 13.8% in Oncology to 28.3% in Urology. Online supplemental table 3 shows more details about the characteristics of patients within specific clinics.
Characteristics of the complete data set and stratified by attended and missed appointments
Prediction model results
The performance of the overall model and models by clinics varied. The AUC of the overall model was 0.771 (95% CI: 0.768 to 0.775). The Oncology and Obstetrics and Gynaecology clinic models had the highest AUCs of 0.845 (95% CI: 0.836 to 0.855) and 0.805 (95% CI: 0.799 to 0.812), respectively, where the performance for Paediatrics was slightly lower (AUC 0.738 (95% CI: 0.732 to 0.744)). The number of appointments in the development and validation data sets for the overall model and by clinic is displayed in online supplemental table 1.
The calibration curves for all models can be found in figure 1. The calibration slope and calibration intercept were variable between models for individual clinics. The Surgery clinic calibration slope and intercept were 1.038 (95% CI: 1.001 to 1.076) and 0.006 (95% CI: −0.032 to –0.045), respectively, and the Gastroenterology clinic model had a slope of 0.987 (95% CI: 0.932 to 1.043) and intercept of 0.001 (95% CI: −0.060 to –0.061). The overall model had a calibration slope of 0.994 (95% CI: 0.979 to 1.009) with a calibration intercept of −0.003 (95% CI: −0.018 to –0.012). See table 2 for more details.
Predictive performance for each model when applied to the validation data
Calibration curves of the overall model and by clinic. Red line indicates a reference line where predicted and observed probabilities are equal (perfect calibration). Each point indicates the predicted and observed probability of missed hospital appointments in each of the 10 strata. Points below the reference line indicate over-prediction and above the line indicates under-prediction.
When validating the overall model in each clinic separately, the model overestimated (Surgery, Urology, Oncology, Gastroenterology, clinics) and underestimated (Diabetic and Endocrine clinic) the actual rate of missed hospital appointments compared with the individual clinic models. For example, the actual rate of missed appointments in the Urology clinic validation data set was 27.9% and the mean predicted rate of missed appointments using the overall model was 32.6%. In contrast, the actual rate of missed appointments for the Diabetic and Endocrine clinic was 25.4%, while the mean predicted rate of missed appointments was 16.2% (table 3).
Actual and predicted probability of missed hospital appointment by the overall model stratified by clinic*
Overbooking simulation
The simulation results (table 4) show that applying systematic overbooking in the Urology clinic (with a high rate of missed appointments) resulted in considerable underuse of available appointments (eg, average underuse across the 1000 iterations of 13.3% with a systematic overbooking percentage of 20%). However, the Oncology clinic (with the lowest rate of missed appointments) underuse was limited to only the 5% and 10% systematic overbooking approaches. The 20% overbooking strategy resulted in a mean percentage of available appointments after overbooking of 0% (95 percentile: −6.9 to 8.8) in the Obstetrics clinic. In comparison, the prediction modelling strategy for the Obstetrics clinic resulted in 2.9% (95 percentile: −3.9 to 10.8) of appointments still available after overbooking. Online supplemental figure 1 shows the visualisation of the simulation results.
Supplemental material
The differences between attended appointments and daily available appointments after applying each overbooking approach expressed as a percentage of average daily available appointments stratified by clinics, based on 1000 iterations
In addition, over the 1000 iterations, the prediction modelling approach resulted in fewer iterations where the differences between the attended appointments and the daily available appointments were positive (ie, clinic underutilised) or zero (ie, attended appointments were equal to available appointments) and less negative (ie, clinic overburden) in most of the 1000 iterations when using the prediction model approach compared with the systematic overbooking approaches across all clinics. For example, out of the 1000 iterations, the 30% overbooking in the Urology clinic showed that the clinic would be underutilised in 732 iterations, the number of attended appointments would be equal to the daily available appointments in 98 iterations and that the overbooking would cause clinic overburden in 170 iterations if applied. However, applying the prediction model showed that running 836 iterations out of the 1000 iterations would show a positive number, with 71 iterations where the daily available appointments were equal to attended appointments and 93 iterations where the clinic would be overburdened with extra patients if the prediction model was used to overbook. See online supplemental tables 4 and 5 for more details.
Discussion
This study developed and validated CPMs for missed hospital appointments in seven outpatient clinics at The Royal Hospital and one overall prediction model including all outpatient clinics (Obstetrics and Gynaecology and Paediatric clinics excluded from the overall prediction model). We found that the developed risk prediction models had good overall discrimination and calibration, and the individual clinic models had increased predictive performance compared with the general model. We also demonstrate the potential use of the developed model to aid in planning for appointment booking. We found that an overbooking strategy based on the clinic-specific risk prediction models resulted, on average, in less clinic overburden than strategies based on fixed overbooking rates (as currently used in the hospital). However, when we take into account the CI and number of iterations that experienced clinical overburden, some systematic overbooking techniques performed ‘better’ on average than the overbooking approach based on prediction model. This is a difficult decision to choose which approach to implement, and that further work undertaking economic evaluation and benefit analysis would be useful.
The development of prediction models to predict missed hospital appointments has been widely reported in the literature.9 Such models have been developed with differences in terms of the predictors included within those models, the size of the data set used, extent of internal validation (ie, splitting the data set into development and validation cohort), the performance measures used to evaluate the models and the algorithms used to predict missed appointments.25 Our study builds on existing literature as we used a large sample size driven with detailed patient data and included patients from multiple clinics. Other studies have used simulated data sets, while other studies used small data sets when compared with the size of our data set.26–28 It has been reported that small sample size would affect the prediction model performance and larger sample size would enhance the model performance.29 30 Predictors of missed hospital appointments used within our models were selected based on their availability in the hospital system, as with other studies.25 However, some published studies did not include age as a predictor of missed hospital appointments in their models.31 Meanwhile, some studies used age as continuous or categorical variables.32 33 Our model applied fractional polynomial transformation for the age variable, which has not been found in any published paper regarding predicting missed hospital appointments.34 The use of such methods, especially with the age variable, has shown an improvement in the model performance, as stated in some studies.35 36
Most studies that develop prediction models for missed hospital appointments were based on data from single clinics.37–40 Our paper compared the performance of an overall model applied to all clinics (except Paediatric clinic and Obstetrics and Gynaecology clinic) versus models for specific clinics. As found, the performance of the individual models was better than the overall model. This could possibly be explained by less heterogeneity in the patients when considering each clinic separately.41 Our models’ performance was comparable with other studies using logistic regression to build their prediction model (AUC of 0.771 in our study compared with AUC of 0.757 and AUC of 0.768 in other studies).42 43 The performance of prediction models for individual clinics varied, showing high AUC and high PCP. According to studies, a high AUC value indicates better results.44 Similarly, higher PCP by the model indicates better model performance.45 The variances within the models might be related to the fact that different data sets were used to build those prediction models for individual clinics. Therefore, the individual clinic’s data set is unique in terms of patients’ characteristics (demographic and clinical characteristics), which caused the models to perform differently. Studies indicate that different data sets will affect model performance.46 Additionally, in most of the published studies, few performance metrics were used to evaluate their model, commonly area under the curve, mean square error and accuracy.47 48 However, models in our study were evaluated using multiple performance metrics such as calibration-in-the-large, calibration intercept, PCP and Brier Score. Using different performance metrics to evaluate the models would give more insight about the results and would provide more informative details.49
There have been many published studies evaluating the overbooking approach based on prediction models.50–55 The overbooking approach based on prediction models was often more effective than the systematic overbooking approach in providing additional room for extra appointments to be scheduled without adding more pressure to the healthcare facilities.56 The same results have been observed in our study where the overbooking approach based on prediction models was better than the systematic overbooking approach. Our paper compared the two different approaches using the same data set, making our approach unique when compared with other studies. The simulation process used in our study shows that an overbooking strategy which takes into account the probability of missed hospital appointments for individual patient based on his/her demographic data and previous appointment data would be better than the standard systematic overbooking.57 58 To evaluate the possible best approach to missed appointments, we compared a simple algorithm to predictive models. Each appointment was evaluated individually and patient attendance was predicted based on historical data. A dynamic ‘look-back window’ was implemented, where each appointment was evaluated and overbooking was determined accordingly. This approach allowed for data-driven scheduling adjustments to optimise clinic capacity while minimising the impact of no-shows. Our study is considered to be the first to predict missed hospital appointments and to compare between systematic overbooking and overbooking based on a prediction model in the Sultanate of Oman.
Strengths
First, we used a large data set to build our models, which was extracted from the hospital system including real cases. Our data set was big when compared with other excited models in other studies,59 60 which improved the accuracy of our models. Second, our models looked at the heterogeneity of patients within different outpatients’ clinics. A specific model was developed for each clinic, taking into consideration that patients within each clinic would be different in their illness and their medical requirements. As a result, the effect of missed hospital appointment predictors would be different in each clinic. For example, waiting time or distance to travel might be a strong predictor for missed appointments in one clinic and might not be an effective predictor in another clinic. Finally, our model included a variety of variables/predictors. Those predictors were stated to be the strong determinants of hospital appointment status. When compared with other models, it was obvious that the number of variables/predictors used in our model was higher than the number of variables/predictors included in models developed by other published studies.61–63 This helped to develop a more sensitive model that would test/evaluate/detect the patients with higher risk of missed appointments accurately.
Limitations
The data set was extracted from one single tertiary hospital. However, there are other similar hospitals in the capital city of Muscat, which provide tertiary level healthcare services. Also, we did not carry external validity of our prediction models by testing these models in different hospitals from other countries. The findings of this study are based on data collected from a tertiary hospital outpatients clinics providing specialised healthcare. Further studies are necessary to determine whether the results are generalised to other regions or countries. However, this work has highlighted the importance of developing clinic-specific risk prediction models and the better performance of risk prediction approaches to simple algorithms. Finally, we split the data into training and testing data sets, but other methods such as cross-validation can be used.48 64 Although other techniques can be preferred as they do not discard any data for training, here we had a huge data set and this reduction in sample size was therefore not likely to impact our findings.
Conclusion
We used data available within the hospital health information management system to develop a prediction model for missed hospital appointments in multiple clinics. The performance of our models was comparable to other studies with good performance. Our study showed that clinic-specific prediction models outperformed the use of the overall model to predict missed appointments for all clinics. The simulation showed that the proposed overbooking approach based on risk prediction models is more effective than the current systematic overbooking approach used within the hospital.
Data availability statement
Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. The data that support the findings of this study are available from Ministry of Health, Sultanate of Oman, but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants. The study was approved by the Study and Research Centre, Ministry of Health, Sultanate of Oman in 2 May 2019 (proposal ID: MoH/CSR/19/10045). Data were anonymised prior to being accessed by the study authors. Data were anonymised prior to being accessed by the study authors.
Acknowledgments
I am deeply grateful to all those who contributed to the development and completion of this work. I extend my sincere thanks to my supervisors, colleagues, and collaborators for their invaluable insights, critical feedback, and unwavering support throughout the research and writing process. I am equally indebted to my family for their steadfast encouragement, patience, and understanding. The successful realization of this work was made possible through the collective dedication, expertise, and generosity of all involved.
References
Footnotes
Contributors AA drafted the ethics application, analysed, interpreted the electronic health record data and drafted the manuscript. DJ oversaw the development of the models, model testing and models results interpretations. VP oversaw the statistical analyses and reviewed the manuscript. TvS reviewed the ethics application, supervised AA and reviewed the manuscript. All authors read and approved the final manuscript. AA is the guarantor of this work and takes full responsibility for the accuracy of the data and the integrity of the research.
Funding This study was funded by the Ministry of Higher Education, Scientific Research and Innovation, Sultanate of Oman. Grant/Award Number: PGE24720.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.