Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED

a b s t r a c t

Objective: The prediction of emergency department (ED) disposition at triage remains challenging. Machine learning approaches may enhance prediction. We compared the performance of several machine learning approaches for predicting two clinical outcomes (critical care and hospitalization) among ED patients with asthma or COPD exacerbation.

Methods: Using the 2007-2015 National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data, we identified adults with asthma or COPD exacerbation. In the training set (70% random sample), using routinely- available triage data as predictors (e.g., demographics, Arrival mode, vital signs, chief complaint, comorbidities), we derived four machine learning-based models: LASSO regression, Random forest, boosting, and deep neural net- work. In the test set (the remaining 30% of sample), we compared their prediction ability against traditional lo- gistic regression with Emergency Severity Index (ESI, reference model).

Results: Of 3206 eligible ED visits, corresponding to weighted estimates of 13.9 million visits, 4% had critical care outcome and 26% had hospitalization outcome. For the critical care prediction, the best performing approach- boosting - achieved the highest discriminative ability (C-statistics 0.80 vs. 0.68), reclassification improvement (net reclassification improvement [NRI] 53%, P = 0.002), and sensitivity (0.79 vs. 0.53) over the reference model. For the hospitalization prediction, random forest provided the highest discriminative ability (C-statistics 0.83 vs. 0.64) reclassification improvement (NRI 92%, P b 0.001), and sensitivity (0.75 vs. 0.33). Results were gen- erally consistent across the asthma and COPD subgroups.

Conclusions: Based on nationally-representative ED data, machine learning approaches improved the ability to Predict disposition of patients with asthma or COPD exacerbation.

Introduction

Obstructive airway diseases, such as asthma and chronic obstructive pulmonary disease (COPD), are important health problems in the U.S. Acute exacerbation of these diseases accounts for 3.1 million emergency department (ED) visits each year [1]. Prior studies have shown that early intervention on these patients in the ED decreases morbidity and mortality [2-4]. Thus, it is important to use ED Triage systems that accu- rately differentiate and prioritize critically ill from stable patients. How- ever, the currently-available systems (e.g., Emergency Severity Index [ESI]) are known to subject to large inter-operator variability [5] and suboptimal prediction ability [6-8] in the ED, including for patients with obstructive airway diseases [9].

* Corresponding author at: Department of Emergency Medicine, Massachusetts General Hospital, 125 Nashua Street, Suite 920, Boston, MA 02114-1101, United States of America.

E-mail address: [email protected] (T. Goto).

The recent advent of machine learning approaches has shown prom- ise to achieve superior prediction ability in varioUS settings and disease conditions (e.g., sepsis) compared to traditional approaches [10]. These modern machine learning approaches have advantages that they ac- count for non-linear high-order interactions between independent var- iables and yield more stable predictions [11]. For example, a retrospective analysis of the data from two urban EDs reported that the use of a machine learning approach improves triage classification in the general ED population [6]. However, to date, no study has inves- tigated the utility of modern machine learning approaches in a large geographically-diverse sample, let alone ED patients with exacerbation of obstructive airway disease.

In this context, we analyzed nationally-representative ED visit data to develop machine learning-based triage models to predict the clinical course of asthma and COPD exacerbation after ED triage, and to com- pare their prediction performance to the traditional approach using lo- gistic regression with ESI information.

https://doi.org/10.1016/j.ajem.2018.06.062

Methods
1. Study design, setting, and samples

We analyzed combined data from the ED component of the 2007-2015 National Hospital and Ambulatory Medical Care Survey (NHAMCS). NHAMCS is a nationally-Representative sample of visits to noninstitutional general and short-stay hospitals, excluding federal, military, and Veterans Administration hospitals, in the 50 sates and the District of Columbia. The survey is conducted annually by the Cen- ters for Disease Control and Prevention’s (CDC) National Center for Health Statistics [12]. For example, the 2015 sample included 21,061 visit records from 267 EDs, resulting in a weighted national sample of 137 million ED patient visits. A detailed description of NHAMCS proce- dures is available in the technical notes section of NHAMCS ED Survey [12]. The institutional review board of Massachusetts General Hospital waived review of the current analysis.

We identified all ED visits made by adult patients (aged >= 18 years)

with asthma or COPD exacerbation by using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) code for asthma (493.xx) and COPD (491.xx, 492.xx, and 496.xx) in the primary diagnosis field [13, 14]. We excluded patients who were dead on ED ar- rival, left before being seen or against medical advice, or with missing predictor variables. We focused on 2007-2015 based on the availability of vital sign data during these years.

Predictors

The predictors for model development were chosen from routinely- available data at ED triage and using a priori knowledge [6, 7]. Specifi- cally, the predictors included patient age, sex, mode of arrival (walk-in vs. ambulance), vital signs (temperature, pulse rate, systolic and dia- stolic blood pressure, respiratory rate, and oxygen saturation), common chief complaints (e.g., dyspnea, cough, chest pain), asthma or COPD sta- tus, and comorbidities (Elixhauser comorbidity measures [15]).

Outcomes

The outcomes of interest were critical care and hospitalization out- comes. Critical care outcome was compositely defined as either direct admission to an intensive care unit or in-hospital death [6]. The hospi- talization outcome was either admission to an inpatient care site or di- rect transfer to an acute care hospital [6].

Statistical analysis

In the training set (70% random sample), we developed five models to predict the probability of each outcome. First, as the reference model, we fit logistic regression model including only the ESI triage measure- ment recorded in the database [7]. NHAMCS encoded triage as immedi- ate (level 1), emergent (level 2), urgent (level 3), semi-urgent (level 4), and non-urgent (level 5). While the majority of EDs use the ESI, 7% of the NHAMCS EDs used other systems that were systematically recoded to the 5-level system by CDC [12]. Next, using the predictors above, we constructed four machine learning prediction models: 1) logistic regres- sion with Lasso regularization, 2) random forest, 3) gradient boosted decision tree [16], and 4) deep neural network.

Lasso regression is a type of regression analysis for both variable se- lection and regularization. Lasso regression has an ability to shrink coef- ficients of variables to zero, minimizing overfitting and eliminating the need to do feature selection on high dimensional data. Random forest is an ensemble of decision trees created by using bootstrap samples of the training data and random feature selection in tree induction. Gradi- ent boosted decision tree is another ensemble approach to parametric modeling. It is an additive model of decision trees estimated by gradient descent. Deep neural network is a class of machine learning algorithms

consisting of multiple layers of nonlinear processing units to learn the value of the parameters that result in the best prediction of outcome. In the deep network, we constructed 4-layer feedforward model with adaptive moment estimation optimizer [17] using Keras implemented in R [18]. To minimize potential overfitting, we used Lasso penalization, out-of-bag estimation, cross-validation, and dropout as well as Ridge penalization, respectively.

In the test set (30% random sample), we measured the prediction performance of each model by computing 1) C-statistics (i.e., area under the receiver-operating-characteristics [ROC] curve), 2) net reclas- sification improvement (NRI; an index to quantify how well a new model reclassifies subjects as compared to the reference model), and

3) prospective prediction results (i.e., sensitivity, specificity, positive predictive value and negative predictive value). To address the class im- balance in the critical care outcome (i.e., the low proportion of out- come), we chose the threshold of prospective prediction results based on ROC curve (i.e., the value with the shortest distance to the perfect model) [11]. In the sensitivity analysis, we measured the prediction per- formance with stratification by primary diagnosis (asthma vs. COPD). A two-sided P value of b0.05 was considered statistically significant. All analyses were performed with R version 3.4.

Results

During 2007-2015, the database recorded 3896 ED visits made by patients with exacerbation of obstructive airway disease. Of these, we excluded 1 death on arrival, 50 who left before being seen or against medical advice, and 639 with missingness on the chosen predictors. We included the remaining 3206 ED visits (weighted estimates of 13,938,778 visits; 95% CI 12,639,034-15,238,522) in the current analy- sis. The analytic and non-analytic cohorts did not differ in most patient characteristics (Supplemental Table 1). Of 3206 patients in the analytic cohort, the median age was 52 (IQR 36-67) years and 60% were female; 58% had asthma exacerbation and 42% had COPD exacerbation (Supple- mental Table 2).

Prediction of critical care outcome

Overall, the rate of critical care outcome was 4%. The discrimination ability of different models represented by ROC curve is illustrated in Fig. 1A. While the reference model had the lowest discriminative ability (C-statistics 0.68; Table 1), all four Machine learning models had a higher discriminative ability (all C-statistics >= 0.76). Particularly, the gradient boosted decision tree model provided the highest ability (C- statistics 0.80). This model also achieved the highest reclassification im- provement (NRI 53%, P = 0.002) over the reference model, with a higher sensitivity (0.79 vs. 0.53). In the stratified analysis with limited statistical power, the machine learning models also provided a higher prediction ability across the asthma and COPD subgroups (Supplemen- tal Table 3).

Prediction of hospitalization outcome

Overall, the rate of hospitalization outcome was 26%. The discrimi- nation of different models represented by ROC curve is shown in Fig. 1B. The reference model had the lowest discriminative ability (C- statistics 0.64; Table 1); all machine learning models had a higher dis- criminative ability (all C-statistics >= 0.82). Particularly, the random for- est model provided the highest ability (C-statistics 0.83). This model also achieved the highest reclassification improvement (NRI 92%, P b 0.001) over the reference model, with a higher sensitivity (0.75 vs. 0.33). In the stratified analysis, the machine learning models also pro- vided higher prediction ability across the subgroups (Supplemental Table 3).

Fig. 1. Receiver-operating-characteristics (ROC) curves of the reference and machine learning models in the test set A) Critical care outcome B) Hospitalization outcome The corresponding values of the area under the receiver-operating-characteristics curve for each model (i.e., C-statistics) are presented in Tables 1.

Table 1

Prediction ability of the reference and four machine learning models

Overall group	C-statistic	NRI	P-value	Sensitivity	Specificity	PPV	NPV
Critical care outcome Reference model	0.68	Reference	-	0.53	0.81	0.09	0.98
Logistic regression with Lasso regularization	0.79	24%	0.17	0.74	0.69	0.08	0.99
Random forest	0.76	1%	0.95	0.76	0.72	0.09	0.99
Gradient boosted decision tree	0.80	53%	0.002	0.79	0.68	0.08	0.99
Deep neural network	0.79	39%	0.04	0.71	0.78	0.11	0.99
Hospitalization outcome Reference model	0.64	Reference	-	0.33	0.84	0.44	0.77
Logistic regression with Lasso regularization	0.82	81%	b0.001	0.73	0.77	0.53	0.88
Random forest	0.83	92%	b0.001	0.75	0.76	0.53	0.89
Gradient boosted decision tree	0.82	80%	b0.001	0.73	0.76	0.53	0.88
Deep neural network	0.82	86%	b0.001	0.69	0.78	0.54	0.87

Abbreviations: NRI, net reclassification improvement; PPV, positive predictive value; NPV, negative predictive value

Variable importance

To gain insights into the contribution of each predictor to the model, we computed the variable importance in the random forest model for each outcome (Fig. 2). For both outcomes, advanced age, vital signs (e.g., respiratory rate, oxygen saturation), arrival mode (i.e., ambulance), and several comorbidities (e.g., arrhythmia, conges- tive heart failure) were important predictors.

Discussion

In this analysis of nationally-representative data on ED patients with asthma or COPD exacerbations, the use of machine learning approaches (i.e., Lasso regression, random forest, gradient boosted decision tree, and deep neural network) significantly improved the ability to predict two clinical outcomes (critical care, hospitalization) over the traditional approach using ESI information. To the best of our knowledge, this is the first study that has applied modern machine learning approaches to the ED patients with exacerbation of obstructive airway disease.

ED triage often presents the first opportunity to identify critically ill patients and improve the efficiency of ED resource allocation. However, prior studies have shown that the discrimination ability of currently- available triage systems is suboptimal [6-8]. While adding a broader set of predictors (e.g., chronic disease severity, physical examination) might improve the ability, this approach is unlikely to be feasible at the ED triage setting because of limited information and time pressure. An alternative strategy to improve the prediction ability is the use of ad- vanced methods - such as modern machine learning approaches - that better address non-linear, higher-order interactions between predictors [11]. Indeed, recent studies have demonstrated that machine learning models improve predictions on Inhospital mortality in ED patients with sepsis [19], acute cardiac complications in patients with chest

pain [20], and readmission in patients hospitalized for heart failure [21], Similarly, single-center studies of pediatric ED patients with asthma exacerbation (n b 1,000) reported that the application of ma- chine learning approaches to detailed clinical data beyond ED triage in- formation (e.g., chronic asthma factors, allergy history, physical examination) achieved moderate discriminative ability [22-24]. The current study builds on these earlier reports, and extends them by dem- onstrating, for the first time, superior ability of modern machine learn- ing approaches on predicting the disposition of ED patients with asthma or COPD exacerbation.

The observed incremental gains over the traditional approach sug- gest the potential power of machine learning approaches. There are sev- eral potential explanations for the gains. First, the ESI - despite the widespread adoption - relies heavily on provider judgment and is known to have a high variability between operators [5]. Additionally, a major element of ESI is the assessment of anticipatED resource use rather than clinical course. Furthermore, the machine learning ap- proaches can model the complex relationships between predictors which cannot be addressed by traditional modeling approaches [11]. Moreover, we also applied rigorous approaches to minimize potential overfitting of the models (e.g., Lasso and Ridge regularization, cross- validation, dropout). However, despite their apparent superiority over the traditional approach, their prediction ability remains imperfect. This might be attributable to the subjectivity of data measurement (e.g., chief complaint), contributions of clinical factors after ED triage (e.g., timeliness and quality of ED management, response of bronchodi- lator treatment), differences in patient preference, provider’s practice patterns, and institutional resources, or any combination of these factors [25, 26]. Notwithstanding the complexity of Disposition decisions in ED patients, our findings support a cautious optimism that these novel ap- proaches can further improve our Predictive ability in at least the large group of patients with asthma or COPD exacerbation.

Fig. 2. Importance of each predictor in the random forest models A) Critical care outcome B) Hospitalization outcome Abbreviations: BP, blood pressure; Cc, chief complaint; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; DM, diabetes mellitus; HTN, hypertension; PVD, peripheral vascular disease.

Potential limitations

Our study has several potential limitations. First, an exclusion of pa- tients with missingness is a potential source of selection bias. However, the characteristics of analytic and non-analytic cohorts were similar, ar- guing against a significant bias. Second, as the current analysis relies on survey data, there might be some misclassification (e.g., misdiagnosis of asthma and COPD). However, NHAMCS is known to take rigorous qual- ity assurance procedures. Indeed, in their 10% quality control sample of data, coding error rates were b1% [12]. Lastly, NHAMCS data do not mea- sure some clinical variables (e.g., chronic severity of illness, use of con- trol medications, prehospital treatment and response, use of non- invasive positive pressure ventilation). Yet, the aim of the study was not to derive predictive models using a rich set of predictors but to de- velop machine learning models to harness a limited set of clinical infor- mation that are currently available in the typical ED triage setting.

Conclusions

Based on the analysis of nationally-representative data of ED pa- tients with asthma or COPD exacerbation, we found that the use of ma- chine learning improved the ability to predict ED disposition over the traditional approach with ESI information. While external prospective validation is necessary, these observations demonstrate an opportunity to apply advanced prediction approaches to routinely-available triage data - as an assistive technology - to enhance the clinician’s triage deci- sion making, which will, in turn, lead to more accurate and efficient clin- ical practice in the ED.

Conflict of interest

Dr. Camargo has providED asthma- and COPD-related consulting ser- vices to AstraZeneca, GlaxoSmithKline, and Mereo. Dr. Hasegawa has re- ceived grants for asthma-related research from Novartis and Teva. The other authors have no relevant financial relationships to disclose.

Prior abstract publication/presentation

None.

Funding

None.

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi. org/10.1016/j.ajem.2018.06.062.

References

HCUPnet. U.S. Department of Health and Human Services. Agency for healthcare re- search and quality. http://hcupnet.ahrq.gov/, Accessed date: 16 May 2018.
Bhogal SK, McGillivray D, Bourbeau J, Benedetti A, Bartlett S, Ducharme FM. Early ad- ministration of Systemic corticosteroids reduces Hospital admission rates for chil- dren with moderate and Severe asthma exacerbation. Ann Emerg Med 2012;60: 84-91 [e83].
Plant PK, Owen JL, Elliott MW. early use of non-invasive ventilation for acute exac- erbations of chronic obstructive pulmonary disease on general respiratory wards: a multicentre randomised controlled trial. Lancet 2000;355:1931-5.
Rowe BH, Spooner C, Ducharme FM, Bretzlaff JA, Bota GW. Early emergency depart- ment treatment of acute asthma with Systemic corticosteroids. Cochrane Database Syst Rev 2001:CD002178.
Hitchcock M, Gillespie B, Crilly J, Chaboyer W. Triage: an investigation of the process and potential vulnerabilities. J Adv Nurs 2014;70:1532-41.
Levin S, Toerper M, Hamrock E, et al. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index. Ann Emerg Med 2018;71:565-74 (e562).
Dugas AF, Kirsch TD, Toerper M, et al. An electronic emergency triage system to im- prove patient distribution by critical outcomes. J Emerg Med 2016;50:910-8.
Arya R, Wei G, McCoy JV, Crane J, Ohman-Strickland P, Eisenstein RM. Decreasing length of stay in the emergency department with a split emergency severity index 3 patient flow model. Acad Emerg Med 2013;20(11):1171-9.
Parshall MB, Doherty GS. Predictors of emergency department Visit disposition for patients with chronic obstructive pulmonary disease. Heart Lung 2006;35(5): 342-50.
Berlyand Y, Raja AS, Dorner SC, et al. How artificial intelligence could transform emergency department operations. Am J Emerg Med 2018 Jan 4 [Epub ahead of print].
Kuhn M, Johnson K. Applied predictive modeling. , Vol. 26Springer; 2013.
2015 NHAMCS Emergency Department public use data file. National Center for Health Statistics. Centers for disease control and prevention. https://www.cdc.gov/ nchs/ahcd/index.htm; 2015. [Accessed May 16, 2018].
Hasegawa K, Tsugawa Y, Brown DF, Camargo Jr CA. A population-based study of adults who frequently visit the emergency department for acute asthma: California and Florida, 2009-2010. Ann Am Thorac Soc 2014;11(2):158-66.
Tsai CL, Sobrino JA, Camargo Jr CA. National study of emergency department visits for acute exacerbation of chronic obstructive pulmonary disease, 1993-2005. Acad Emerg Med 2008;15(12):1275-83.
Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidi- ties in ICD-9-CM and ICD-10 administrative data. Med Care 2005;43(11):1130-9.
Chen T, Guestrin C. Xgboost: a scalable tree boosting systemProc of the 22nd ACM SIGKDD Intern Conference Knowledge Discoverty Data Mining; 2016. p. 785-94. https://doi.org/10.1145/2939672.2939785.
Kingma DP, Ba J Adam. A method for stochastic optimization. arXiv preprint arXiv:14126980; 2014.
R interface to Keras. https://github.com/rstudio/keras/; 2017, Accessed date: 16 May

2018.

AJEM