American Journal of Emergency Medicine (2012) 30, 1491-1500

Original Contribution

Development and validation of the excess mortality ratio-based Emergency Severity Index^?

Ki Jeong Hong MD ^a,b, Sang Do Shin MD, PhD ^a,b,?, Young Sun Ro MD ^b,c,

Kyoung Jun Song MD ^a,b, Adam J. Singer MD ^d

^aDepartment of Emergency Medicine, Seoul National University College of Medicine, Seoul, Korea ^bLaboratory of Emergency Medical Service, Seoul National University Hospital Biomedical Research Institute, Seoul, South Korea ^cSeoul National University School of Public Health, Seoul, South Korea

^dDepartment of Emergency Medicine, Stony Brook University and Medical Center, NY, USA

Received 17 July 2011; revised 1 November 2011; accepted 9 December 2011

Abstract

Purpose: The purpose of this study is to develop and validate the excess mortality ratio-based Emergency Severity Index (EMR-ESI) that feasibly and objectively assesses the severity of emergency department (ED) patients based on their chief complaints.

Methods: We used data from the National Emergency Department Information System of Korea from January 2006 to December 2009. We obtained information on mortality and the corresponding chief complaints exhibited by patients presenting to all EDs. The EMR-ESI was computed from the ratio of sex-age standardized hospital mortality for each chief complaint and the sex-age standardized mortality of the entire population of Korea. We tested the discriminatory power of the EMR-ESI on the prediction of hospital outcomes using the Area Under the Receiver Operating Characteristic Curve from a multivariate logistic regression model. This model was adjusted for clinical parameters, and the goodness of fit was estimated using the Hosmer-Lemeshow logistic model.

Results: Included in the study were 4 713 462 patients who presented 7557 chief complaint codes from 2006 to 2008. The EMR-ESI had a range of 0 to 6389.45 (mean +- SD, 1.11 +- 4.67; median, 0.70). The adjusted odds ratio of the EMR-ESI (unit, 1.0) for hospital mortality was 1.11 (95% confidence interval, 1.11-1.12). The AUCs for predicting hospital mortality, ED mortality, admission mortality, and admission were 0.95, 0.98, 0.90, and 0.74, respectively. There were 3 422 865 patients from 2009 who were included for external validation, and the AUCs for predicting mortality in the hospital, the ED, the inpatient ward, and for predicting admission were 0.95, 0.99, 0.90, and 0.75, respectively.

Conclusion: The EMR-ESI was notably useful in predicting hospital mortality and the admission of emergency patients.

(C) 2012

^? Previous presentation: This study was presented at the 5th Asian Congress on Emergency Medicine in Busan, Korea, in May 2009 and at the 5th Mediterranean Emergency Medicine Congress in Valencia, Spain, in September 2009.

* Corresponding author. Tel.: +82 2 2072 3257; fax: +82 2 741 7855.

E-mail addresses: [email protected] (S.D. Shin), [email protected] (K.J. Hong), [email protected] (Y.S. Ro), [email protected] (K.J. Song), [email protected] (A.J. Singer).

0735-6757/$ - see front matter (C) 2012 doi:10.1016/j.ajem.2011.12.011

Introduction

Background

Accurately assessing the severity of emergency depart- ment (ED) patients and predicting adverse outcomes are important for initial triage and treatment. These findings are also important parts of the quality assessment. Several triage tools have been suggested to assess the severity of ED patients and predict their outcomes. The Emergency Severity Index and the Canadian Triage and Acuity Scale (CTAS) are common examples [1]. Both Triage systems are based on a Severity assessment completed by Triage nurses. However, severity assessment depends on an individual’s knowledge of specific triage guidelines, which may vary with personal experience and working environ- ments [2]. The policies and medical resources of each institute or community health care system may also influence the triage score and make the score less objective. A more feasible and valid tool is needed that is less subjective and consuming of resources. This new tool could be used to more compare ED performance objectively among hospitals, communities, and even countries while adjusting for severity.

Fortunately, electronic medical records and other health care information technologies have been adopted into the patient care processes of many hospitals [3,4]. Such an electronic database has provided an infrastructure to measure and evaluate quantitatively the ED process, facilitate the improvement of patient safety, and ensure the quality of care [5]. Therefore, predicting ED severity using a tool that uses information from a large, population-based electronic medical database might have great clinical value. Among the various itEMS used in predicting ED severity, the chief presenting complaint of ED patients would be a good variable to have in an electronic database. The chief complaint is also likely to be important in the development of a final diagnosis. Chief complaints are already a main component in the CTAS triage system [6,7] and in the Emergency Medical Services dispatch protocols, such as the Medical Priority Dispatch System [8,9]. These chief complaints are usually coded in an electronic medical record according to the Unified Medical Language System (UMLS) [10]. Therefore, we are able to derive unique and universal codes for the chief complaint of each patient. If severity could be estimated using chief compliant codes, it would be possible to assign specific severities based on the chief complaints of all ED patients.

The goal of this investigation

The goal of this investigation was to develop and validate a feasible and objective tool to assess the severity of ED patients and to predict their clinical outcome using ED chief complaint codes.

Materials and methods

Study design

This investigation was a retrospective observational study using the National Emergency Department Information System (NEDIS) of Korea, which is a nationwide electronic Emergency medical care database. The institutional review board at the study institution approved this project.

Study setting

This was a nationwide study in a country with a population of 48 million. Based on the data from 2007, there are approximately 440 EDs, including 16 level 1 regional EDs, 4 specialty care EDs, 117 level 2 local EDs, and 303 local emergency facilities, designated by the national and local governments. Level 1 EDs, specialty care EDs, and level 2 EDs provide highly qualified emergency care 24 hours a day, 7 days a week, and are staffed by board-certified emergency physicians. The annual number of patients treated in Korean EDs was approximately

8.5 million in 2008 with an observed yearly increase of 10%. The 16 level 1 EDs and the 4 specialty care EDs have their own general inpatient ward (no. of beds, 30) and emergency intensive care beds (no. of beds, 20).

Study phases and study population

The different phases of the study are summarized in Fig. 1. The first phase was the development of the excess mortality ratio-based ESI (EMR-ESI), the second phase was the internal validation of the EMR-ESI, the third phase was the external validation of the EMR-ESI, and the fourth phase compared the performance of the EMR-ESI with the current version of the ESI (version 4). The first and second phases used a primary data set. The third phase used a secondary data set, and the final phase used a tertiary data set.

The primary and secondary data sets were from the NEDIS. The NEDIS is notably similar to the Canadian Emergency Department Information System [7], which includes demographic parameters, clinical parameters (chief complaint codes, vital signs, and mental status), Diagnosis codes, and treatment parameters. These data were collected from a network automatically merged with the main national computer server and each hospital medical record server. The patient information was transmitted from the hospital to the main server within

2 weeks of discharge from the ED or ward (when the patient was admitted).

The NEDIS was initiated in July 2004 and was set up in 16 designated level 1 EDs. In January 2006, 45 level 2 EDs joined the NEDIS. In December 2006, 4 specialty care EDs and 49 level 2 EDs joined this system. There were also

Steps Data source (study period) No of ED

First step

Development of EMR-ESI

Primary

dataset

Jan. 2006-Dec. 2008

118 EDs

(36 months)

Second step

Internal Validation

Primary

dataset

Jan. 2006-Dec. 2008

118 EDs

(36 months)

Third step

External Validation

Secondary

dataset

Jan. 2009-Dec. 2009

(12 months)

119 EDs

Fourth step

Comparison of performance

Tertiary

dataset

Jan. 2009-Dec. 2009

(12 months)

1 ED

Fig. 1 Study steps and data sources. The primary and secondary data sets were from the NEDIS database, but the tertiary data set was from a single regional ED, which had the information necessary to calculate the ESI version 4.

another 3 level 3 EDs participating in this system. As the number of EDs involved in the NEDIS increased in the course of several years, data transmission was unstable for a few months during the initial stage. When the NEDIS was initially implemented in the EDs, data transmission was unstable, due to missing or erroneous chief complaint codes or disposition results. Therefore, we included only patients visiting the ED after the system was stabilized. The monthly number of patients had been steady in each hospital by that point. Excluded from the analysis were patients for whom the following information was lacking: chief complaint code, ED and hospital inpatient outcome after treatment, systolic blood pressure (SBP), respiratory rate, and heart rate. The chief complaint code was decided by emergency physicians, and other parameters were recorded by emergency nurses. The chief complaint was determined at the time of arrival to the ED. Most of the information was prospectively recorded and transferred to the NEDIS server 14 days after hospital discharge. In the NEDIS database, most of the primary information concerning demographics, mode of arrival to the ED, ED arrival time, and disposition time were recorded by an emergency nurse, whereas the diagnosis code, final diagnosis, and treatments administered in the ED or inpatient ward are recorded by a physician. The chief complaint of patients was also determined and recorded by a physician at the time of arrival to the ED. The National emergency medical center, the coordinating headquarters for the nationwide emergency care system, conducts quality control on a monthly basis using sampled NEDIS data from each hospital. Every year, the National

Emergency Medical Center, under the Emergency Medical Services act, evaluates all EDs with respect to structure requirements, processes, and outcomes. To complete this evaluation, the NEDIS database is used, and such variables as trauma care, stroke care, and myocardial infarction management are reviewed. The first step of this nationwide quality assessment of ED performance is to evaluate the validity of NEDIS information. Because significant financial support may accompany a positive evaluation based on the NEDIS data, all EDs should try to meet the requirements and have a quality control program in place. There are various quality control programs that are available.

In the primary data set for the development and internal validation of the EMR-ESI, we included all patients visiting EDs from January 2006 to December 2008 (36 months) whose information was registered in the NEDIS database (total no. of ED, 118). The secondary data set for the external validation included information from patients presenting to an ED from January to December 2009 (12 months; total no. of ED, 119). A tertiary data set for comparing the EMR-ESI and the ESI was constructed from patients visiting the ED from January to December 2009 (12 months) in an urban Tertiary teaching hospital with 47 000 annual visits to the ED. Unlike the primary or secondary data set, this data set has triage results based on the ESI version 4 (original ESI), which is popular in the United States [11]. The ESI-4 was scored and recorded by triage nurses who had attended a training program recommended by the Emergency Nurse’s Association [1,11].

Outcome measure

Our primary outcome was the discriminatory power of the EMR-ESI for the prediction of hospital mortality, ED mortality, and admission mortality. Emergency department mortality was defined as mortality occurring during the ED stay, and admission mortality was defined as mortality occurring after admission. Hospital mortality was defined as the overall mortality in the ED or mortality after admission. We also evaluated the Predictive ability of the EMR-ESI for inpatient admission. Patients who died in the ED, were admitted, or transferred to another facility were considered to be “admissions.”

Calculation of EMR-ESI

The excess mortality ratio was defined as the ratio of the sex-age standardized mortality for the population compared with the same specific condition and sex-age standardized mortality as the general population [12-14]. To develop the EMR-ESI, we used the UMLS code corresponding to each initial chief complaint. We calculated the hospital mortality for each patient group with the same chief complaint UMLS code. We computed the ratio of age-sex standardized hospital mortality for each group and the age-sex standard- ized mortality of the entire 2006 population of Korea. Thus, the EMR-ESI for each UMLS code for hospital mortality would always be greater than “0”, and the greater the EMR- ESI, the more severe the condition of the patient with that chief complaint. Therefore, each UMLS code had its own EMR-ESI value.

Development and internal validation

We calculated descriptive statistics for the demographic and clinical parameters of the participants using means and SD for continuous variables and percentages for categorical variables in the primary data set.

We assessed the internal validity of the EMR-ESI using area under the receiver operating characteristic curve with 95% confidence intervals (CIs) from various logistic regression models. First, we assessed the internal validity of the EMR-ESI using the AUC with a 95% CI from a univariate logistic model using the EMR-ESI alone. Second, we assessed the internal validity using the AUC with a 95% CI from a multivariate logistic regression model adjusted for sex and age categorized by 10-year intervals. Finally, we assessed the internal validity using the AUC with a 95% CI from a multivariate logistic regression model adjusted for 7 clinical parameters: sex, age (measured in 10-year intervals), ambulance use (prehospital ambulance, interhospital ambulance, or nonuse of ambulance), mental status (alert, verbal response, pain response, or unresponsive), SBP (shock b90 mm Hg or normal >=90 mm Hg), respiratory rate (RR b10 or RR N30

or normal, 10 <= RR <= 30] and heart rate (HR b60 or HR N100 or normal, 60 <= HR <= 100) (Table 1). We reported the adjusted odds ratio (OR) of each variable with a 95% CI and used the Hosmer-Lemeshow ?2 to test the goodness of fit of the model. The primary data set was used for the internal validation.

External validation and comparison of Discriminative performance for hospital outcome

The secondary data set was used for external validation using the same method, with a univariate and multivariate logistic regression adjusting for the same clinical parameters. We also conducted an external validation using subgroup data sets, for example, data from level 1 EDs or from level 2 EDs. To compare the EMR-ESI and the original ESI, we compared the discriminative power of both methods for the prediction of hospital mortality and admission using an AUC analysis.

Results

Characteristics of data sets

(primary, secondary, and tertiary data sets)

We obtained clinical data on 6 195 275 patients who visited 118 eligible EDs between January 2006 and December 2008. Of these patients, 807 925 (13.0%) were excluded because data transmission was unstable, due to missing or erroneous chief complaint codes or disposition results. Also excluded were 673 888 patients (10.9%) because of the lack of clinical information (chief complaint code, ED and hospital inpatient outcome after treatment, SBP, RR, and/or HR). Thus, 4 713 462 patients (76.1%) were included in the development of the EMR-ESI (primary data set). To assess external validity, we gathered clinical information from 3 920 218 patients who visited 119 eligible EDs from January 2009 to December 2009. According to the same exclusion criteria, 183 914 episodes (4.7%) were excluded because of the instability of chief complaint code inputs and disposition results. Another 313 439 patients (7.9%) were excluded because of lack of clinical informa- tion. Thus, 3 422 865 cases (87.4%) were used for external validation and the subgroup analysis (secondary data set). Regarding the tertiary data set, from January to December 2009 (12 months), 49 394 patients visited an urban academic tertiary-level ED, and 2280 (4.6%) of these patients were excluded because of lack of clinical information. Thus, 47 114 patients (95.4%) were included in the comparison of the discriminatory performance in hospital outcomes be- tween the EMR-ESI and the original ESI (tertiary data set). Table 1 displays the demographic and clinical characteristics of the 3 study populations.

Characteristic	Primary data set a			Secondary data set b			Tertiary data set c
	n	%		n	%		n	%

Total	4 713 462	100	3 422 865	100	47 114	100
Age (y), mean +- SD	35.3 +- 24.5		33.7 +- 24.7		39.9 +- 26.2
Sex
Male	2 535 817	53.8	1 825 948	53.4	24 646	52.3
Female	2 177 645	46.2	1 596 917	46.7	22 468	47.7
Ambulance use
Prehospital ambulance	561 705	11.9	397 152	11.6	5102	10.8
Interhospital ambulance	204 662	4.3	135 560	4.0	2136	4.5
Nonambulance	3 947 095	83.7	2 890 153	84.4	39 876	84.7
Mental status
Alert	4 573 666	97.0	3 323 705	97.2	45 437	96.4
Verbal response	55 755	1.2	36 455	1.1	1066	2.3
Painful response	41 654	0.9	28 217	0.8	372	0.8
Unresponsiveness	42 387	0.9	31 291	0.9	239	0.5
SBP (mm Hg)
90 <= SBP	3 825 952	81.2	2 640 674	77.2	43 918	93.2
0 <= SBPb90	887 510	18.8	782 191	22.9	3196	6.8
RR (per min)
10 <= RR <= 30	4 184 775	88.8	2 971 108	86.8	43 790	92.9
0 <= RR b 10, 30 b RR	528 687	11.2	451 757	13.2	3324	7.1
HR (per min)
60 <= HR <= 100	3 294 957	69.9	2 262 979	66.1	27 047	57.4
0 <= HR b 60, 100 b HR	1 418 505	30.1	1 159 886	33.9	20 067	42.6
ED disposition
Discharge	3 667 746	77.8	2 714 419	79.3	34 442	73.1
Transfer to other hospital	100 129	2.1	57 971	1.7	1333	2.8
Admission	910 302	19.3	628 224	18.4	11 169	23.7
Death in ED	35 285	0.8	22 251	0.7	170	0.4
a For development and internal validation of the EMR-ESI. b For external validation of the EMR-ESI. c For comparison of discriminative performance for hospital outcome of the EMR-ESI.

Development of the EMR-ESI

Table 1 Demographic and clinical characteristics of the 3 study populations

The NEDIS was designed with the UMLS code-based chief complaint code system. According to the semantic network of the UMLS, 7557 chief complaint codes in the primary data set were classified into signs or symptoms (68.1%), injuries or poisonings (12.5%), diseases or syndromes (6.5%), and findings (4.4%), respectively. The number of ED visits per each UMLS code was 623.6 +- 8788.1 (mean +- SD). The most frequent UMLS chief complaint code was abdominal pain, followed by fever, headache, dizziness, and vomiting. The EMR-ESI for each chief complaint code was between 0 to 6389.45 (mean +- SD,

1.11 +- 4.67; median, 0.70; 25 percentile, 0.24; 75 percentile, 1.19). For example, the EMR-ESI for the chief complaint code of “chest pain” was 1.234. This finding means that a patient coming to the ED with chest pain is 1.234 times more likely to die in the hospital compared with the overall population with the same age and sex (Appendix A). Examples of calculations of the EMR-ESI with a specific chief complaint code are described in Table 2. The exact

numerator and denominator used in the calculation of the EMR-ESI and the length of hospitalization (from ED visit to death) for each chief complaint are also explained in Table 2. To identify the frequently used chief complaint codes of Critical illnesses, we listed chief complaint codes used more than 100 times during the study period with a high EMR-ESI value. There were 1086 (14.4%) codes of the 7558 codes possible that were related to ED chief complaints causing more than 100 ED visits. “Hanging” was associated with the highest EMR-ESI among all chief complaints related to more than 100 visits. Other highly estimated chief complaint codes

are presented in Table 2.

Internal validation

Using a multivariate logistic regression model adjusted for age, ambulance use, mental status, SBP, RR, HR, and sex, the adjusted OR of the EMR-ESI for hospital mortality was 1.11 (95% CI, 1.11-1.12) per a 1.0 increase. The adjusted OR for ED mortality was 1.07 (95% CI, 1.07-1.07), and the adjusted OR for admission mortality was 1.01 (95% CI, 1.01-1.01).

Male			Female			hospitalization of observed
ED	Observed	Expected	ED	Observed	Expected	deaths (mean
visits	deaths a	deaths b	visits	deaths a	deaths b	+- 95%CI,min)

Hanging	250	74	2.52	245	62	1.07	514.4 +- 1045.7	37.85
Cardiopulmonary resuscitation	253	168	5.17	151	112	3.01	292.5 +- 559.5	34.22
Arrested progression	278	232	6.63	155	125	4.66	194.2 +- 439.2	31.63
Coma states	172	108	2.83	81	47	2.11	387.5 +- 928.2	31.36
Cardiac respiratory arrest	118	104	3.62	63	58	2.11	212.7 +- 485.5	28.27
Heart arrest	411	343	11.49	234	199	8.17	213.1 +- 524.0	27.57
Comatose	951	605	22.62	572	347	14.46	474.0 +- 1864.3	25.68
Death, sudden	86	83	3.57	74	71	2.74	104.2 +- 223.4	24.43
Cessation of life	1766	1588	62.93	1242	1130	55.87	136.1 +- 376.1	22.88
Death (finding)	75	72	3.13	76	74	3.54	47.3 +- 128.3	21.90
Apnea	2945	2595	108.75	2216	1970	105.28	128.0 +- 379.8	21.33
Determination of death	265	235	10.00	170	145	8.12	218.5 +- 544.1	20.96
Death by strangulation	66	65	2.88	51	51	2.74	70.0 +- 234.8	20.64
Dead on arrival at hospital	7070	6726	296.22	5630	5378	296.41	71.0 +- 285.3	20.42
a Number of observed overall inhospital deaths with a specific chief complaint code by sex group in the primary data set. b Number of expected overall inhospital deaths with a specific chief complaint code by sex group derived from the age-sex standardized mortality of general population. c The EMR-ESI is calculated by the equation: EMR-ESI = (male observed death + female observed death)/(male expected death + female expected death).

The adjusted ORs for other potential variables are displayed in Table 3. High EMR-ESI, old age, interhospital ambulance use, poor mental status, SBP below 90 mm Hg, respiratory distress,

Table 2 Examples of calculation of the EMR-ESI of chief complaint codes with more than 100 ED visits

Chief complaint code No. of ED visits and deaths with a specific chief complaint code Length of

EMR- ESI ^c

Table 3 Adjusted ORs and 95% CI of the EMR-ESI and clinical parameters on the hospital outcome

tachycardia, and bradycardia showed higher adjusted ORs for hospital outcomes. The AUC value of the EMR-ESI showed good discriminatory power for the prediction of hospital, ED,

Variables	Total		Hospital mortality		ED mortality		Admission mortality
	n		AOR a 95% CI		AOR a 95% CI		AOR a 95% CI	n

EMR-ESI b	4 713 462	1.11	1.11	1.12	1.07	1.07	1.07	1.01	1.01	1.01
Age c	4 713 462	1.60	1.59	1.61	1.52	1.51	1.54	1.54	1.53	1.55
Ambulance use
Prehospital	561 705	1			1			1
Interhospital	204 662	1.64	1.60	1.68	1.33	1.27	1.39	1.64	1.60	1.69
Nonuse	3 947 095	0.35	0.34	0.36	0.22	0.21	0.23	0.39	0.38	0.40
Mental status
Alert	4 573 666	1			1			1
Verbal	55 755	2.75	2.65	2.85	3.51	3.27	3.78	3.12	3.00	3.25
Painful	41 654	3.41	3.29	3.54	4.44	4.16	4.75	4.48	4.31	4.66
Unresponsive	42 387	12.13	11.67	12.60	26.27	25.04	27.56	1.67	1.57	1.77
SBP (mm Hg)
90 <= SBP	3 825 952	1			1			1
0 <= SBP b 90	887 510	2.88	2.79	2.97	8.78	8.36	9.22	1.18	1.13	1.23
RR (per min)
10 <= RR <= 30	4 184 775	1			1			1
0 <= RR b 10, 30 b RR	528 687	1.38	1.34	1.43	2.52	2.40	2.65	0.74	0.71	0.78
HR (per min)
60 <= HR <= 100	3 294 957	1			1			1
0 <= HR b 60, 100 b HR	1 418 505	2.26	2.20	2.31	2.07	1.97	2.17	2.40	2.34	2.46
AOR indicates adjusted OR. a Adjusted for EMR-ESI, age by 10 years, ambulance use, mental status, SBP category, RR category, and HR category (sex was used as a class strata). b EMR-ESI: per 1.0. c Age: per 10 years.

and admission mortality, with reasonable power to predict admission being demonstrated (Table 4).

External validation and subgroup analysis

We tested the external validity of the EMR-ESI for predicting hospital outcomes using the new data set (secondary data set). The adjusted OR of the EMR-ESI for hospital mortality was 1.09 (95% CI, 1.09-1.10). The adjusted ORs for ED mortality and admission mortality were 1.07 (95% CI, 1.07-1.07) and 1.57 (95% CI, 1.56-1.59), respec-

tively. The AUC showed strong good discriminatory power for the prediction of hospital, ED, and admission mortality (Table 5). When we performed a subgroup analysis, both the level 1 ED group and the level 2 ED group showed excellent discriminatory power for the prediction of hospital outcomes. This finding was similar to that of the entire group. There were 600 848 visits (17.6%) in 16 level 1 EDs and 2 822 017 visits (82.4%) in 103 level 2 EDs. The EMR-ESI in the subgroups also showed good discriminatory power for prediction of hospital outcome, regardless of the level of ED.

Comparison of performance between the EMR-ESI vs the original ESI

A total of 47 114 patients were included in the tertiary data set. When we compared the performances between the

Table 4 Internal validation of the EMR-ESI for the prediction of hospital outcome

Hospital AUC 95% CI Hosmer- P outcome Lemeshow ?2

Not adjusted Hospital 0.901 0.900 0.903 1 883 377.8 b.001 mortality ED mortality 0.962 0.961 0.963 2 625 910.5 b.001 Admission 0.832 0.829 0.834 588287.2 b.001 mortality Admission 0.681 0.680 0.681 5 256 063.4 b.001 Adjusted for age and sex a Hospital 0.927 0.926 0.927 3 593 789.4 b.001 mortality ED mortality 0.967 0.966 0.968 3 893 188.2 b.001 Admission 0.849 0.847 0.851 982005.9 b.001 mortality Admission 0.712 0.711 0.713 5 649 950.4 b.001 Adjusted for age, sex, and other clinical parameters b Hospital 0.947 0.946 0.948 5 060 858.4 b.001 mortality ED mortality 0.984 0.984 0.985 8 051 708.8 b.001 Admission 0.895 0.893 0.896 1 542 391.4 b.001 mortality Admission 0.738 0.737 0.738 6 423 059.6 b.001

a Adjusted for age by 10 years (sex was used as a class strata). b Adjusted for age by 10 years, ambulance use, mental status, SBP category, RR, and HR category (sex was used as a class strata).

Hospital AUC outcome	95% CI		Hosmer- Lemeshow ?2	P
Not adjusted Hospital 0.884 0.882 0.886 898416.0 b.001 mortality ED mortality 0.955 0.954 0.957 1 098 233.8 b.001 Admission 0.815 0.812 0.817 316695.5 b.001 mortality Admission 0.659 0.659 0.660 3 214 537.1 b.001 Adjusted for age and sex a Hospital 0.926 0.924 0.927 2 631 148.3 b.001 mortality ED mortality 0.968 0.966 0.969 2 460 219.7 b.001 Admission 0.860 0.858 0.862 861990.3 b.001 mortality Admission 0.730 0.729 0.730 4 248 566.4 b.001 Adjusted for age, sex, and other clinical parametersb Hospital 0.947 0.946 0.948 3 655 318.9 b.001 mortality ED mortality 0.986 0.985 0.987 5 785 562.5 b.001 Admission 0.902 0.901 0.904 1 318 351.5 b.001 mortality Admission 0.753 0.752 0.754 4 726 534.7 b.001 Subgroup analysis: adjusted for age, sex, and other clinical parameters b
Level 1 EDs (ED n = 16) Hospital 0.922 mortality ED mortality 0.978 Admission 0.875 mortality Admission 0.750 Level 2 EDs (ED n = 103) Hospital 0.953 mortality ED mortality 0.987 Admission 0.909 mortality Admission 0.752	0.919	0.924	549882.2	b.001
	0.976	0.981	840239.8	b.001
	0.872	0.879	261594.5	b.001
	0.749	0.752	1 044 330.0	b.001
	0.952	0.954	3 264 018.5	b.001
	0.987	0.988	5 163 122.3	b.001
	0.907	0.910	1 061 059.6	b.001
	0.752	0.753	3 643 076.1	b.001
a Adjusted for age by 10 years (sex was used as a class strata). b Adjusted for age by 10 years, ambulance use, mental status, SBP category, RR, and HR category (sex was used as a class strata).

EMR-ESI vs the original ESI-4 using the AUC for the prediction of hospital outcome, the EMR-ESI showed a significantly better predictive ability than the original ESI-4 for the prediction of hospital mortality (P b .001), ED mortality (P b .001), admission mortality (P = .024), and hospital admission (P b .001) (Table 6).

Table 5 External validation of the EMR-ESI for the prediction of hospital outcome

Limitations

First, although we obtained a large sample size of clinical data from the NEDIS, we excluded 23.9% of the original data set because there was no chief complaint code or other

Table 6 Comparison of the discriminative performance of the EMR-ESI and original ESI-4 for predicting outcomes
Hospital outcome	EMR-ESI AUC	95% CI		Original ESI-4 AUC	95% CI	P
Hospital mortality	0.825	0.808-0.842		0.641	0.614-0.669	b.001
ED mortality	0.943	0.918-0.968		0.678	0.611-0.744	b.001
Admission mortality	0.781	0.761-0.801		0.754	0.733-0.775	.024
Admission	0.653	0.647-0.658		0.633	0.629-0.638	b.001

clinical information. Although NEDIS was initiated in late 2004, the data transmission system was notably unstable for a significant period. In particular, the disposition result was missing in many cases. Information on hospital outcome was more complete from secondary data set from 2008, due to a better linkage between the hospital electronic medical system and the NEDIS data server. The cases with a high quality of information (76.1%) were used for the development of the EMR-ESI. However, 87.4% of the data in the secondary data set were used for external validation.

Second, the data set had a significant proportion of missing values for vital signs (SBP, RR, and HR). These signs were especially common in small, level 2 EDs for 2 reasons. One reason was that the care provider did not record or check the vital signs, possibly because the patients were mildly ill or were critically ill. The other reason was due to the improper recording of values, for example, the presence of negative values. These errors were decreased in the secondary data set. Third, the tertiary analysis comparing the EMR-ESI to the original ESI version 4 was limited to an urban academic hospital. Because many of the hospitals in Korea do not routinely determine the original ESI, we derived the tertiary data set for comparing the EMR-ESI to the ESI from the hospital administering the ESI by trained triage nurses. To generalize the results, data sets derived from more hospitals

using the ESI would be required.

A fourth limitation of our study is its limited generalizabil- ity. We tested the external validity of our EMR-ESI using a single ED database (tertiary data set). Therefore, it may not extrapolate well to other EDs. Our emergency care system is also different from those in North America and Europe. Our results also may not be applicable to EDs that do not use an electronic code system based on UMLS chief complaint codes. Finally, the EMR-ESI used many kinds of chief complaint codes and showed a large SD of EMR-ESI values for each chief complaint. The benefits of using the EMR-ESI include its objectiveness and feasibility. However, implementing this tool into clinical practice would require that chief complaint codes be unified and refined to prevent overestimation or

underestimation of chief complaints.

Discussion

The goal of this investigation was to develop a feasible and objective scaling tool to estimate ED severity and predict

clinical outcomes. To develop and internally validate the EMR-ESI, we collected clinical data from more than 4 700 000 ED patient visits from across the nation. We also analyzed more than 3 400 000 ED visits to externally validate the EMR-ESI. This study was possible, due to a vast nationwide database that housed electronic medical records from each hospital. These technological improvements in medical record keeping presented a unique opportunity to collect data every day and everywhere, as new information and records accumulated in the computer system in a real- time manner. Therefore, we were able to use this rich data set to develop our triage tool with great validity [15,16].

In clinical practice, the prediction of hospital outcomes, especially for emergency patients, is highly important [17]. Thus, Prediction tools for clinical outcomes in trauma, acute coronary syndrome, stroke, and even poisoning have been previously developed and used in real practice [18-21]. However, there is a large gap when it comes to clinical prediction tools for the many other clinical syndromes and conditions that may present to an ED. A universal and objective tool to predict clinical outcomes that is feasible, repeatable, and objective is greatly needed.

The use of 3- or 5-level triage tools, such as the CTAS and ESI, have been encouraged, despite being both resource consuming and time consuming. These triage tools are also relatively subjective [22-24]. The assessment of severity by a triage nurse using the ESI could be influenced by many factors, including the policy at that particular institution or the nurse’s level of training. The EMR-ESI is an objective triage tool that is not affected by the occurrence of case mix. It also has a low possibility of being influenced by other external factors. The objectivity of the EMR-ESI is based on the use of chief complaints and objective physiologic profiles, rather than subjective and potentially presumptive assessments.

Critical care researchers also have developed several scoring systems commonly used in intensive care units [25- 27]. These scoring systems use clinical variables measuring specific physiologic functions. For instance, to assess severity using the Acute Physiology and Chronic Health Evaluation III, Serum bilirubin and Albumin levels are required. In the ED, adding a laboratory examination during triage is not appropriate because of the cost and time delays associated with doing so. To prioritize ED patients using the EMR-ESI, no further laboratory tests are required, and only History taking to determine the chief complaint and

measurement of simple physiologic parameters (eg, vital signs and mental status) is required. The use of feasible, relatively simple, and rapidly available clinical parameters is one of the major advantages of the EMR-ESI.

The EMR-ESI had a good discriminatory power for the prediction of mortality. In the subgroup analysis based on the level of each ED, a good predictive value was still maintained.

Although the EMR-ESI showed an overall good perfor- mance, it was less predictive for admission mortality compared with hospital mortality or ED mortality. The AUC value of the EMR-ESI was relatively low in the prediction of admission mortality compared with the hospital or ED mortality. Discrepancy of the adjusted OR in admission mortality between the internal and external validation (1.01-1.57) was also larger than in hospital or ED mortality. To predict admission mortality more accu- rately, the EMR-ESI should predict whether admission will occur and include the occurrence of fatal complications over a long-term period. The EMR-ESI is mainly based on the initial chief complaint at the time of arrival to the ED. The initial chief complaint also has limitations in predicting the long-term clinical outcome after admission. Admission could be affected by many confounding factors regardless of the initial clinical severity. These factors include ward occupan- cy rate at the institution, the Financial burden of medical care, and the compliance of patients. Death occurring after admission could be affected by many factors during the relatively long time that the patient spends in the ward after the initial clinical assessment in the ED [28,29].

In total, 7558 chief complaint codes were used to develop the new tool, and these complaints covered most of the medical terminology required to describe initial chief complaints of ED patients. Each chief complaint code received a unique value in the EMR-ESI. However, there were several chief complaint codes representing the same disease entity or similar clinical status. Thus, it was possible for a young patient with a low predicted mortality in the general population, who died (especially with a disease entity with an extremely low incidence rate), to lead to overestimation of the EMR-ESI, due to the excess mortality ratio. Therefore, unification of similar chief complaint codes and refinement of overestimated and underestimated chief codes are required before implementing the EMR-ESI.

Ensuring the quality of care in emergency medicine is important to improve patient safety. To assess the quality of performance in an ED, development of a valid and feasible quality indicator is required [30,31]. Lindsay P et al con- ducted a survey to determine if mortality was an acceptable clinical indicator of ED performance using a modified Delphi method [30], and at least two thirds of the experts strongly agreed on mortality as a clinical indicator for several conditions, such as acute myocardial infarction. The EMR- ESI was developed based on hospital mortality and thus showed a strong predictive ability for mortality. The Agency for Healthcare Research and Quality has recommended that

“death in low mortality disease related groups” is one component of patient safety indicators [32]. These indicators are intended to identify inhospital deaths in patients admitted for an extremely low-mortality condition. The EMR-ESI also identified clinical conditions with relatively high mortality compared with the general population. Therefore, the EMR- ESI could possibly be applied as a New indicator that quantifies the safety of the ED patient. However, further research is required before adopting the EMR-ESI as a quality indicator. We are currently planning to develop a Mortality prediction formula for emergency patients based on the EMR-ESI. Further research using the EMR-ESI for developing a prediction model of hospital mortality is required. Ultimately, if it is proven successful, the EMR-ESI could easily apply chief complaints and minimal clinical parameters to predict hospital mortality.

Conclusion

We developed and validated the EMR-ESI using UMLS- based chief complaint codes that were highly predictive of hospital outcomes. In the future, the EMR-ESI may be used as a simple triage tool and for the evaluation of ED performance and excess mortality.

Supplementary materials related to this article can be found online at doi:10.1016/j.ajem.2011.12.011.

References

1. Fernandes CM, Tanabe P, Gilboy N, Johnson LA, McNair RS, Rosenau AM, et al. Five level triage: a report from the ACEP/ENA five level triage task force. J Emerg Nurs 2005;31:39-50.
2. Dong SL, Bullard MJ, Meurer DP, Colman I, Blitz S, Holroyd BR, et al. emergency triage: comparing a novel computer triage program with standard triage. Acad Emerg Med 2005;12:502-7.
3. DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A, et al. electronic health records in ambulatory care - a national survey of physicians. N Engl J Med 2008;359:50-60.
4. Burt CW, Sisk JE. Which physicians and practices are using electronic medical records? Health Aff 2005;24:1334-43.
5. Handler JA, Feied CF, Coonan K, Vozenilek J, Gillam M, Peacock PR, et al. Computerized physician Order entry and online decision support. Acad Emerg Med 2004;11:1135-41.
6. Murray M, Bullard M, Grafstein E. Revisions to the canadian emergency department triage and acuity scale implementation guidelines. CJEM 2004;6:421-7.
7. Grafstein E, Unger B, Bullard M, Innes G. Canadian Emergency Department Information System (CEDIS) Presenting Complaint List (Version 1.0). CJEM 2003;5:27-34.
8. Michael GE, Sporer KA. Validation of low-acuity emergency medical services dispatch codes. Prehosp Emerg Care 2005;9:429-33.
9. Sporer KA, Johnson NJ, Yeh CC, Youngblood GM. Can Emergency Medical Dispatch codes predict Prehospital interventions for common 9-1-1 call types? Prehosp Emerg Care 2008;12:470-8.
10. Travers DA, Haas SW. Unified medical language system coverage of emergency-medicine chief complaints. Acad Emerg Med 2006;13: 1319-23.
11. Gilboy N, Tanabe P, Travers DA, Rosenau AM, Eitel DR. Emergency Severity Index Version 4: implementation handbook. Rockville (Md): Agency for Healthcare Research and Quality; 2005.
12. Geraldine MC, Yates DW, Hollis S. UnExpected mortality in patients discharged from the emergency department following an episode of nontraumatic chest pain. Eur J Emerg Med 2008;15:3-8.
13. Gunnarsdottir OS, Rafnsson V. Mortality of the users of a hospital emergency department. Emerg Med J 2006;23:269-73.
14. Huber-Wagner S, Lefering R, Qvick LM, Korner M, Kay MV, Pfeifer KJ, et al, Working Group on Polytrauma of the German Trauma Society. Effect of whole-body CT during trauma resuscita- tion on survival: a retrospective, multicentre study. Lancet 2009;373: 1455-61.
15. Burt CW, McCaig LF, Valverde RH. Analysis of Ambulance transports and diversions among US emergency departments. Ann Emerg Med 2006;47:317-26.
16. Gravel J, Manzano S, Arsenault M. Validity of the Canadian Paediatric Triage and Acuity Scale in a tertiary care hospital. CJEM 2009;11:23-8.
17. Hargrove J, Nguyen HB. Bench-to-bedside review: outcome pre- dictions for critically ill patients in the emergency department. Crit Care 2005;9:376-83.
18. Bulut M, Koksal O, Korkmaz A, Turan M, Ozguc H. Childhood falls: characteristics, outcome, and comparison of the Injury Severity Score and New Injury Severity Score. Emerg Med J 2006;23:540-5.
19. Body R, Carley S, McDowell G, Ferguson J, Mackway-Jones K. Can a modified Thrombolysis in Myocardial Infarction risk score outperform the original for risk stratifying emergency department patients with chest pain? Emerg Med J 2009;26:95-9.
20. Hallevi H, Albright KC, Martin-Schild SB, Barreto AD, Morales MM, Bornstein N, et al. Recovery after ischemic stroke: criteria for good outcome by level of disability at day 7. Cerebrovasc Dis 2009;28:341-8.
21. Senarathna L, Eddleston M, Wilks MF, Woollen BH, Tomenson JA, Roberts DM, et al. Prediction of outcome after Paraquat poisoning by measurement of the plasma paraquat concentration. QJM 2009;102: 251-9.
22. Tanabe P, Gimbel R, Yarnold PR, Adams JG. The emergency severity index (version 3) 5-level triage system scores predict ED resource consumption. J Emerg Nurs 2004;30:22-9.
23. Wuerz RC, Milne LW, Eitel DR, Travers D, Gilboy N. Reliability and validity of a new five-level triage instrument. Acad Emerg Med 2000;7:236-42.
24. Wuerz RC, Travers D, Gilboy N, Eitel DR, Rosenau A, Yazhari R. Implementation and refinement of the emergency severity index. Acad Emerg Med 2001;8:170-6.
25. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a Severity of disease classification system. Crit Care Med 1985;13: 818-29.
26. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, et al. The APACHE III prognostic system. risk prediction of hospital mortality for critically ill hospitalized adults. Chest 1991;100:1619-36.
27. Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993;270:2957-63.
28. Mutrie D, Bailey SK, Malik S. Individual emergency physician admission rates: predictably unpredictable. CJEM 2009;11:149-55.
29. Parkhe M, Myles PS, Leach DS, Maclean AV. Outcome of emergency department patients with delayed admission to an intensive care unit. Emerg Med 2002;14:50-7.
30. Lindsay P, Schull M, Bronskill S, Anderson G. The development of indicators to measure the quality of clinical care in emergency departments following a modified-Delphi approach. Acad Emerg Med 2002;9:1131-9.
31. Graff L, Stevens C, Spaite D, Foody J. Measuring and improving quality in emergency medicine. Acad Emerg Med 2002;9:1091-107.
32. U.S. Department of Health and Human Services (DHHS), Agency for Healthcare Research and Quality. Guide to Patient Safety Indicators. Rockville (Md): DHHS, AHRQ; 2006. Available at http://www. qualityindicators.ahrq.gov/Downloads/Software/SAS/V30/psi_ guide_v30.pdf. Accessed on December 30, 2011.