a b s t r a c t

Objectives: To demonstrate the accuracy, sensitivity, and specificity of the Emergency Severity Index , quick sepsis-related organ failure assessment , Systemic Inflammatory Response Syndrome criteria, and National Early Warning Score for predicting in-hospital mortality and intensive care unit (ICU) ad- mission in suspected sepsis patients.

Methods: A retrospective cohort study conducted at a tertiary care hospital, Thailand. Suspected sepsis was de- fined by a combination of (1) hemoculture collection and (2) the initiation of Intravenous antibiotics therapy during the emergency department (ED) visit. The accuracy of each scoring system for predicting in-hospital mor- tality and ICU admission was analyzed.

Results: A total of 8177 patients (median age: 62 years, 52.3% men) were enrolled in the study, 509 (6.2%) of whom died and 1810 (22.1%) of whom were admitted to the ICU. The ESI and NEWS had comparable accuracy for predicting in-hospital mortality (AUC of 0.70, 95% confidence interval [CI] 0.68 to 0.73 and AUC of 0.73, 95%

CI 0.70 to 0.75) and ICU admission (AUC of 0.75, 95% CI 0.74 to 0.76 and AUC of 0.74, 95% CI 0.72 to 0.75). The ESI level 1-2 had the highest sensitivity for predicting in-hospital mortality (96.7%), and qSOFA >=2 had the highest specificity (86.6%).

Conclusion: The ESI was accurate and had the highest sensitivity for predicting in-hospital mortality and ICU ad-

mission in suspected sepsis patients in the ED. This confirms that the ESI is useful in both ED triage and predicting adverse outcomes in these patients.

(C) 2020

Introduction

Sepsis patients are at high risk for related morbidity and mortality. Therefore, early recognition and rapid appropriate management in the first few hours after emergency department (ED) triage are necessary in order to improve outcomes in these patients [1-4].

A number of sepsis screening tools have been validated for predicting clinical deterioration including the quick Sepsis-related Organ Failure Assessment (qSOFA), Systemic Inflammatory Response Syndrome criteria, and National early warning score [5-12]. A positive qSOFA (score >=2) and positive SIRS criteria (score

>=2) are strongly associated with mortality in sepsis patients in the ED. [8,10,13] A positive qSOFA score was more specific than positive SIRS criteria but had lower sensitivity [7,8,11,14]. However, because screen- ing based on positive SIRS criteria is more sensitive, it may lead to un- necessary admission and, thus, has a limited role in emergency

* Corresponding author at: Faculty of Medicine, Khon Kaen University, 123/2000 Mitraparp Rd, Muang, Khon Kaen 40002, Thailand.

E-mail address: kpraew@kku.ac.th (P. Kotruchin).

settings [8,15]. Recent studies have shown the NEWS to be more accu- rate in predicting mortality in sepsis patients in the ED than the qSOFA or SIRS criteria [5,7,10,16]. Nonetheless, calculating NEWS scores at the bedside can be time-consuming, which limits the practicality of this method [5,17].

The Emergency Severity Index is a five-level ED triage scale currently used by majority of the EDs in the United States that has be- come increasingly widely used globally including in Asian countries [18-21]. The ESI ranges from 1 (highest severity; need for immediate treatment) to 5 (lowest severity; non-urgent) based on disease severity and expected resources needed and can be quickly calculated by trained personnel [18,19]. A recent study found that ESI combined with qSOFA score was a useful Prognostic tool for predicting in-hospital mortality in suspected sepsis patients in the ED. [22] There has only been one pro- spective cohort study comparing ESI with other sepsis screening tools, which found that ESI had high accuracy for predicting in-hospital mor- tality and Intensive care unit admission [23]. However, this find- ing needs to be confirmed due to a limit number of sepsis patients involved in that study. To our knowledge, there has been no large retro- spective cohort study conducted to evaluate the accuracy of ESI and

https://doi.org/10.1016/j.ajem.2020.06.005

0735-6757/(C) 2020

other sepsis Screening tools for predicting in-hospital mortality and ICU admission in an emergency setting. We aimed to demonstrate the accu- racy, sensitivity, and specificity of the ESI, qSOFA, SIRS criteria, and NEWS for predicting in-hospital mortality and ICU admission in suspected sepsis patients in the ED.

Methods

Study design and setting

This retrospective cohort study was conducted at the ED in a tertiary care university hospital with approximately 60000 annual ED visits in Khon Kaen, Thailand. All data were extracted from a computer-based hospital information system. The protocol was approved by the Khon Kaen University Institutional Review Board (HE621165).

Selection of participants

In this consecutive cohort, we included patients with suspected sep- sis who visited the ED between January 1, 2016 and December 31, 2018 by a combination of (1) hemoculture collection and (2) the initiation of intravenous antibiotics therapy during the ED visit. We performed a comprehensive search of the hospital database to identify all patients who met this definition. Patients’ clinical data and prescribed antimicro- bial drugs were reviewed by two of the authors (PP, SK) to exclude pa- tients (1) with age b18 years, (2) administered Prophylactic antibiotics,

(3) presenting with cardiac arrest or symptoms directly related to trauma, (4) transferred from another hospital, or (5) with missing clin- ical data.

Data collection

We used the hospital ED database to collect demographic data in- cluding age, sex, Charlson Comorbidity Index, ESI triage level, initial he- modynamic parameters at ED triage (body temperature [BT], blood pressure [BP], respiratory rate [RR], heart rate [HR], peripheral oxygen saturation [SpO2], level of consciousness), laboratory results (white blood cell count, lactate values), treatments (oxygen therapy, mechani- cal ventilator, vasopressors) and patients outcome (In-hospital death, ICU admission). The ESI triage level was also collected from the ED data- base. In our institute, the ED triage was performed by a triage nurse dur- ing the shift using ESI system. The triage nurse at our institution must have at least 2-year working experience in the ED and have undergone formal ESI triage training. They must also attend to the annual refresher ESI training session every year.

Definitions

The ESI is a five-level ED triage scale that ranges from ESI 1 (high se- verity; need for immediate treatment) to ESI 5 (low severity; non- urgent) according to the ESI algorithm. The qSOFA score consists of three clinical variables: systolic BP (SBP) <= 100 mmHg, RR greater than or equal 22 breaths/min, and altered mental status (defined as a Glasgow Coma Scale score b14). The score ranges from 0 to 3, and a positive qSOFA score is defined as 2 or more points. The SIRS score ranges from 0 to 4 points. One point is assigned for each of the fol- lowing: RR N20 breaths/min, body temperature (BT) higher than 38.0 ?C or lower than 36 ?C, HR N90 beats/min, and white blood cell count N12000/mm3 or b4000/mm3, or N10% bands. A positive SIRS score was defined as 2 or more points. The NEWS score ranges from 0 to 20, based on abnormal values for the following variables: RR, SpO₂, supple- mental oxygen, BT, SBP, HR, and level of consciousness.

Outcomes

The primary and secondary outcomes were in-hospital mortality and the ICU admission at any time point after hospital admission.

Analysis

Categorical variables were compared between groups using ?2 tests, while the continuous variables were compared using student t-tests or Wilcoxon rank sum tests based on the distribution of the data. Accuracy of the scoring systems for predicting both outcomes (in-hospital mor- tality and the ICU admission) was analyzed and reported as area under the receiver operating characteristic-curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). The Youden’s J statistic was calculated to assess the opti- mal cut-off points for the different scores. Statistical analyses were per- formed using STATA version 10.1 (College Station, Texas, USA).

Results

Characteristics of study subjects

A total of 169578 ED visits were recorded during the study period. Of those, 12556 patients met the criteria for suspected sepsis. There were 4379 patients excluded due to age (n = 1758), having been referred from another hospital (n = 1382), having been administered prophy- lactic antibiotics (n = 880), missing clinical data (n = 261), having pre- sented with trauma (n = 83), or having presented with cardiac arrest (n = 15). The final study cohort consisted of 8177 patients (Fig. 1).

The median age of the patients was 62 years old, and 52.3% were men. Of the total study cohort, 6490 patients (79.4%) were categorized into ESI level 1-2, 6149 (75.2%) manifested 2 or more SIRS criteria,

1230 (15.0%) had qSOFA scores >=2, and 2917 (35.7%) had NEWS >=6.

Fig. 1. Study population. There were total of 169578 emergency department visits. Of those, 12556 patients met the criteria for suspected sepsis. There were 4379 patients excluded due to age (n = 1758), having been referred from another hospital (n = 1382), having been administered Prophylactic antibiotics (n = 880), missing clinical data (n = 261), having presented with trauma (n = 83), or having presented with cardiac arrest (n = 15). The final study cohort consisted of 8177 patients.

When we sub-categorized patients who had ESI level 1-2 with by their other scores, we found that 90.2%, 100%, and 99.7% had SIRS criteria >=2, qSOFA >=2, and NEWS >=6, respectively. There were 397 patients (4.9%) who had sepsis-induced hypotension, 575 (7%) who experienced septic shock, and 788 (9.6%) who had positive hemocultures. The first serum lactate was N2 mmol/L in 2193 of 4694 patients (46.7%) who had serum lactate data.

Regarding the Treatment outcomes, there were 509 patients (6.2%) who died and 1810 (22.1%) who were admitted to the ICU. Other base- line characteristics are summarized Table 1.

Accuracy of the scoring systems

Based on the AUC, the NEWS exhibited the highest accuracy for predicting in-hospital mortality in ED patients with suspected sepsis (AUC: 0.73, 95% CI 0.70 to 0.75), followed by ESI (AUC: 0.70, 95% CI

0.68 to 0.73) and qSOFA (AUC of 0.70, 95% CI 0.68 to 0.72), as shown in Fig. 2. The SIRS criteria exhibited the lowest accuracy (AUC: 0.58, 95% CI 0.55 to 0.60). The ESI had a highest accuracy for predicting ICU admission (AUC of 0.75, 95% CI 0.74 to 0.76), followed by the NEWS

(AUC of 0.74, 95% CI 0.72 to 0.50), qSOFA (AUC: 0.697, 95% CI 0.68 to 0.71), and SIRS criteria (AUC: 0.570, 95% CI 0.69 to 0.71; Fig. 3).

Sensitivity and specificity of the scoring systems

Exploratory analyses using the Youden index showed that ESI Level 1-2, qSOFA >=2, NEWS >=6, and SIRS criteria >=2 might be the optimal cut- off points for predicting in-hospital mortality (see Supplementary ma- terial). The sensitivity for predicting in-hospital mortality was highest for ESI level 1-2 at 96.7%, compared with 88.0% for SIRS criteria >=2, 67.6% for NEWS >=6 and 39.5% for qSOFA >=2. The specificity for predicting in-hospital mortality was highest for qSOFA >=2 at 86.6%, following by NEWS >=6 at 66.4%, SIRS criteria >=2 at 25.6%, and ESI level 1-2 at 21.8% (Table 2). When the ESI cut-off point was changed from level 1-2 to level 1, the specificity increased to 88.3%. Note that ESI level 1 had the highest specificity of all of the scores but at the cost of some sensitivity.

Discussion

The present study explored the accuracy of the ESI and other scoring systEMS used for predicting in-hospital mortality and ICU admission in suspected sepsis patients. The accuracy of the ESI, NEWS, and qSOFA for predicting in-hospital mortality was compa- rable. The ESI and NEWS exhibited the highest accuracy for predicting ICU admission. The SIRS criteria fell short in terms of ac- curacy in predicting both in-hospital mortality and ICU admission. The most important finding was that ESI is a useful tool for predicting adverse outcomes in suspected sepsis patients in the ED, as ESI level 1-2 exhibited the highest sensitivity for predicting both in-hospital mortality and the need for ICU admission when compared to the other scores.

The ESI triage system was developed nearly two decades ago to facil- itate the prioritization of ED patients based on disease severity and ex- pected resource needs. Since then, the ESI has been gaining in popularity and has become the most commonly used triage system in the ED in the United States [19,20,25]. One of the benefits of the ESI is its rapidity in identifying ED patients who need immediate attention based on the judgement of experienced medical personnel [18]. Beyond triage, ESI has also been studied for its accuracy as a risk discrimination tool in selected groups of patients. A recent study demonstrated the higher accuracy of ESI in predicting admission to the ICU (AUC = 0.86) when compared to other scores [23]. However, as that study ex- amined only a small number of sepsis patients, its results should be interpreted cautiously. Our findings addressed questions as to the accu- racy of ESI as a tool for predicting adverse outcomes (namely, ICU ad- mission and in-hospital mortality) in suspected sepsis patients with a larger sample size (albeit in a different region) than the previous study. We found that the ESI level 1 had exceptional high specificity than ESI level 1-2, however, its sensitivity was much lower. We aimed for a tool that was not only appropriate for using as a predicting poor outcome in the suspected sepsis patients, but also an optimal tool for tri- age. Therefore, we decided to explore the potential use of ESI level 1-2 instead of ESI level 1 in the present study. Furthermore, ESI level 1-2 had the highest sensitivity, higher than SIRS criteria, which have tradi- tionally been used in sepsis screening in the ED.

Table 1 Baseline characteristics of suspected sepsis patients.
	All patients	ESI level 1-2	SIRS criteria >=2	qSOFA >=2	NEWS >=6
	(N = 8177)	(N = 6490)	(N = 6149)	(N = 1230)	(N = 2917)
Age, yr -median	62	63	65	67	68
(min – max)	(18-100)	(18-100)	(18-100)	(18-100)	(18-100)
Men	4275 (52.3)	3423 (52.7)	3231 (52.5)	717 (58.3)	1593 (54.6)
Charlson comorbidity index -median (min – max)	3 (0-13)	4 (0-13)	4 (0-13)	4 (0-13)	4 (0-13)
Severity of patients
Sepsis	7205 (88.1)	5567 (85.8)	5385 (87.6)	667 (54.2)	2249 (77.1)
Sepsis induced hypotension	397 (4.9)	375 (5.8)	284 (4.6)	269 (21.9)	260 (8.9)
Septic shock	575 (7.0)	548 (8.4)	480 (7.8)	294 (23.9)	408 (14.0)
Positive hemoculture	788 (9.6)	687 (10.6)	681 (11.1)	181 (14.7)	374 (12.8)
Lactate values
First lactate N2 mmol/L	(N = 4694)	(N = 4214)	(N = 3817)	(N = 1084)	(N = 2427)
	2193 (46.7)	2047 (31.5)	1862 (30.3)	653 (53.1)	1300 (44.6)
Treatment
Oxygen therapy	3258 (39.8)	3065 (47.2)	2716 (44.2)	823 (66.9)	2130 (73.0)
Mechanical ventilator	597 (7.3)	597 (9.2)	505 (8.2)	208 (16.9)	433 (14.8)
Vasopressors	575 (7.0)	548 (8.4)	480 (7.8)	294 (23.9)	408 (14.0)
Disposition type
Ward	5315 (65.0)	4021 (62.0)	3920 (63.7)	645 (52.4)	1639 (56.2)
Intensive care	1311 (16.0)	1260 (19.4)	1098 (17.9)	489 (39.8)	907 (31.1)
Discharge	1551 (19.0)	1209 (18.6)	1131 (18.4)	96 (7.8)	371 (12.7)
Outcome In-hospital death	509 (6.2)	492 (7.6)	448 (7.3)	201 (16.3)	344 (11.8)
ICU stay	1810 (22.1)	1721 (26.5)	1516 (24.7)	608 (49.4)	1153 (39.5)

Data are presented as No. (%) unless otherwise indicated.

ED, emergency department; NEWS, National Early Warning Score; qSOFA, quick Sepsis-related Organ Failure Assessment; SIRS criteria, Systemic Inflammatory Response Syndromes.

Fig. 2. AUC of ESI, SIRS criteria, qSOFA, and NEWS for predicting in-hospital mortality showed that NEWS had a highest AUC (AUC: 0.73, 95% CI 0.70 to 0.75) for predicting in-hospital mortality, following by ESI (AUC: 0.70, 95% CI 0.68 to 0.73), qSOFA (AUC of 0.70, 95% CI 0.68 to 0.72) and SIRS criteria (AUC: 0.58, 95% CI 0.55 to 0.60). AUC, area under the receiver operating characteristic curve; ESI, Emergency Severity Index; NEWS, National Early Warning Score; qSOFA, quick Sepsis-related Organ Failure Assessment; SIRS, Systemic Inflammatory Response Syndrome.

There are several reasons that may explain why high ESI (level 1-2) was an accurate predictor of high-risk sepsis patients in the ED. First, the first step of ESI triage differs from other sepsis- identification tools, in that it relies clinical judgement (“visual as- sessment” by experienced triage personnel) to identify critically ill patients. No measurement of vital signs or laboratory parameters are required for this step. The triage nurse or doctor must detect life-threatening conditions by considering whether the patient needs immediate Life-saving interventions (i.e., cardiopulmonary re- suscitation, airway management, etc.). If such intervention is neces- sary, the patient is categorized as ESI level 1. This visual assessment step is the way ESI triage most differs from other sepsis identification tools. This might explain why ESI was one of the most accurate tools

in this study. The second step of ESI is to evaluate whether the pa- tient has a clinical confusion, lethargy, disorientation, distress, se- vere pain, or vital signs in the danger zone. If the patient has any of these signs/symptoms, they are categorized as ESI level 2 [18]. The ESI level 1-2 was found to have high sensitivity (96.7%) for predicting in-hospital mortality. A higher cut-off point, or ESI level 1, was also found to be specific for predicting in-hospital mortality (specificity: 88.3%).

Various sepsis-scoring systems have been designed for using in dif-

ferent situations and for differing purposes [5]. The SIRS criteria, origi- nally used in sepsis screening, have been criticized for being too sensitive and having low specificity [5,8]. By contrast, qSOFA has seen limited use in EDs due to its very low sensitivity [5,10,12,15]. The

Fig. 3. AUC of ESI, SIRS criteria, qSOFA, and NEWS for predicting ICU admission showed that ESI had a highest AUC (AUC of 0.75, 95% CI 0.74 to 0.76) for predicting ICU admission, following by ESI (AUC of 0.74, 95% CI 0.72 to 0.50), qSOFA (AUC: 0.697, 95% CI 0.68 to 0.71) and SIRS criteria (AUC: 0.570, 95% CI 0.69 to 0.71). AUC, area under the receiver operating characteristic curve; ESI, Emergency Severity Index; NEWS, National Early Warning Score; qSOFA, quick Sepsis-related Organ Failure Assessment; SIRS, Systemic Inflammatory Response Syndrome.

Table 2

Sensitivity, specificity, PPV and NPV of scoring systems for predicting in-hospital mortality and ICU admission in patients with suspected sepsis at the emergency department.

	In-hospital mortality					ICU admission
	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)		Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
ESI
1-5	100.0	0.0	6.2	NA		100.0	0.0	22.2	NA
1-4	100.0	0.3	6.2	100.0		100.0	0.3	22.2	100.0
1-3	99.2	7.3	6.6	99.3		98.8	8.5	23.5	96.1
1-2a	96.7	21.8	7.6	99.0		95.1	25.1	26.5	94.7
1	42.6	88.3	19.5	95.9		45.4	95.4	73.8	86.0
SIRS
>=1	98.4	6.6	6.5	98.4		97.0	7.2	22.9	89.5
>=2a	88.0	25.6	7.3	97.0		83.8	27.2	24.7	85.5
>=3	53.4	57.6	7.7	94.9		50.6	59.1	26.0	80.8
>=4	11.2	88.7	6.2	93.8		13.4	89.3	26.3	78.4
qSOFA
>=1	94.1	30.3	8.2	98.7		91.6	34.6	28.5	93.5
>=2a	39.5	86.6	16.3	95.6		33.6	90.2	49.4	82.7
>=3	6.1	99.2	34.4	94.1		3.37	99.5	67.8	78.4
NEWS
>=1	98.6	6.7	6.6	98.7		98.0	7.6	23.2	93.1
>=2	97.1	15.3	7.1	98.7		96.1	17.6	24.9	94.1
>=3	93.7	25.3	7.7	98.4		92.6	28.9	27.0	93.3
>=4	84.7	40.3	8.6	97.5		83.5	45.1	30.2	90.6
>=5	77.4	54.3	10.1	97.3		75.2	60.2	34.9	89.5
>=6a	67.6	66.4	11.8	96.9		63.7	72.3	39.5	87.5
>=7	54.0	77.3	13.6	96.2		49.4	82.4	44.3	85.1
>=8	42.8	85.5	16.4	95.7		37.6	89.7	51.0	83.5
>=9	30.8	91.4	19.2	95.2		25.5	94.4	56.6	81.7
>=10	21.4	94.9	21.8	94.8		16.5	96.8	59.5	80.3
>=11	15.5	97.3	27.3	94.5		10.3	98.4	64.7	79.4
>=12	9.0	98.6	29.9	94.2		5.8	99.2	67.5	78.7
>=13	4.1	99.4	30.0	94.0		2.9	99.7	74.3	78.3
>=14	1.8	99.7	29.0	93.9		1.3	99.9	77.4	78.1
>=15	0.8	99.9	44.4	93.8		0.5	100.0	100.0	78.0

ESI, Emergency Severity Index; NEWS, National Early Warning Score; PPV, Positive predictive value; NPV, Negative predictive value; qSOFA, quick Sepsis-related Organ Failure Assessment; SIRS, Systemic Inflammatory Response Syndrome.

^a Indicating the optimal cut-off points based on Youden’s index for predicted in-hospital mortality.

NEWS is more accurate when compared with both SIRS criteria and qSOFA, but its calculation is time-consuming calculation due to the large number of parameters, making it a less than ideal tool in an emer- gency setting [5,17]. The ESI is a promising all-in-one tool, both for pri- oritizing patients who visit EDs (as it has been traditionally used) and for predicting prognoses and outcomes of suspected sepsis patients. When a patient is classified as “suspected sepsis” based on a front- door ESI triage of 1-2, the doctor can initiate treatment immediately in accordance with the Sepsis Bundle [2] without a need for re- evaluation using other conventional sepsis screening tools.

This study had several strengths. First, it is the first study in an Asian population to demonstrate the accuracy of ESI as a prognostic tool com- pared with the traditional sepsis scoring systems in an emergency set- ting. Second, regardless of the retrospective design, our data were directly derived from a reliable electronic medical record with minimal missing data. Third, the initial hemodynamic parameters at ED triage were used for calculation rather than the worst values during the patient’s ED stay [7,16,24]. We were thus able to account for any possi- ble confounding factors such as the effects of oxygen therapy, intrave- nous fluid supplement, antibiotic administration, etc.

We are aware that our study had some potential limitations. First, experienced and well-trained medical personnel are required in order for the ESI system to be effective, as the first two steps are based solely on visual assessment. Therefore, the triage nurse must be properly trained in ESI use in order for the results of this study to apply in a given setting [19]. Secondly, ESI is an ED tool, and further study is needed to prove its efficacy for use in other departments (inpatient ward or ICU). Third, this was a single-center investigation in a tertiary care hospital in Thailand, where there are many patients with suspected

sepsis from tropical disease (e.g. leptospirosis, scrub typhus, melioidosis), which may limit its generalization.

Conclusion

The ESI was accurate in predicting in-hospital mortality and ICU admission in suspected sepsis patients in an emergency setting. Therefore, in addition to being useful in prioritizing these patients, it is also a promising tool for predicting adverse outcomes of suspected sepsis without the need for re-evaluation using other con- ventional tools.

Supplementary data to this article can be found online at https://doi. org/10.1016/j.ajem.2020.06.005.

Credit authorship contribution statement Pariwat Phungoen: Conceptualization, Methodology, Data curation,

Formal analysis, Writing – original draft, Writing – review & editing.

Sukanya Khemtong: Conceptualization, Methodology, Data curation, Formal analysis, Writing – original draft, Writing – review & editing. Korakot Apiratwarakul: Data curation, Formal analysis, Writing – re- view & editing. Kamonwan Lenghong: Conceptualization, Methodol- ogy, Writing – review & editing. Praew Kotruchin:Writing – original draft, Writing – review & editing.

Declaration of competing interest

The authors have no conflicts of interest to declare.

Acknowledgements

The authors would like to thank Dr. Dylan Southard for his editing of this manuscript. This study was funded by a grant from the Faculty of Medicine, Khon Kaen University, Thailand (grant number MN63203).

References

1. Seymour CW, Gesten F, Prescott HC, et al. Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med. 2017;376:2235-44. https:// doi.org/10.1056/NEJMoa1703058.
2. Levy MM, Evans LE, Rhodes A. The surviving sepsis campaign bundle: 2018 update. Intensive Care Med. 2018;44:925-8. https://doi.org/10.1007/s00134-018-5085-0.
3. Pruinelli L, Westra BL, Yadav P, et al. Delay within the 3-hour surviving sepsis cam- paign guideline on mortality for patients with severe sepsis and septic shock. Crit Care Med. 2018;46:500-5. https://doi.org/10.1097/CCM.0000000000002949.
4. Rhodes A, Evans LE, Alhazzani W, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Intensive Care Med. 2017;43:304-77. https://doi.org/10.1007/s00134-017-4683-6.
5. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the emergency department. Am J Emerg Med. 2019;37:1490-7. https://doi.org/10.1016/j.ajem.2018.10.058.
6. Song J-U, Sin CK, Park HK, Shim SR, Lee J. Performance of the quick sequential (sep- sis-related) organ failure assessment score as a prognostic tool in infected patients outside the intensive care unit: a systematic review and meta-analysis. Crit Care. 2018;22:28. https://doi.org/10.1186/s13054-018-1952-x.
7. Goulden R, Hoyle M-C, Monis J, et al. qSOFA, SIRS and NEWS for predicting Inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med

   J. 2018;35:345-9. https://doi.org/10.1136/emermed-2017-207120.

   Jiang J, Yang J, Mei J, Jin Y, Lu Y. Head-to-head comparison of qSOFA and SIRS criteria in predicting the mortality of infected patients in the emergency department: a meta-analysis. Scand J Trauma Resusc Emerg Med. 2018;26:56. https://doi.org/10. 1186/s13049-018-0527-9.

8. Liu Y-C, Luo Y-Y, Zhang X, et al. Quick sequential organ failure assessment as a prog- nostic factor for infected patients outside the intensive care unit: a systematic re- view and meta-analysis. Intern Emerg Med. 2019;14:603-15. https://doi.org/10. 1007/s11739-019-02036-0.
9. Raith EP, Udy AA, Bailey M, et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection ad- mitted to the intensive care unit. JAMA. 2017;317:290-300. https://doi.org/10.1001/ jama.2016.20328.
10. Brink A, Alsma J, Verdonschot RJCG, et al. Predicting mortality in patients with suspected sepsis at the emergency department; a retrospective cohort study com- paring qSOFA, SIRS and National Early Warning Score. PLoS ONE. 2019;14: e0211133. https://doi.org/10.1371/journal.pone.0211133.
11. Tusgul S, Carron P-N, Yersin B, Calandra T, Dami F. Low sensitivity of qSOFA, SIRS criteria and sepsis definition to identify infected patients at risk of complication in

    the prehospital setting and at the emergency department triage. Scand J Trauma Resusc Emerg Med. 2017;25:108. https://doi.org/10.1186/s13049-017-0449-y.

    Singer M, Deutschman CS, Seymour CW, et al. The third international consensus def- initions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315:801-10. https://doi. org/10.1001/jama.2016.0287.

12. Chen F-C, Kung C-T, Cheng H-H, et al. Quick Sepsis-related Organ Failure Assessment predicts 72-h mortality in patients with suspected infection. Eur J Emerg Med. 2019; 26:323. https://doi.org/10.1097/MEJ.0000000000000563.
13. Song J-U, Sin CK, Park HK, Shim SR, Lee J. Performance of the quick sequential (sep- sis-related) organ failure assessment score as a prognostic tool in infected patients outside the intensive care unit: a systematic review and meta-analysis. Crit Care. 2018;22:28. https://doi.org/10.1186/s13054-018-1952-x.
14. Churpek MM, Snyder A, Han X, et al. Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med. 2017;195:906-11. https://doi.org/10.1164/rccm.201604-0854OC.
15. Smith GB, Prytherch DR, Jarvis S, et al. A comparison of the ability of the physiologic components of medical emergency team criteria and the U.K. National Early Warn- ing Score to discriminate patients at risk of a range of adverse clinical outcomes. Crit Care Med. 2016;44:2171-81. https://doi.org/10.1097/CCM.0000000000002000.
16. Publications. ESI | Emergency Severity Index. https://www.esitriage.com/ publications-english. [Accessed 24 August 2019].
17. Tam HL, Chung SF, Lou CK. A review of triage accuracy and future direction. BMC Emerg Med. 2018;18:58. https://doi.org/10.1186/s12873-018-0215-0.
18. Mistry B, Stewart De Ramirez S, Kelen G, et al. Accuracy and reliability of emergency department triage using the Emergency Severity Index: an international multicenter assessment. Ann Emerg Med. 2018;71:581-587.e3. https://doi.org/10.1016/j. annemergmed.2017.09.036.
19. Savatmongkorngul S, Yuksen C, Suwattanasilp C, Sawanyawisuth K, Sittichanbuncha

    Y. Is a mobile Emergency Severity Index triage better than the paper ESI? In- tern Emerg Med. 2017;12:1273-7. https://doi.org/10.1007/s11739-016-1572-x.

    Kwak H, Suh GJ, Kim T, et al. prognostic performance of Emergency Severity Index combined with qSOFA score. Am J Emerg Med. 2018;36:1784-8. https://doi. org/10.1016/j.ajem.2018.01.088.

20. Nieves Ortega R, Rosin C, Bingisser R, Nickel CH. Clinical scores and formal triage for screening of sepsis and adverse outcomes on arrival in an emergency department all-comer cohort. J Emerg Med. 2019;57(4):453-460.e2. https://doi.org/10.1016/j. jemermed.2019.06.036.
21. Freund Y, Lemachatti N, Krastinova E, et al. Prognostic accuracy of Sepsis-3 criteria for in-hospital mortality among patients with suspected infection presenting to the emergency department. JAMA. 2017;317:301-8. https://doi.org/10.1001/jama. 2016.20329.
22. McHugh M, Tanabe P, McClelland M, Khare RK. More patients are triaged using the Emergency Severity Index than any other triage acuity system in the United States. Acad Emerg Med. 2012;19:106-9. https://doi.org/10.1111/j.1553-2712.2011.01240. x.